Sunday, October 8, 2017

Spring Batch Decision with Spring Boot / Java Configuration Example (with equivalent XML)

I have a spring batch application using spring boot and java configuration.

Environment:
Spring Boot / Spring Boot Starter Batch Version 1.5.6.RELEASE
Oracle Java 8

BACKGROUND:
The spring batch application was working fine with a job with three steps - Each of the three steps had a reader/processor/writer where it was reading from the DB, processing db record specific metadata in the processor, and then writing the data records.  A job listener is doing a beforeJob and afterJob database record writing and updating for auditing purposes.  Each of the steps are doing something specific for each type of data being processed.

The Java config looked like the below for the Job.
@Bean
Job cherryShoeJob(JobBuilderFactory jobBuilderFactory, CherryShoeListener jobListener,
  @Qualifier("noOpStep") Step noOpStep,
  @Qualifier("step1") Step step1, @Qualifier("step2") Step step2, @Qualifier("step3") Step step3) {
  return jobBuilderFactory.get("cherryShoeJob").incrementer(new RunIdIncrementer()).listener(jobListener)
   .flow(step1).next(step2).next(step3).end().build();
}

Instead of using a scheduler to determine if a job was to be ran, a cron job is being used to call the executable jar at a scheduled time.  This was so the spring batch application didn't always have to be running on the server; instead it would start, run, and have the process completely stop.

PROBLEM:
We needed to add a second job.  Both jobs did not run at the same scheduled times.  The obvious answer would have been to switch the Jobs to use the scheduler; that would have been easy to solve; just schedule each Job with a specific time to run.  But that requires the spring batch application to constantly be running.  Because of the need to have the application start, run, then stop completely, this needed to be solved a different way.  I decided on this:

SOLUTION:
The solution below makes this work for one job.  A second job could easily be added via the same solution.

1.  add application.properties specific configuration for the job to be disabled.  If I wanted the job to run, then pass in the optional command line argument "--cherryshoe.job1.job.enabled=true" to enable the job.
# Overwrite the default cherryshoe.job1.job.enabled configuration property with a command line arguments "--cherryshoe.job1.job.enabled=true" to enable this job.
# Command line always have preference over the default configuration options.
cherryshoe.job1.job.enabled=false

java -jar cherryshoe-batch.jar --cherryshoe.job1.job.enabled=true

2.  Create a JobExecutionDecider to determine if the job should run or not.  This was determined by the cherryshoe.job1.job.enabled application.properties value, or the overridden value via command line argument.  If the job was disabled the FlowExecutionStatus would be STOPPED, if enabled the FlowExecutionStatus would be UNKNOWN.

@Component
public class CherryShoeExecutionDecider implements JobExecutionDecider {

 private static final Logger LOGGER = LoggerFactory.getLogger(CherryShoeExecutionDecider.class);

 @Value("${cherryshoe.job1.job.enabled}")
 protected Boolean jobEnabled;

 @Override
 public FlowExecutionStatus decide(JobExecution jobExecution, StepExecution stepExecution) {
  FlowExecutionStatus status = FlowExecutionStatus.STOPPED;
  if (jobEnabled)
   status = FlowExecutionStatus.UNKNOWN;

  LOGGER.info(
    "Job enabled[" + jobEnabled + "],[ flowExecutionStatus[" + status.getName() + "]");

  return status;
 }

}

3.  Create a "No Op" Step and a "No Op" Tasklet.  My new first step would be a no-op step just to be able to use a Decider to determine to continue running the Job, or stop.  If the job should continue, then it goes through the original three steps. If the job should stop, then jump to the no-op step again, just so we can end.
@Configuration
public class NoOpStepConfig {

 private static final Logger LOGGER = LoggerFactory.getLogger(NoOpStepConfig.class);

 /**
  * Step noOpStep
  * 
  * @param stepBuilderFactory
  * @return
  */
 @Bean
 Step noOpStep(StepBuilderFactory stepBuilderFactory, NoOpTasklet noOpTasklet) {
  LOGGER.debug("NoOp Step processing...");
  return stepBuilderFactory.get("noOpStep").tasklet(noOpTasklet).build();
 }
}

@Component
public class NoOpTasklet implements Tasklet {

 private Logger logger = LoggerFactory.getLogger(NoOpTasklet.class);

 @Override
 public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
  logger.debug("NoOp Tasklet processing...");
  return RepeatStatus.FINISHED;
 }

}

4.  The new Java config looked like the below for the Job.
@Bean
 Job cherryShoeJob(JobBuilderFactory jobBuilderFactory, CherryShoeListener jobListener,
   CherryShoeExecutionDecider jobExecutionDecider, @Qualifier("noOpStep") Step noOpStep,
   @Qualifier("step1") Step step1, @Qualifier("step2") Step step2, @Qualifier("step3") Step step3) {

  // Since we are using command line argument values to determine if a Job
  // should run or not (vs using a scheduler to determine that),
  // we must solve this by: 1. Step 1 is a no-op step just so we can use a
  // Decider to determine if we should continue running the Job, or stop.
  // If we continue
  // running the job, then it goes through each notification type step. If
  // we stop, then we jump to the no-op step again, just so we can end.
   return jobBuilderFactory.get("cherryShoeJob").incrementer(new
   RunIdIncrementer()).listener(jobListener)
     .flow(noOpStep).next(jobExecutionDecider)
       .on(FlowExecutionStatus.STOPPED.getName()).to(noOpStep)
     .from(noOpStep).next(jobExecutionDecider)
       .on(FlowExecutionStatus.UNKNOWN.getName()).to(step1).next(step2).next(step3)
     .end().build();
 }

XML Configuration
Step 2 - 4 above can be roughly converted into the XML configuration below.  NOTE: I did not test this with an XML config.
<!-- 2. CherryShoeExecutionDecider determines the FlowExecutionStatus of STOPPED or UNKNOWN depending on if if application.properties cherryshoe.job1.job.enabled is true or false -->
<beans:bean id="cherryShoeExecutionDecider" class="com.cherryshoe.batch.decider.CherryShoeExecutionDecider"/>

<!-- 3. NoOpTasklet does nothing -->
<beans:bean id="noOpTasklet" class="com.cherryshoe.batch.tasklet.NoOpTasklet" />

<!-- 4.  Job Configuration -->
<job id="cherryShoeJob">
    <!-- 3. NoOpStep does nothing but use NoOpTasklet -->
    <step id="noOpStep" next="jobExecutionDecider">
        <tasklet ref="noOpTasklet" />
    </step>
 
    <decision id="jobExecutionDecider" decider="cherryShoeExecutionDecider">
        <next on="STOPPED" to="noOpStep" />
        <next on="UNKNOWN" to="step1" />
    </decision>
        
    <step id="step1" next="step2" >
    </step>
        
    <step id="step2" next="step3">
    </step>

    <step id="step3">
    </step>
</job>


These were very helpful articles:
http://javasampleapproach.com/spring-framework/spring-batch/spring-batch-programmatic-flow-decision#4_Create_Flow_Decision
https://stackoverflow.com/questions/21782008/how-to-terminate-step-within-a-spring-batch-split-flow-with-a-decider
https://narmo7.wordpress.com/2014/05/14/spring-batch-how-to-setup-a-flow-job-with-java-based-configuration/
https://github.com/N4rm0/spring-batch-example/blob/master/src/main/java/springbatch/flowjob/JobConfig.java

1 comment:

I appreciate your time in leaving a comment!