Spring Boot curl example with multipart/form-data and JSON metadata in POST request

Below are two curl examples of Spring Boot controller method signatures for a multipart/form-data POST request, one with request parameter metadata, one with JSON request body metadata.

Environment:
Spring Boot / Spring Boot Starter Batch Version 1.5.6.RELEASE
Oracle Java 8

1. multipart/form-data POST request with request parameter metadata

curl:
curl -k -H 'Content-Type: multipart/form-data' -F 'docFiles=@test1.txt;type=text/plain' 'https://localhost:8443/cherryshoe-app/document/file/upload/REVIEW?id=3554&docTypeId=1'

-k is to  to turn off curl's verification of the self-signed certificate
-F Contents to be read from file use the @

Matching Java Controller Method Signature:

@RequestMapping(value = "/document/file/upload", consumes = "multipart/form-data", produces = "application/json;charset=utf-8", method = RequestMethod.POST)

@ResponseBody
public DocumentUploadMetadata fileUpload(
    @RequestParam(value = "id", required = true) final Long reviewId,
    @RequestParam(value = "docTypeId", required = true) final Long docTypeId,
    @RequestParam("docFiles") List<MultipartFile> docFiles, Principal principal)
    throws IllegalArgumentException, NullPointerException, Exception {
}

2. multipart/form-data POST request with JSON request body metadata

curl:
curl -k -H 'Content-Type: multipart/form-data' -F 'docFiles=@test1.txt;type=text/plain' -F metadata='{"reviewId":3554,"docTypeId":1,"customFileName":"customFileName.txt","commentText":"This is a really long text, it may even be in HTML"};type=application/json' 'https://localhost:8443/cherryshoe-app/document/file/upload'

-k is to  to turn off curl's verification of the self-signed certificate
-F Contents to be read from file use the @

Matching Java Controller Method Signature:

@RequestMapping(value = "/document/file/upload", consumes = "multipart/form-data", produces = "application/json;charset=utf-8", method = RequestMethod.POST)

@ResponseBody
public DocumentUploadMetadata fileUpload(@RequestPart("docFiles") List<MultipartFile> docFiles,
    @RequestPart("metadata") DocumentUploadMetadata metadata, Principal principal)
    throws IllegalArgumentException, NullPointerException, Exception {
}

This Signature also works, had to do this to work with ng-file-upload library:

/**
 * Takes a list of document MultipartFile's, and JSON metadata.
 * 
 * NOTE: To make this work with ng-file-upload: - have to pass the JSON
 * metadata as a String, and then deserialize the JSON String manually in
 * the controller.
 * 
 * @param docFiles
 * @param metadata
 * @param principal
 * @return
 * @throws IllegalArgumentException
 * @throws NullPointerException
 * @throws Exception
 */
@RequestMapping(value = "/document/file/upload", consumes = "multipart/form-data", produces = "application/json;charset=utf-8", method = RequestMethod.POST)
@ResponseBody
public DocumentUploadMetadata fileUpload(@RequestParam("docFiles") List<MultipartFile> docFiles,
        @RequestParam("metadata") String metadataJson, Principal principal)
        throws IllegalArgumentException, NullPointerException, Exception {
  // deserialize metadataJson JSON string manually
}

POJO for JSON metadata serialization/deserialization:

public class DocumentUploadMetadata {
    private Long reviewId;
    private Long docTypeId;
    private String customFileName;
    private String commentText;

    public DocumentUploadMetadata() {
        super();
    }
    
    // getters and setters are needed for jackson serialization/deserialization, add them for this to work
}

Helpful curl documentation: https://curl.haxx.se/docs/manual.html

Spring Batch Decision with Spring Boot / Java Configuration Example (with equivalent XML)

I have a spring batch application using spring boot and java configuration.

Environment:
Spring Boot / Spring Boot Starter Batch Version 1.5.6.RELEASE
Oracle Java 8

BACKGROUND:
The spring batch application was working fine with a job with three steps - Each of the three steps had a reader/processor/writer where it was reading from the DB, processing db record specific metadata in the processor, and then writing the data records.  A job listener is doing a beforeJob and afterJob database record writing and updating for auditing purposes.  Each of the steps are doing something specific for each type of data being processed.

The Java config looked like the below for the Job.

@Bean
Job cherryShoeJob(JobBuilderFactory jobBuilderFactory, CherryShoeListener jobListener,
  @Qualifier("noOpStep") Step noOpStep,
  @Qualifier("step1") Step step1, @Qualifier("step2") Step step2, @Qualifier("step3") Step step3) {
  return jobBuilderFactory.get("cherryShoeJob").incrementer(new RunIdIncrementer()).listener(jobListener)
   .flow(step1).next(step2).next(step3).end().build();
}

Instead of using a scheduler to determine if a job was to be ran, a cron job is being used to call the executable jar at a scheduled time.  This was so the spring batch application didn't always have to be running on the server; instead it would start, run, and have the process completely stop.

PROBLEM:
We needed to add a second job.  Both jobs did not run at the same scheduled times.  The obvious answer would have been to switch the Jobs to use the scheduler; that would have been easy to solve; just schedule each Job with a specific time to run.  But that requires the spring batch application to constantly be running.  Because of the need to have the application start, run, then stop completely, this needed to be solved a different way.  I decided on this:

SOLUTION:
The solution below makes this work for one job.  A second job could easily be added via the same solution.

1.  add application.properties specific configuration for the job to be disabled.  If I wanted the job to run, then pass in the optional command line argument "--cherryshoe.job1.job.enabled=true" to enable the job.

# Overwrite the default cherryshoe.job1.job.enabled configuration property with a command line arguments "--cherryshoe.job1.job.enabled=true" to enable this job.
# Command line always have preference over the default configuration options.
cherryshoe.job1.job.enabled=false

java -jar cherryshoe-batch.jar --cherryshoe.job1.job.enabled=true

2.  Create a JobExecutionDecider to determine if the job should run or not.  This was determined by the cherryshoe.job1.job.enabled application.properties value, or the overridden value via command line argument.  If the job was disabled the FlowExecutionStatus would be STOPPED, if enabled the FlowExecutionStatus would be UNKNOWN.

@Component
public class CherryShoeExecutionDecider implements JobExecutionDecider {

 private static final Logger LOGGER = LoggerFactory.getLogger(CherryShoeExecutionDecider.class);

 @Value("${cherryshoe.job1.job.enabled}")
 protected Boolean jobEnabled;

 @Override
 public FlowExecutionStatus decide(JobExecution jobExecution, StepExecution stepExecution) {
  FlowExecutionStatus status = FlowExecutionStatus.STOPPED;
  if (jobEnabled)
   status = FlowExecutionStatus.UNKNOWN;

  LOGGER.info(
    "Job enabled[" + jobEnabled + "],[ flowExecutionStatus[" + status.getName() + "]");

  return status;
 }

}

3.  Create a "No Op" Step and a "No Op" Tasklet.  My new first step would be a no-op step just to be able to use a Decider to determine to continue running the Job, or stop.  If the job should continue, then it goes through the original three steps. If the job should stop, then jump to the no-op step again, just so we can end.

@Configuration
public class NoOpStepConfig {

 private static final Logger LOGGER = LoggerFactory.getLogger(NoOpStepConfig.class);

 /**
  * Step noOpStep
  * 
  * @param stepBuilderFactory
  * @return
  */
 @Bean
 Step noOpStep(StepBuilderFactory stepBuilderFactory, NoOpTasklet noOpTasklet) {
  LOGGER.debug("NoOp Step processing...");
  return stepBuilderFactory.get("noOpStep").tasklet(noOpTasklet).build();
 }
}

@Component
public class NoOpTasklet implements Tasklet {

 private Logger logger = LoggerFactory.getLogger(NoOpTasklet.class);

 @Override
 public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
  logger.debug("NoOp Tasklet processing...");
  return RepeatStatus.FINISHED;
 }

}

4.  The new Java config looked like the below for the Job.

@Bean
 Job cherryShoeJob(JobBuilderFactory jobBuilderFactory, CherryShoeListener jobListener,
   CherryShoeExecutionDecider jobExecutionDecider, @Qualifier("noOpStep") Step noOpStep,
   @Qualifier("step1") Step step1, @Qualifier("step2") Step step2, @Qualifier("step3") Step step3) {

  // Since we are using command line argument values to determine if a Job
  // should run or not (vs using a scheduler to determine that),
  // we must solve this by: 1. Step 1 is a no-op step just so we can use a
  // Decider to determine if we should continue running the Job, or stop.
  // If we continue
  // running the job, then it goes through each notification type step. If
  // we stop, then we jump to the no-op step again, just so we can end.
   return jobBuilderFactory.get("cherryShoeJob").incrementer(new
   RunIdIncrementer()).listener(jobListener)
     .flow(noOpStep).next(jobExecutionDecider)
       .on(FlowExecutionStatus.STOPPED.getName()).to(noOpStep)
     .from(noOpStep).next(jobExecutionDecider)
       .on(FlowExecutionStatus.UNKNOWN.getName()).to(step1).next(step2).next(step3)
     .end().build();
 }

XML Configuration
Step 2 - 4 above can be roughly converted into the XML configuration below.  NOTE: I did not test this with an XML config.

<!-- 2. CherryShoeExecutionDecider determines the FlowExecutionStatus of STOPPED or UNKNOWN depending on if if application.properties cherryshoe.job1.job.enabled is true or false -->
<beans:bean id="cherryShoeExecutionDecider" class="com.cherryshoe.batch.decider.CherryShoeExecutionDecider"/>

<!-- 3. NoOpTasklet does nothing -->
<beans:bean id="noOpTasklet" class="com.cherryshoe.batch.tasklet.NoOpTasklet" />

<!-- 4.  Job Configuration -->
<job id="cherryShoeJob">
    <!-- 3. NoOpStep does nothing but use NoOpTasklet -->
    <step id="noOpStep" next="jobExecutionDecider">
        <tasklet ref="noOpTasklet" />
    </step>
 
    <decision id="jobExecutionDecider" decider="cherryShoeExecutionDecider">
        <next on="STOPPED" to="noOpStep" />
        <next on="UNKNOWN" to="step1" />
    </decision>
        
    <step id="step1" next="step2" >
    </step>
        
    <step id="step2" next="step3">
    </step>

    <step id="step3">
    </step>
</job>

These were very helpful articles:
http://javasampleapproach.com/spring-framework/spring-batch/spring-batch-programmatic-flow-decision#4_Create_Flow_Decision
https://stackoverflow.com/questions/21782008/how-to-terminate-step-within-a-spring-batch-split-flow-with-a-decider
https://narmo7.wordpress.com/2014/05/14/spring-batch-how-to-setup-a-flow-job-with-java-based-configuration/
https://github.com/N4rm0/spring-batch-example/blob/master/src/main/java/springbatch/flowjob/JobConfig.java

Spring Boot - Example Email with Thymeleaf HTML Templates

I have a Spring Boot backend application that needed to send emails.

Java application email libraries include: Java mail native, Spring, and apache email (and others).  Lots of articles say apache email API is more intuitive, but others say if you are already using a Java/Spring application, then to go ahead and use the Spring one.   I decided to try the Spring Boot one first, since we have a Java/Spring application.

The Spring boot library satisfied the three criteria below, and seems to work great on my local environment:

1. the ability to send simple text email 
2. the ability to send HTML email (support for HTML emails already had the Thymeleaf HTML templating library provided with Spring Boot)
3. the ability to attach a file(s) to the email

The application is using the "Starters" spring boot jars (i.e. spring-boot-starter-web, spring-boot-starter-data-jpa, spring-boot-starter-security, etc), so I just needed to make the application aware of the spring-boot-starter-thymeleaf jar file for the application to use it.

Environment:
Windows 7
Oracle Java 1.8.0_66
Spring Boot 1.3.1.RELEASE
Sprint Boot 1.3.1.RELEASE spring-boot-starter-thymeleaf

The example below supports criteria number 2 to send HTML emails using Thymeleaf email templates.  I've also included unescaped HTML, passing multiple variables to HTML template, and inline image in the example below as well.

Spring Boot Unit Test error - Could not instantiate bean class [org.springframework.data.domain.Pageable]: Specified class is an interface

Below is an example with a rest web service using Spring Data, using a Pageable parameter (vs explicit @RequestParam's use of page, size, and sort parameters).  During unit testing of the controller method, I kept getting the "Could not instantiate bean class [org.springframework.data.domain.Pageable]: Specified class is an interface" error, and took some time to figure out how to make this work.  To make this work for the uint test, I chose a default Pageable object.  Below are my notes.

Environment:
Windows 7
Oracle Java 1.8.0_66
Spring Boot 1.3.1.RELEASE

Controller:

@RestController
@EnableAutoConfiguration
@CrossOrigin
public class CherryShoeController {

 
@Autowired
  private CherryShoeService cherryshoeService;

  // Use a @PageableDefault, for defaults if UI call does not supply it.  Also helps in unit testing to set Pageable values
  @RequestMapping(value = "/cherryshoe/data/listing", method = RequestMethod.GET,
     produces = "application/json;charset=utf-8")
  public Page<CherryShoeCustom> cherryshoeListing(
   @PageableDefault(page = 0, size = 10, sort = "id", direction = Sort.Direction.ASC) Pageable pageRequest,
   Principal principal) throws Exception
  {

    Page<CherryShoeCustom> pageListing = cherryshoeService.getDataPage(null,
     pageRequest.getPageNumber(), pageRequest.getPageSize(), pageRequest.getSort());
  
    return pageListing;
  }
 
 ...
}

Controller Unit Test:

public class CherryShoeControllerUnitTest {

  @InjectMocks
  private CherryShoeController unit;
 
  @Mock
  private Authentication mockAuthentication;
 
  @Mock
  private CherryShoeService mockCherryShoeService;
 
  private MockMvc mockMvc;
 
  @Before
  public void setup() throws Exception {
    // Process mock annotations
    MockitoAnnotations.initMocks(this);

    mockMvc = MockMvcBuilders.standaloneSetup(unit).build();
  }
 
  @Test
  public void cherryshoeListing_HappyTest() throws Exception {
    // *********************************************************
    // override mockMvc setup for paging support
    // Reference:
    // http://stackoverflow.com/questions/22174665/isolated-controller-test-cant-instantiate-pageable
    // The problem with pageable can be solved by providing a custom argument handler.
    // If this is set you will run in a ViewResolver Exception (loop). To avoid this you
    // have to set a ViewResolver (an anonymous JSON ViewResolver class for example).

    // Made this match the @PageableDefault default in the controller method
    Sort customSort = new Sort(Direction.ASC, "id");
    PageRequest customPageRequest = new PageRequest(0,10, customSort);
  
    mockMvc = MockMvcBuilders.standaloneSetup(unit)
             .setCustomArgumentResolvers(new PageableHandlerMethodArgumentResolver())
             .setViewResolvers(new ViewResolver() {
                 @Override
                 public View resolveViewName(String viewName, Locale locale) throws Exception {
                     return new MappingJackson2JsonView();
                 }
             })
             .build();
    // *********************************************************
  
    String fiscalYear = "2017";

    List<CherryShoeCustom> wrapperList = new ArrayList<CherryShoeCustom>();
    CherryShoeCustom wrapper = new CherryShoeCustom();
    wrapper.setFiscalYear(fiscalYear);
    wrapperList.add(wrapper);
    Page<CherryShoeCustom> pageList = new PageImpl<>(wrapperList);
  
    // define expectations on methods called by mocks inside the method
    // under test 
    when(mockCherryShoeService.getDataPage(customPageRequest.getPageNumber(),
      customPageRequest.getPageSize(), customPageRequest.getSort())).
      thenReturn(pageList);
  
    MvcResult result = mockMvc.perform(get("/cherryshoe/data/listing")
      .principal(mockAuthentication))
      .andReturn();

    System.out.println("results: " + result.getResponse().getContentAsString());

    // results are in the Page<T> data model format, I have a Page<CherryShoeCustom> data returning back
    // so modeled it in CherryShoeCustomPaging
    CherryShoeCustomPaging pagingReturn =
      objectMapper.readValue(result.getResponse().getContentAsString(), CherryShoeCustomPaging.class);
    System.out.println("****************" + pagingReturn);

    // verify method called once
    verify(mockCherryShoeService).getDataPage(customPageRequest.getPageNumber(),
      customPageRequest.getPageSize(), customPageRequest.getSort());
  
    // assertions
    assertEquals(HttpStatus.OK.value(), result.getResponse().getStatus());
    assertEquals(pagingReturn.getContent().get(0).getFiscalYear(), fiscalYear);
 }

}

CherryShoeCustomPaging: Data model returned from the cherryshoeService.getDataPage call (which called a repository with a custom query)

public class CherryShoeCustomPaging extends BasePaging {
  List<CherryShoeCustom> content;

  public List<CherryShoeCustom> getContent() {
    return content;
  }

  public void setContent(List<CherryShoeCustom> content) {
    this.content = content;
  }
 
  ...
}

BasePaging: Base data model returned from all Page<T> return types

public class BasePaging {
  private Integer totalElements;
  private Integer totalPages;
  private Boolean last;
  private Boolean size;
  private Boolean number;
  private List<BaseSorting> sort;
  private Boolean numberOfElements;
  private Boolean first;
 
  ...
}

BaseSorting: Sorting data model returned as a part of Page<T> return types

public class BaseSorting {
  private String direction;
  private String property;
  private Boolean ignoreCase;
  private String nullHandling;
  private Boolean ascending;
 
  ...
}

Helpful Links:
https://www.petrikainulainen.net/programming/spring-framework/spring-data-jpa-tutorial-part-seven-pagination
http://stackoverflow.com/questions/22174665/isolated-controller-test-cant-instantiate-pageable

Spring boot custom logback configuration

I have a spring boot application (v1.3.1.RELEASE) running on tomcat that needed to be re-factored to use spring boot provided logging tools for more robust and efficient logging.

Environment:
Windows 7
Oracle Java 1.8.0_66
Spring Boot 1.3.1.RELEASE
centos-release-6-8.el6.centos.12.3.x86_64
logback-core-1.1.3

Existing

1. Application console logs writing to <tomcat_home>/logs/catalina.out
2. Console log levels controlled via application.properties logging.level.* configuration, i.e.: logging.level.com.cherryshoe.app=DEBUG
3. Audit log manually written to via a BufferedWriter to a path and filename (This was the biggest driver to the refactoring because of inefficiency)

New Solution
The application is using the "Starters" spring boot jars (i.e. spring-boot-starter-web, spring-boot-starter-data-jpa, spring-boot-starter-security, etc), so Logback is used by default under the covers.  Since this was the case, I configured a custom spring-logback.xml file and made it available in the classpath so it would be automatically recognized and loaded.

Below are two solutions that were adequate - the difference between the two is how the application log file cherryshoe-app.log is implemented / configured.  The audit log file cherryshoe_audit.log is implemented / configured the same way in both solutions.  Pick what suits you best!

Option 1 Summary -

1. Configure application logs to write to a custom application log file ONLY (not to console) via application.properties and logback-spring.xml configuration 
• application.properties:
• logging.file property configured
• logback-spring.xml
• FILE appender is the preferred approach because it is provided by spring boot framework for logback and already supports rolling logs with a Size-Based triggering policy (When size is reached, it will roll the log to <filename>.1, <filename>.2, etc)
• All appender-ref's to CONSOLE have been disabled.  This should be the case for deployment to non-local environments.  For local development environments, you can enable it, so you don't have to tail an extra file (the logging.file configured file) during development
• Log levels are configured by package
• Audit logs are writing to a custom audit log file using a Sized-Based or Time-Based rolling policy
2. In the java classes, use LoggerFactory.getLogger(getClass()) to write to either application or audit log

application.properties

...
logging.file=./cherryshoe-app.log
...

logback-spring.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <!-- include base.xml to get access to FILE and CONSOLE appenders -->
    <include resource="org/springframework/boot/logging/logback/base.xml" />

    <property name="log_location" value="." />
    <property name="cherryshoe_audit_file_name" value="cherryshoe_audit.log" />
    
    <!-- logger additivity=false ensures logs only goes to appender(s) specified, but not to any appender attached higher in the hierarchy. -->
    
    <!-- Configure an application specific log file (i.e. cherryshoe-app.log) via configuration in application.properties.
            NOTE: This is the preferred, there is a default rollingPolicy configured for FILE, but not for CONSOLE.  We should just
            not use CONSOLE.
            #logging.file=./cherryshoe-app.log
    
        We can limit logs that go to catalina.out for console output by commenting out / removing appender-ref to CONSOLE 
        OR
        override via https://docs.spring.io/spring-boot/docs/current/reference/html/howto-logging.html with FILE appender only
    -->
    <root level="INFO">
        <appender-ref ref="FILE" />
        <!--<appender-ref ref="CONSOLE" />-->
    </root>
    
    <!-- Define logging levels for CONSOLE and FILE which are inherited from base.xml -->    
    <logger name="com.cherryshoe.app" level="DEBUG" additivity="false">
        <appender-ref ref="FILE" />
        <!--<appender-ref ref="CONSOLE" />-->
    </logger>    
        
    <!-- Default Spring logback defines CONSOLE and FILE appenders, we'll make a new appender to support audit logs called "CHERRYSHOE-AUDIT" -->
    <appender name="CHERRYSHOE-AUDIT"
        class="ch.qos.logback.core.rolling.RollingFileAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
        <file>${log_location}/${cherryshoe_audit_file_name}</file>

        <!-- Two choices for rolling appenders, only enable one:  1. Size-Based or 2. Time-Based.-->
        <!-- 1. Size-Based Triggering Policy -->
         <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
             <fileNamePattern>${log_location}/${cherryshoe_audit_file_name}.%i</fileNamePattern>
         </rollingPolicy>
         <triggeringPolicy
             class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
             <maxFileSize>10MB</maxFileSize>
         </triggeringPolicy>

        <!-- 2. Time-Based Rolling Policy - %d is a daily rollover with format yyyy-MM-dd -->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${log_location}/cherryshoe_audit.log.%d</fileNamePattern>
        </rollingPolicy>
    </appender>
    
    <logger name="com.cherryshoe.app.audit" level="INFO" additivity="false">
        <appender-ref ref="CHERRYSHOE-AUDIT"/>
    </logger>
    
</configuration>

application logging in java classes

package com.cherryshoe.app;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

...

private Logger log = LoggerFactory.getLogger(getClass());

...
log.info("Start application logging.");
...

audit logging in java classes

package com.cherryshoe.app.audit;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

...
    private Logger log = LoggerFactory.getLogger(getClass());
...
log.info("Start audit logging.");
...

Option 2 Summary -

1. Configure application logs to write to a custom application log file ONLY (not to console) via logback-spring.xml configuration
• Do not use the Spring boot default FILE appender because it has a Size-Based rolling policy, instead let's use a Time-Based rolling policy by date for the custom application log
• All appender-ref's to CONSOLE have been disabled.  This should be the case for deployment to non-local environments.  For local development environments, you can enable it, so you don't have to tail an extra file (the logging.file configured file) during development
• Log levels are configured by package
• Audit logs are writing to a custom audit log file using a Sized-Based or Time-Based rolling policy
2. In the java classes, use LoggerFactory.getLogger(getClass()) to write to either application or audit log

application.properties - no changes needed because logging.path is used in default log configuration only, not with custom logging.

logback-spring.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <!-- include defaults.xml and console-appender.xmlto get access to CONSOLE appender only -->
    <include resource="org/springframework/boot/logging/logback/defaults.xml" />
    <include resource="org/springframework/boot/logging/logback/console-appender.xml" />
    
    <!-- properties used inside this configuration file -->
    <property name="log_location" value="." />
    <property name="cherryshoe_app_file_name" value="cherryshoe-app.log" />
    <property name="cherryshoe_audit_file_name" value="cherryshoe_audit.log" />
    
    <!-- logger additivity=false ensures logs only goes to appender(s) specified, but not to any appender attached higher in the hierarchy. -->
    
    <!-- Cannot override default FILE appender to change it to a Time-Based Rolling Policy, 
    we'll make a new file appender called "CHERRYSHOE-FILE" --> 
    <appender name="CHERRYSHOE-FILE"
        class="ch.qos.logback.core.rolling.RollingFileAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
        <file>${log_location}/${cherryshoe_app_file_name}</file>

        <!-- Time-Based Rolling Policy - %d is a daily rollover with format yyyy-MM-dd -->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${log_location}/${cherryshoe_app_file_name}.%d</fileNamePattern>
        </rollingPolicy>
    </appender>
    
    <root level="INFO">
        <appender-ref ref="CHERRYSHOE-FILE" />
        <!--<appender-ref ref="CONSOLE" />-->
    </root>
    
    <logger name="com.cherryshoe.app" level="DEBUG" additivity="false">
        <appender-ref ref="CHERRYSHOE-FILE" />
        <!--<appender-ref ref="CONSOLE" />-->
    </logger>        
        
    <!-- Define a new appender to support audit logs called "CHERRYSHOE-AUDIT" -->
    <appender name="CHERRYSHOE-AUDIT"
        class="ch.qos.logback.core.rolling.RollingFileAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
        <file>${log_location}/${cherryshoe_audit_file_name}</file>

        <!-- Time-Based Rolling Policy - %d is a daily rollover with format yyyy-MM-dd -->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${log_location}/${cherryshoe_audit_file_name}.%d</fileNamePattern>
        </rollingPolicy>
    </appender>
    
    <logger name="com.cherryshoe.app.audit" level="INFO" additivity="false">
        <appender-ref ref="CHERRYSHOE-AUDIT"/>
    </logger>
    
</configuration>

application logging in java classes

package com.cherryshoe.app;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

...

private Logger log = LoggerFactory.getLogger(getClass());

...
log.info("Start application logging.");
...

audit logging in java classes

package com.cherryshoe.app.audit;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

...

    private Logger log = LoggerFactory.getLogger(getClass());
...
log.info("Start audit logging.");
...

Backup the file by date:
Here's a sample bash script to backup the audit file by date.  A cron job should be scheduled for the crurent day to search for the file created yesterday.

#!/bin/bash
# This script will scp the audit log file from YESTERDAY over to the file server for backup purposes
CHERRYSHOE_AUDIT_FILENAME=cherryshoe_audit.log

YESTERDAY=`date +%Y-%m-%d -d yesterday`
YEAR=${YESTERDAY:0:4}
MONTH=${YESTERDAY:5:2}
CHERRYSHOE_AUDIT_FILENAME_YESTERDAY=$CHERRYSHOE_AUDIT_FILENAME.$YESTERDAY

echo $YESTERDAY
echo $CHERRYSHOE_AUDIT_FILENAME_YESTERDAY

if [ -e "$CHERRYSHOE_AUDIT_FILENAME_YESTERDAY" ]
  then
    echo $CHERRYSHOE_AUDIT_FILENAME_YESTERDAY exists
    #TODO SSH mkdir location/$YEAR/$MONTH && SCP file over to fileserver 
else
  echo $CHERRYSHOE_AUDIT_FILENAME_YESTERDAY does not exist
fi

My own additional questions:
Which appender is used when using LoggerFactory.getLogger(getClass())?

• Whichever package your class is located, it is going to use the appenders configured there.  In the logback-spring.xml file example above - for the com.cherryshoe.app package, a java class that lives there will use the appenders assigned in the logger for that package
• You could also explicitly grab the logger by name if you wanted, vs counting on having the class be inside the package described by the name attribute (NOTE: This only worked for me on Windows not on CentOS).   In the logback-spring.xml file above I defined the below logger with name "cherryshoeaudit".  

      <logger name="cherryshoeaudit" level="INFO" additivity="false">
          <appender-ref ref="CHERRYSHOE-AUDIT"/>
      </logger>
  

  And retrieved it by name in the java class

      private static final String CHERRYSHOE_AUDIT_LOG = "cherryshoeaudit";
      private Logger log = LoggerFactory.getLogger(CHERRYSHOE_AUDIT_LOG );
  

  I could also just as easily defined the logger as

      <logger name="com.cherryshoe.app.audit" level="INFO" additivity="false">
          <appender-ref ref="CHERRYSHOE-AUDIT"/>
      </logger>
  

  and then have a choice of having a java class live inside the package com.cherryshoe.app.audit, or retrieving the logger by name "com.cherryshoe.app.audit" regardless of where the java class lived.

Helpful Links

• https://logback.qos.ch/manual/appenders.html
• https://github.com/spring-projects/spring-boot/tree/master/spring-boot/src/main/resources/org/springframework/boot/logging/logback
• https://docs.spring.io/spring-boot/docs/current/reference/html/howto-logging.html
• https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-logging.html

Spring boot ehcache timeToLiveSeconds not being honored

I have a Spring Boot backend application (version 1.3.1.RELEASE) where caching is enabled on certain database calls.  I ran into a problem where the caching eviction time was not being honored.

Steps to enable using ehcache 2.x as cache provider:

1. @EnableCaching was added to the configuration
2. ehcache.xml was created on the root of the classpath with configuration for the cache
<cache name="dataToCache" maxEntriesLocalHeap="7" timeToLiveSeconds="600">
</cache>

3. Added following annotation to the database Database service method
   @Cacheable(cacheNames = "dataToCache", key = "#root.methodName + '-' + #dataToCache")
   
4. Manually tested if caching was working by setting spring.jpa.show-sql to true in application.properties.  And called the api that would
   call the service.  Only the first time an actual call to the database was made.  This was good.  I assumed all was well.
   

Problem:
I was expecting after 10 mins (the timeToLiveSeconds attribute was set to 600 seconds) the dataToCache cache would be evicted and new data that was added within that ten minutes would now show up.  Nope.

My problem was that I assumed caching was working since the first time an actual call to the database was made (and logged).  I incorrectly assumed caching was working with ehcache 2.x because I had added ehcache.xml to my classpath.  Simple caching was being used because "If no caching library is present in the application, simple caching is used by default".

Logs show this:
2016-08-17 09:46:15.250  INFO 4512 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.cache.SimpleCacheConfiguration' of type [class org.springframework.boot.autoconfigure.cache.SimpleCacheConfiguration$$EnhancerBySpringCGLIB$$79234830] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

Solution:
I had inadvertently forgotten to add the ehcache jar file to the classpath!!!    After the ehcache jar was added to classpath, my timeToLiveSeconds was being honored, and now the logs verify the following:
2016-08-17 10:13:10.876  INFO 14056 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.cache.EhCacheCacheConfiguration' of type [class org.springframework.boot.autoconfigure.cache.EhCacheCacheConfiguration$$EnhancerBySpringCGLIB$$5d32863] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2016-08-17 10:13:11.883  INFO 14056 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'ehCacheCacheManager' of type [class net.sf.ehcache.CacheManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2016-08-17 10:13:11.905  INFO 14056 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'cacheManager' of type [class org.springframework.cache.ehcache.EhCacheCacheManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

Spring boot - is it possible to have Global Cors Configuration using only application.properties?

No, not for Spring Boot 1.3.1.RELEASE at least.

I have a Spring Boot backend application, and an AngularJS front-end application; they run on application servers on different ports (front end on port 3000, backend on port 8443). Because of this, the spring boot application needed to support CORS.

OLD
I originally had the application set up with Global CORS configuration, similar to the following.  All REST endpoints, HTTP GET/PUT/POST supported, with localhost port 3000 as the allowed Origin.

public class CherryShoeApplication {
    public static void main(String[] args) {
        SpringApplication.run(CherryShoeApplication.class, args);
    }
    
    /*
     * Since spring boot 1.3, Global CORS configuration can be defined by registering 
     * a WebMvcConfigurer bean with a customized addCorsMappings(CorsRegistry) method
     */
     @Bean
    public WebMvcConfigurer corsConfigurer() {
        return new WebMvcConfigurerAdapter() {
            @Override
            public void addCorsMappings(CorsRegistry registry) {
                registry.addMapping("/**")
                .allowedMethods("GET", "PUT", "POST")
                .allowedOrigins("http://localhost:3000")
                ;
             }
         };
    }
}

Needed to update the WebSecurityConfig to allow all HTTP OPTIONS through.  Spring Security pre-authorization was used to protect the application, in this particular example.

public class WebSecurityConfig extends WebSecurityConfigurerAdapter {
    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http.addFilterBefore(ssoFilter(),
        RequestHeaderAuthenticationFilter.class)
        .authenticationProvider(preauthAuthProvider()).csrf().disable()
        .authorizeRequests()
        .antMatchers(HttpMethod.OPTIONS, "/**").permitAll() // allow CORS OPTIONS calls through
        .anyRequest().authenticated();
    }
}

NEW
I wanted to move Global CORS configuration out of the Java Config and into application.properties, so it'd be easier to update without a code update.

The same Java Config above (public WebMvcConfigurer corsConfigurer()) is now configured as the following in application.properties:

# ENDPOINTS CORS CONFIGURATION (EndpointCorsProperties)
# Set whether credentials are supported. When not set, credentials are not supported.
endpoints.cors.allow-credentials=false
# Comma-separated list of headers to allow in a request. '*' allows all headers.
endpoints.cors.allowed-headers=*
# Comma-separated list of methods to allow. '*' allows all methods.
endpoints.cors.allowed-methods=GET,PUT,POST
# Comma-separated list of origins to allow. '*' allows all origins. When not set, CORS support is disabled.
endpoints.cors.allowed-origins=http://localhost:3000
# Comma-separated list of extra headers to include in a response, these get exposed by default 
# Cache-Control, Content-Language, Content-Type, Expires, Last-Modified, Pragma
#endpoints.cors.exposed-headers=
# How long, in seconds, the response from a pre-flight request can be cached by clients. 
#endpoints.cors.max-age=1800 

Key takeaway:
There was no way to do a global cors configuration via essentially an "empty" WebMvcConfigurer corsConfigurer (In other words, completely removing the WebMvcConfigurer corsConfigurer @Bean/method), and then use only the application.properties to configure it.  To get the application.properties cors parameters recognized, the @CrossOrigin annotation also needs to be added to each controller. I decided to create a base class and have each controller extend it, so it'd be easier to maintain in the future.

/**
 * Base class so all controllers can extend this class to inherit the @CrossOrigin annotation
 */
@CrossOrigin
public class CrossOriginController {
    // empty, just need the CrossOrigin annotation
}

If you know of any other way to do a global cors configuration using application.propeties cors parameters, please let me know!

Integrating JRebel into a Spring Boot Gradle run

This post talks about integrating JRebel with a Spring Boot Gradle run; it builds upon my last post on How to remote debug a gradle run.

Environment:
Windows 7
Gradle 2.9
Java 1.8.0_66

Step 1: Update build.gradle file

I am using gradle version 2.9, so was able to put the following after the buildscript {} script block in build.gradle:
plugins {
  id "org.zeroturnaround.gradle.jrebel" version "1.1.3"
}

My project is building a war file, so I added the following after apply plugin: 'war', right before the applicationDefaultJvmArgs configuration:

war.dependsOn(generateRebel)

Now, after running 'gradle build' or 'gradle war', I see the rebel.xml file is generated here and with the following contents:
$ find . -name rebel.xml -print
./build/classes/main/rebel.xml

<?xml version="1.0" encoding="UTF-8"?>
<application xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.zeroturnaround.com" xsi:schemaLocation="http://www.zeroturnaround.com http://www.zeroturnaround.com/alderaan/rebel-2_0.xsd">

  <classpath>
    <dir name="D:/workspaces/cherryshoe-team/cherryshoe/cherryshoe-app/build/resources/main">
    </dir>
    <dir name="D:/workspaces/cherryshoe-team/cherryshoe/cherryshoe-app/build/classes/main">
    </dir>
  </classpath>

  <web>
    <link target="/">
      <dir name="D:/workspaces/cherryshoe-team/cherryshoe/cherryshoe-app/src/main/webapp">
      </dir>
    </link>
  </web>

</application>

This article helped a lot.

NOTE:  You could check in the rebel.xml in your code base at this point, and add in a ${workspace.home} variable and then add in the workspace.home variable to your JVM, so each developer could use the generated rebel.xml file at this point.  Or we could have each developer run 'gradle build' or 'gradle war' one time to generate this file themselves for their specific environment.

Step 2: Activate jrebel for standalone agent
Install JRebel without installing any IDE plugins

• Download current stable version of jrebel no setup zip https://zeroturnaround.com/software/jrebel/download/
• Standalone install: http://manuals.zeroturnaround.com/jrebel/standalone/index.html
• Extract the jrebel stand alone zip to C:\Apps
• Read readme.txt, which says to run bin\activate-gui.cmd to activate the license (Use an activation code, company license server url, or your stand-alone jrebel license file)
• Add in JREBEL_HOME to system environment variables

Step 3:  IDE configuration (if using eclipse for example)

• Project -> Build automatically is checked
• The compilation output directory of the eclipse IDE and the JRebel monitored classes directory (defined in rebel.xml) have to match in order for the class reloading to work. Therefore, make sure that your IDE is compiling classes into the same directory where both your Gradle project model and the rebel.xml file are expecting them (build/classes/main by default, as opposed to bin which is the default for Eclipse).
• Right click on Project -> Build Path -> Configure Build Path -> Source Tab -> Default output folder change it from <project name>/bin to <project name>/build/classes/main
• NOTE:  I have a problem right now when I change the default output folder in eclipse where when I run JUnit unit or JUnit integration tests, eclipse says it can't find the compiled class files.  I still need to figure this out while also having JRebel work, so I don't have to switch back and forth.

Step 4:  Enable jrebel javaagent with the JVM options for the java application
I decided to solve this by updating JVM options via the build.gradle application plugin applicationDefaultJvmArgs property.

• Update applicationDefaultJvmArgs to include jrebel configuration

OLD:
applicationDefaultJvmArgs = [
    "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005"
]

NEW:
applicationDefaultJvmArgs = [
    "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005", "-javaagent:C:\\Apps\\jrebel\\jrebel.jar"
]

Make sure the parameters are in this format:  ["-Dparam1", "-Dparam2"].  I originally updated it to the below, but got an error since the gradle task was trying to fork another process [detailed error below].

WRONG:
applicationDefaultJvmArgs = [
    "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 -javaagent:C:\\Apps\\jrebel\\jrebel.jar"
]

ERROR:

$ gradle bootRun

:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:findMainClass
:bootRun
FATAL ERROR in native method: JDWP No transports initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)
ERROR: transport error 202: bind failed: Cannot assign requested address
ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [debugInit.c:750]
:bootRun FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':bootRun'.
> Process 'command 'C:\Apps\Java\jdk1.8.0_66\bin\java.exe'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 10.85 secs


You don't need to add any configuration in <HOME>/.gradle/gradle.properties to have this work.

Helpful documentation:

• https://docs.gradle.org/current/userguide/application_plugin.html
• https://manuals.zeroturnaround.com/jrebel/standalone/gradle.html
• http://manuals.zeroturnaround.com/jrebel/standalone/index.html

How to remote debug a gradle run

I'm using a spring boot application that is run by gradle via "gradle bootRun". The following are two ways to remote debug the java application:

1. "gradle bootRun --debug-jvm" is a shortcut to remote debug by adding the following to the JVM parameters "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005" 
2. The application plugin applicationDefaultJvmArgs property can be added to build.gradle. I don't like to suspend the application, so note suspend=n.
   
   apply plugin: 'application' applicationDefaultJvmArgs = [ "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" ]
   
   You don't need to add any configuration in <HOME>/.gradle/gradle.properties to have this work for either solution.

Helpful documentation:

• https://docs.gradle.org/current/userguide/application_plugin.html
• http://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#howto-remote-debug-gradle-run