Enabling PM2 logging configuration for a Node.js application

I'm working on an application where PM2 manages the node processes that are started with npm.  The default PM2 log configuration was being used, so my task was to configure PM2 logs for added functionality such as:  log location, log rotation, timestamps in logs, etc.  This article will only be talking about how to configure a base ecosystem.config.js file for the PM2 log configuration to be recognized.

Environment:
CentOS Linux release 7.3.1611 (Core)
pm2 version 3.2.3
npm version 6.4.1
node version v8.16.0

Originally, PM2 was managing the node processes with npm start scripts with the following commands:
HOST=cherryshoe-dev.com pm2 start npm --name cherryshoe-dev -- run startdev
HOST=cherryshoe.com NODE_ENV=production pm2 start npm --name cherryshoe -- start

The --name "cherryshoe-dev" or "cherryshoe" is the name of the PM2 app name that is started.
The "-- run startdev" and the "-- start" were indicating to call the npm script located in package.json.  i.e. scripts.startdev or scripts.start in the below json snippet.

package.json looked like:

  "scripts": {
    "start": "npm run server",
    "startdev": "concurrently \"npm run server\" \"npm run client\"",
    "server": "cross-env node --max-old-space-size=8192 ./bin/cherryshoeServer.js",
    "client": "node cherryshoe-client.js"
  }

To enable PM2 logging configuration, the ecosystem file had to be configured to call the custom npm scripts appropriately, depending on the environment.

// If you look at a default ecosystem.config.js, you'll see that
// an assumption is made that each npm start command is the same.  The app has
// different npm custom scripts defined in package.json to run depending
// on the environment - so because of this, decided to have multiple apps
// and to call them by "pm2 start --only <app-name>" to start.
module.exports = {
  apps : [
    // Local DEV app config
    {
      name: 'cherryshoe-dev',
      script: 'npm',
      // call custom start dev npm script from package.json
      args: 'run startdev',

      env: {
        NODE_ENV: 'development'
      }
    },
    // PROD app config
    {
      name: 'cherryshoe',
      script: 'npm',
      args: 'start',

      env: {
        NODE_ENV: 'production'
      }
    }
  ],
};

After this change, the command to have PM2 manage the start of the node process with npm became easier:
HOST=cherryshoe-dev.com pm2 start --only cherryshoe-dev
HOST=cherryshoe-dev.com pm2 start --only cherryshoe

The main differences between the original and the new way are:
1.  npm is now started with the ecosystem file with the "script" and "args" attributes in the ecosystem.config.js file.  Since the npm script startdev is custom, it needs to be prepended with "run" or "run-script"
2.  --name from the old command is now named within the ecosystem file
3.  to start a specific app vs starting all apps defined in the ecosystem.config.js file, the new start command now only has to add the "--only <app-name>" attribute
4.  NODE_ENV from the old command is now configured with the ecosystem file
5. Decided to have multiple apps and to call them by --only <app-name> to start, which can call different npm scripts to start for different environments

Additional notes:
1.  Additional app configs could be added to the ecosystem.config.js file for additional environments
2.  Don't need to specify path to ecosystem file if it's in the current directory as where you are running
3.  We could also make this even more generic and have the ecosystem.config.js be controlled by a tool like ansible to replace variables as needed for each environment, so we wouldn't have to have an app for development and a separate one for production.

These articles helped a lot:

• https://stackoverflow.com/questions/31579509/can-pm2-run-an-npm-start-script
• https://medium.freecodecamp.org/introduction-to-npm-scripts-1dbb2ae01633
• https://pm2.io/doc/en/runtime/guide/log-management/?utm_source=pm2&utm_medium=website&utm_campaign=rebranding#log-files
• https://pm2.io/doc/en/runtime/guide/ecosystem-file/?utm_source=pm2&utm_medium=website&utm_campaign=rebranding#generate-a-template
• https://pm2.io/doc/en/runtime/reference/ecosystem-file/
• https://pm2.io/doc/en/runtime/guide/ecosystem-file/
• https://medium.freecodecamp.org/introduction-to-npm-scripts-1dbb2ae01633
• https://stackoverflow.com/questions/31579509/can-pm2-run-an-npm-start-script
• http://pm2.keymetrics.io/docs/usage/application-declaration/#generate-configuration (stay on the old documentation)

Elastic Stack Multiple Index Search Examples

I am working with Elastic Stack (elasticsearch, logstash, and kibana) for a report where data needed to be joined with two indexes, where the level_X_id to level_Y_id could be matched upon.  The level_X_id to level_Y_id attributes exist in both indexes.  NOTE: You can have multiple documents in each index that have level_X_id to level_Y_id, not just one document that matches with an exact match.

Environment:
Elasticsearch 5.0.2
Logstash 5.0.2
Kibana 5.0.2

Here are multiple ways to do that:
1.  Use multiple indexes in your _search API.  This returns all documents where field name(s) match in both indexes.

GET /cherryshoe_primary_idx,cherryshoe_secondary_idx/_search
{
  "query": {
    "bool": {
      "must": {
        "query_string": {
          "analyze_wildcard": false,
          "query": "level_1_id:7268 AND level_2_id:7292"
        }
      }
    }
  }
}

2. Using Terms Query to specify primary index to match on, only returning the secondary index’s records.  This returns all documents where field name(s) match in both indexes.  The Elasticsearch API for Terms Query was not very clear, it took a while for me to get the query to work so I will explain it in detail below:

• Retrieves cherryshoe_secondary_idx documents, where both cherryshoe_primary_idx and cherryshoe_secondary_idx documents have matching "level_1_id" value of "3629".  
• query.terms.level_1_id json attribute refers to the cherryshoe_secondary_idx index attribute.
• query.terms.level_1_id.path value of level_1_id refers to the query.terms.level_1_id.index document json structure "_source.level_1_id".  You can see this in Kibana -> Discover -> cherryshoe_primary_idx.  Expand one of the results -> and instead of the "Table" view look at the "JSON" view.  You'll notice the "_source" JSON object holds all the index attributes.
• query.terms.level_1_id.type json attribute refers to the document json structure "_type".  You can see this in Kibana -> Discover -> cherryshoe_primary_idx.  Expand one of the results -> and instead of the "Table" view look at the "JSON" view.  You'll notice the "_type" JSON attribute has value "logs".

Single "terms":

GET cherryshoe_secondary_idx/_search
{
    "query" : {
        "terms" : {
            "level_1_id" : {
                "index" : "cherryshoe_primary_idx",
                "type" : "logs",
                "id" : 3629,
                "path" : "level_1_id"
            }
        }
    }
}

I thought I could immediately put multiple Terms in the query, to add additional attributes, but you can't have multiple Terms be defined and return the results you expect.  For example, the below runs with valid syntax, but doesn't return any data.  I haven't been able to find documentation to say that you cannot have multiple Terms in a query work.  Interesting because you can also have one Term.

Multiple "terms":

GET cherryshoe_secondary_idx/_search
{
    "query" : {
   "bool": {
    "must": 
    [{
     "terms" : {
      "level_1_id" : {
       "index" : "cherryshoe_primary_idx",
       "type" : "logs",
       "id" : 3629,
       "path" : "level_1_id"
      }
     }
    },
    {
     "terms" : {
      "level_2_id" : {
       "index" : "cherryshoe_primary_idx",
       "type" : "logs",
       "id" : 3719,
       "path" : "level_2_id"
      }
     }
    }]
   }
  }
}

3. Using the multi-search template, which allows you to execute several search template requests within the same API.  It returns records from either index depending on the query criteria that you want from each respective index.  NOTE:  each "index" and "query" json should not span multiple lines.

POST /_msearch
{"index": "cherryshoe_primary_idx" }
{"query":{"bool":{"must":{"query_string":{"analyze_wildcard":false,"query":"level_1_id:3629 AND tier_2_fa_id:level_2_id"}}}}}
{"index": "cherryshoe_secondary_idx" }
{"query":{"prefix":{"level_id":"_3629_3719_"}}}

Elasticsearch nuances - default field text length and lucene tokenizer

The web application I work on has a reporting module where data is ETLed with Logstash and stored in Elasticsearch. There is a reporting module where you can specify multiple filters, i.e state filter, program filter, etc.

Environment:
Elasticsearch 5.0.2
Windows 10

There were two issues that came up in recent months:

1. Looking at all field mappings for a particular index you can see that fields with type "text" has a max value of 256, defined by "ignore_above": 256.  This is the default setting of "text" fields.  Performing the following GET to retrieve index field mappings -

• curl http://localhost:9200/{index_name} 
• i.e. curl http://localhost:9200/cherryshoe_idx 

returns a JSON that looks something like -

{
  "cherryshoe_idx": {
    "aliases": {},
    "mappings": {
      "logs": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "text_data_that_can_be_very_long": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "unique_id": {
            "type": "long"
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1546610232085",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "cC1mdfLfSi68sZe6r-QNLA",
        "version": {
          "created": "5000299"
        },
        "provided_name": "cherryshoe_idx"
      }
    }
  }
}

PROBLEM and SOLUTION:
One of the filters was using the "text_data_that_can_be_very_long" field to filter on; sometimes the value was being cut off because of the length restriction.  Because of this, an additional field was added for the "id" value of the filter (text_data_that_can_be_very_long_id), the query was updated to use the "id" field of this value to filter instead, and the "ignore_above": 256 restriction was removed for "text_data_that_can_be_very_long" for data display purposes.

Updated field mapping json snippet:

  "text_data_that_can_be_very_long": {
    "type": "text",
    "fields": {
      "keyword": {
        "type": "keyword"
      }
    }
  },
  "text_data_that_can_be_very_long_id": {
    "type": "long"
  }

2. As I mentioned above, the report can specify multiple filters, one of them the state filter -

JPA Json Serialization/Deserialization to/from MySQL

I'm using spring boot with JPA and a MySQL backend database with several JSON columns.  The following is the JSON converter class used to save and retrieve data to the JSON column.  This article helped a lot.

This worked with:
Windows 7
Oracle Java 1.8.0_66
MySql 5.7.10.0
Spring Boot 1.3.1.RELEASE

Steps:

1. Create a POJO to represent the JSON object that will be saved to the column.
   

   package com.cherryshoe.model;
   
   /*
    * Represents a "documents" database column that holds json document data
    */
   public class JsonDocuments {
       private String docId;
       private String docName;
   
       public String getDocId() {
           return docId;
       }
   
       public void setDocId(String docId) {
           this.docId = docId;
       }
   
       public String getDocName() {
           return docName;
       }
   
       public void setDocName(String docName) {
           this.docName = docName;
       }
   
       public JsonDocuments() {
           super();
       }
   
       public JsonDocuments(String docId, String docName) {
           super();
           this.docId = docId;
           this.docName = docName;
       }
   
       @Override
        public int hashCode() {
         ...
        }
   
       @Override
        public boolean equals(Object obj) {
         ...
        }
   
       @Override
       public String toString() {
           return "JsonDocuments [docId=" + docId + ", docName=" + docName + "]";
       }
   
   }

2. Create a Converter Class that implements AttributerConverter.
   
   
   • Annotate the class with JPA converter annotation
   • I need to save a list of documents so the type argument to AttributeConverter is List<JsonDocuments> vs JsonDocuments
   • We are using the Jackson JSON library to serialize/deserialize POJO's to JSON and vice versa.  ObjectMapper is thread safe so it is declared as a static variable in the JpaJsonDocumentsConverter class.
   • Override convertToDatabaseColumn and convertToEntityAttribute methods with objectMapper writeValue and readValue calls

   package com.cherryshoe.utils;
   
   import java.io.IOException;
   import java.util.ArrayList;
   import java.util.Arrays;
   import java.util.List;
   
   import javax.persistence.AttributeConverter;
   import javax.persistence.Converter;
   
   import org.slf4j.Logger;
   import org.slf4j.LoggerFactory;
   import org.cherryshoe.model.JsonDocuments;
   
   import com.fasterxml.jackson.core.JsonProcessingException;
   import com.fasterxml.jackson.databind.ObjectMapper;
   
   @Converter
   public class JpaJsonDocumentsConverter implements
           AttributeConverter<List<JsonDocuments>, String> {
   
       // ObjectMapper is thread safe
       private final static ObjectMapper objectMapper = new ObjectMapper();
   
       private Logger log = LoggerFactory.getLogger(getClass());
   
       @Override
       public String convertToDatabaseColumn(List<JsonDocuments> meta) {
           String jsonString = "";
           try {
               log.debug("Start convertToDatabaseColumn");
   
               // convert list of POJO to json
               jsonString = objectMapper.writeValueAsString(meta);
               log.debug("convertToDatabaseColumn" + jsonString);
   
           } catch (JsonProcessingException ex) {
               log.error(ex.getMessage());
           }
           return jsonString;
       }
   
       @Override
       public List<JsonDocuments> convertToEntityAttribute(String dbData) {
           List<JsonDocuments> list = new ArrayList<JsonDocuments>();
           try {
               log.debug("Start convertToEntityAttribute");
   
               // convert json to list of POJO
               list = Arrays.asList(objectMapper.readValue(dbData,
                       JsonDocuments[].class));
               log.debug("JsonDocumentsConverter.convertToDatabaseColumn" + list);
   
           } catch (IOException ex) {
               log.error(ex.getMessage());
           }
           return list;
       }
   }
   

3. The JPA Entity class that contains the JSON column needs to have the @Convert annotation for the documents column.  NOTE:  You could also have in the JpaJsonDocumentsConverter class added the autoapply attribute to the @Converter annotation and set that to true to have JPA apply this converter to all entity attributes of type List<JsonDocuments>.
   

   @Entity
   @Table(name = "TABLE_WITH_DOCUMENTS")
   public class TableWithDocuments implements Serializable
   {
   
        /**
        *
        */
        private static final long serialVersionUID = 3781459465416706159L;
       
        ... entity attributes
       
        @Convert(converter = JpaJsonDocumentsConverter.class)
        private List<JsonDocuments> documents;
       
        ... entity attributes
   
    }



4. Verified with an integration test on the Spring Data JPA Repository class that uses this entity that these methods worked.  I did notice that the documents column could not be null, so when you create the new TableWithDocuments record, the documents column had to be set as an empty JsonArray if there were no documents.  Adding in null checks in the JpaJsonDocumentsConverter  convertToDatabaseColumn and convertToEntityAttribute  did not help, it was inside the JPA code that would error out.

Spring 3.2 @ControllerAdvice to handle Controller Exceptions

My Java/Spring web application has controllers that either return ModelAndView to a jsp page or return json for ajax calls.  Exceptions can occur at any time, and I noticed that on ajax success the data returned was json, but on ajax error the data returned was regular text; all ajax calls should return the same kind of data, namely json.  There had to be a way to solve this problem generically!

Spring 3.2 to the rescue - it introduced a new annotation called @ControllerAdvice (http://docs.spring.io/spring/docs/3.2.x/spring-framework-reference/html/new-in-3.2.html#new-in-3.2-webmvc-controller-advice)  that defines methods that applies to all @RequestMapping rest url methods, and in particular to help with exceptions with @ExceptionHandler.  We can use this to have a generic exception handling solution to have all ajax calls return json, and all ModelAndView return html.

• @ControllerAdvice allows you to define ONE controller class to handle ALL exceptions that could occur (no specific exceptions defined needed)
• You can control the HTTP status code returned
• You can control the json message returned by grabbing the exception message from the specific exception that occurred

Here's how you do it:

1. Your controllers won't change, the code will still continue to throw any number of specific exceptions:

@Controller
public class ExceptionController {
    
    @RequestMapping(value = "/randomException", method = RequestMethod.GET)
    public String randomException(Authentication auth, HttpServletRequest request) throws Exception {    
       
        throw new NumberFormatException(" " +
                "Test ControllerAdvice randomException[" + "NumberFormatException" + "]");

        [...]
    }

    @RequestMapping(value = "/mavException", method = RequestMethod.GET)   
    public ModelAndView modelAndViewException(Authentication auth, HttpServletRequest request) throws Exception {  
     
        throw new UnexpectedRollbackException("Test ControllerAdvice Exception for mav");
        [...]
    }
}

2. Write ONE new Controller, this will act as your ControllerAdvice controller. Notice:
• the class is annotated with @ControllerAdvice
• @ExceptionHandler is annotated above your generic "handleException" function
• handleException function takes generic Exception e as a parameter
• it checks the accept header to see how the controller function was invoked (via ajax or via form submit; randomException or mavException, respectively).  *Update*  We have to check for null first in case the request doesn't specify a response type expected.  If the request doesn't specify it, then simply return text/html.
• It returns json or html

@ControllerAdvice
public class CherryShoeControllerAdvice {

    /*
     * Handles JSON and HTML
     */
    @ExceptionHandler
    @ResponseBody
    @ResponseStatus(HttpStatus.BAD_REQUEST)
    public String handleException(HttpServletRequest request, HttpServletResponse response, Exception e) throws IOException {    
        String acceptHeader = request.getHeader("Accept");
       
        // If Accept header exists, check if it expects a response of type json, otherwise just return text/html
        // Use apache commons lang3 to escape json values
        if(acceptHeader.contains("application/json")) {
            // return as JSON
            String jsonString = 
                    "{\"success\": false, \"message\": \"" + StringEscapeUtils.escapeJson(e.getMessage()) + "\" }";
        
            System.out.println("In handleGeneric" + e.getMessage());
            return jsonString;
        } else {
            //return as HTML
            response.setContentType("text/html");
            return response.toString();
        }
    }

}

BTW - if you wanted to specify each type of exception going to a separate @ExceptionHandler you could.

BTW - The main cons for using pre-Spring 3.2 @ExceptionHandler (http://docs.spring.io/spring/docs/3.1.x/spring-framework-reference/html/mvc.html#mvc-exceptionhandlers) by itself are you have to:

• Define an ExceptionHandler for EACH controller (or have each controller inherit from one common base)
• Define EACH exception type to be handled – Or have each controller throw the specific type of exception expected