Wednesday, August 14, 2019

Immutable.js examples with Map and List, fromJS, getIn, and get

I stumbled across Immutable.js in the code base today.  Here's a run down of how to set up an Immutable map/list, create a Immutable map/list using Immutable.fromJS() and how to use the getIn/get functions.

Environment:
immutable 3.8.2
assert 1.4.1
jest 20.0.3

describe('Immutable Map / List and fromJS Examples', function() {
  it('Immutable Map / List Example', function() {

    const immutableObj = Immutable.Map({
      property1: 'property1',
      property2: Immutable.List(['Cherry']),
      property3: 'property3',
      propertyMap: Immutable.Map({
        propertyMapProperty: 'Shoe',
      }),
    });
    let result = immutableObj.getIn(['propertyMap', 'propertyMapProperty'])
    let expectedResult = "Shoe";

    assert.equal(expectedResult, result);

    let immutableList = immutableObj.get('property2');
    result = immutableList.get(0);
    expectedResult = "Cherry";
    assert.equal(expectedResult, result);
  });

  it('Immutable fromJS Example', function() {

    const immutableObj = Immutable.fromJS({
      property1: 'property1',
      property2: ['Cherry'],
      property3: 'property3',
      propertyMap: {propertyMapProperty: "Shoe"},
    });

    let result = immutableObj.getIn(['propertyMap', 'propertyMapProperty'])
    let expectedResult = "Shoe";

    assert.equal(expectedResult, result);

    let immutableList = immutableObj.get('property2');
    result = immutableList.get(0);
    expectedResult = "Cherry";
    assert.equal(expectedResult, result);
  });
});


This article helped a lot.

Friday, August 9, 2019

Ansible - accessing .bashrc changes in subsequent task

My project uses ansible for local development setup, and I've found several examples where I needed to make updates to the .bashrc file:  updating the $PATH environment variable and creating a new environment variable for the VM's bridged IP address.  Accessing the changes in ~/.bashrc works fine when you logout and login again (or source it in the current shell) for anything done after the ansible playbook is run.  But, there's been a nuance I've encountered several times the past couple of months that I am sharing here to more easily remember the reasoning the next time it happens!

Environment: 
CentOS Linux release 7.3.1611 (Core)
ansible 2.8.2

After making updates to the ~/.bashrc file with an ansible task, the updates to the ~/.bashrc file is NOT available by default when performing the next ansible task.

For example, I needed to make updates to the ~/.bashrc file to update the $PATH environment variable to include another location where the yarn dependency was installed.  Another, I needed to export a new environment variable to more easily access the VM's bridged IP address.

To allow this to work as the intended user (vs root) I needed to switch user with su, but I also needed to have the command run inside a new login shell with an environment similar to a real login (the -l option for su).  Otherwise it wouldn't re-run the .bashrc file to get access to the just made changes in the ansible task that had just occurred.

NOTE:  {{user}} and {{project_path}} is set up in ansible's group_vars inventory.

- block:
  - name: export bridged ip address in .bashrc
    blockinfile:
      dest: '/home/{{user}}/.bashrc'
      block: |
       export BRIDGED_IP_ADDRESS=$(hostname -I | awk '{print $1}')
      marker: '# {mark} ANSIBLE MANAGED BLOCK - export BRIDGED_IP_ADDRESS'
      owner: '{{user}}'
      group: 'wheel'
      create: yes
  - name: command that needs access to $BRIDGED_IP_ADDRESS
    command: '<command>'
    args:
      chdir: '{{project_path}}'
    # The -l indicates to start shell as login shell with an environment similar to a real login
    become_method: su
    become_user: '{{user}}'
    become_flags: '-l'

This article helped a lot.

Friday, July 26, 2019

Culmination of hodgepodge realizations of WebSockets (Circa July 2019)

I've read many articles about WebSockets the past month, and each article gives one to several tidbits of information, but not enough to answer all my questions.

Environment:
CentOS Linux release 7.3.1611 (Core)
node v8.16.0
nginx version: nginx/1.12.2

Here's the culmination of what I've learned so far.  I am using Node.js with Express, "ws" for the Node WebSocket library, and native WebSocket javascript for the front-end.

  1. So many articles to read!  Some of the early articles I read taught how to code WebSocket connections, frames, listeners, etc yourself!  This is good reference but definitely not ideal to actually do! There's many libraries out there that can do this for you  I eventually chose "ws" as I discussed in my last post.
    • The only con I see right now it doesn't have a fallback method to long-polling like some libraries available (i.e. socket.io).   This doesn't concern me because WebSockets are supported by all major browsers - please reference number 7 below for more thoughts.
  2. Not explicitly clear if you are supposed to keep client connections open and re-use them as needed (or open a new one for each request).  
    • The typical use case for keeping client connection open and re-use them is for a chat type of application.
    • My use case (where I am opening a WebSocket connection each request) is that "occasionally" there will be requests that take "longer than usual", and where it used to be HTTP there would be a HTTP response timeout on the web server level (set at 5 mins).  To not have to keep increasing this timeout, a different solution was needed - polling, WebSockets, etc.  WebSockets were chosen for several reasons which I won't go into here.
  3. The Node backend is clustered with one instance per CPU, and a WebSocket Server is running alongside each Node Server instance, so it can handle many WebSocket client connections concurrently.  If you want a large number multiple concurrent WebSocket connections concurrently, take a look at this article.
  4. The WebSocket protocol discusses the concept of ping/pong messages for the server to know each client is still connected, as well as conversely each client connection can still communicate with the server (i.e. Wifi connection terminated unexpectedly):
    • If the WebSocket Server sends a ping or pong, then the client should have to be able to recognize a ping/pong was sent, and respond, but there is no client side javascript code that can send or recognize/receive ping/pong frames, it's either supported by the browser or not.  
      • As referenced from this article - “ping/pong frames” are used to check the connection, sent from the server, the browser responds to these automatically. 
      • This article was helpful in teaching me that "Unfortunately, the Websocket protocol does not provide a similar method for the client, to find out if it is still connected to the server. This can happen, if the connection simply disappears without further notification. In order to have the client recognize this, some Javascript code has to be added to the client code responsible for the Websocket".
      • Since my application controls both client and server code, adding in a specific header/metadata in a web socket message with `WebSocket#send()` vs using `WebSocket#ping()` can be used to implement a heartbeat. 
        • This article - helped me build a heartbeat system with messages.
      • During server development testing, I noticed that the heartbeat message was sometimes sent to the client and sometimes not:
        • Noticed that if paused in debugger, the setInterval function used for sending a heartbeat message from server -> client wasn't working, since the worker that is paused (so the heartbeat from server -> client is NOT sent).
        • If setTimeout(sleep) function was enabled (simulating actual work being perform), the setInterval check DOES RUN for the worker (so the heartbeat from server -> client is sent).
  5. Web Server configuration needs configuration to:
  6. This repo had a great explanation of all the various WebSocket libraries out there.
  7. WebSockets are supported by all the major browser,  but Internet Explorer (11) still has a default maximum number of client connections at 6.
  8. Easiest to understand guide were the WebSockets API documentation by mozilla.org.

Monday, June 24, 2019

Common HTTP Status Code Errors when getting started with WebSockets

I'm working on an application where the frontend is React and the backend is Node running on CentOS 7.  I am introducing WebSockets for some data requests that can be long-running queries.  I haven't used WebSockets in the past so wanted to share two HTTP error statuses I repeatedly got when starting out.  There are multiple reasons why I chose the Node "ws" library for WebSocket communication:

  • Actively maintained, roughly new version per month
  • Implements both WebSocket protocol versions 8 and 13 for Node
  • Straightforward to use, and easy to use with the Express Node web framework.  The server is used for both http and WebSockets communication
  • Over 13 million weekly downloads (most I found from any of the WebSocket libraries) on npmjs.com
  • Handles WebSocket frames for you

Environment:
CentOS Linux release 7.3.1611 (Core)
node v8.16.0
nginx version: nginx/1.12.2

Error 1 - HTTP Status Code 404
The first three browsers gave at least an HTTP Response error code, Firefox didn't provide much info on the error message.

  1. In IE: SCRIPT12008: WebSocket Error: Incorrect HTTP response. Status code 404, Not Found
  2. In Edge: SCRIPT12008: SCRIPT12008: WebSocket Error: Incorrect HTTP response. Status code 404, Not Found
  3. In Chrome: WebSocket connection to 'wss://cherryshoe.com/wsapi' failed: Error during WebSocket handshake: Unexpected response code: 404
  4. In Firefox:
    • The connection to wss://cherryshoe.com/sockjs-node/314/jv0jnivp/websocket was interrupted while the page was loading.
    • Firefox can’t establish a connection to the server at wss://cherryshoe.com/wsapi/.

Problem and Solution:  The nginx web server was not configured for WebSocket support, had to add the following to the /wsapi location
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

Error 2 - HTTP Status code 502

The first three browsers gave at least an HTTP Response error code, Firefox didn't provide much info on the error message.

  1. In IE: SCRIPT12008: WebSocket Error: Incorrect HTTP response. Status code 502, Bad Gateway
  2. In Edge: SCRIPT12008: SCRIPT12008: WebSocket Error: Incorrect HTTP response. Status code 502, Bad Gateway
  3. In Chrome: WebSocket connection to 'wss://cherryshoe.com/wsapi/' failed: Error during WebSocket handshake: Unexpected response code: 502
  4. In Firefox:
    • The connection to wss://cherryshoe.com/sockjs-node/957/mxbslcja/websocket was interrupted while the page was loading.
    • Firefox can’t establish a connection to the server at wss://cherryshoe.com/wsapi/.
Problem and Solution 2:
I would get multiple successful client connections (one for Firefox, two or three for Chrome/IE/Edge), but then the next connection consistently received Status Code 502.  This error essentially kills any communication with the WebSocket Server itself (not just that specific client connection), which would require a restart of the Node server.

The problem was that in close event handler ws.on('close') I was calling close on the WebSocketServer itself (wss.close()).  I didn't actually want to close the WebSocketServer itself, only wanted to close the connection, but in the close event handler ws.on('close') the connection is already closed!

The client code message event handler onmessage would close the socket after it received a 'Done' message, which caused the WebSocket Server ws.on('close') to be called.
    let socket;

    ...
    socket setup code
    ...

    // this is over https, so have to make it web socket over https "wss:"
    socket = new WebSocket('wss://cherryshoe.com/wsapi/');

    // message received from web socket server
    socket.onmessage = function(evt) {

      // If data request is 'Done'
      socket.close([1000], "Work complete");
    };


The server code web socket close event handler was closing the WebSocket server:

var WebSocketServer = require('ws').Server;

...
code to setup server
...


var wss = new WebSocketServer({server: server});
wss.on('connection', function (ws, socket) {
    console.info("websocket connection open");

    // send status back to client
    ws.send('Connection established');

    // Handle when message is sent from client
    ws.on('message', async (requestData) => {
        // process the request code here
        ws.send('Done');
    });

    ws.on('error', function (ws, code, reason) {
      console.info("websocket connection error");
    });

    ws.on('close', function (ws, code, reason) {
      console.info("websocket connection close");
      wss.close();
    });
});


The solution was to remove the wss.close from the ws.on('close') code:

wss.on('connection', function (ws, socket) { 

   ...

   ws.on('close', function (ws, code, reason) {
      // connection is closed, but you don't actually want to close down the
      // WebSocketServer so do not do wss.close()!
      console.info("websocket connection close");
   });
   
   ...

});

Sunday, May 19, 2019

Enabling PM2 logging configuration for a Node.js application

I'm working on an application where PM2 manages the node processes that are started with npm.  The default PM2 log configuration was being used, so my task was to configure PM2 logs for added functionality such as:  log location, log rotation, timestamps in logs, etc.  This article will only be talking about how to configure a base ecosystem.config.js file for the PM2 log configuration to be recognized.

Environment:
CentOS Linux release 7.3.1611 (Core)
pm2 version 3.2.3
npm version 6.4.1
node version v8.16.0

Originally, PM2 was managing the node processes with npm start scripts with the following commands:
HOST=cherryshoe-dev.com pm2 start npm --name cherryshoe-dev -- run startdev
HOST=cherryshoe.com NODE_ENV=production pm2 start npm --name cherryshoe -- start

The --name "cherryshoe-dev" or "cherryshoe" is the name of the PM2 app name that is started.
The "-- run startdev" and the "-- start" were indicating to call the npm script located in package.json.  i.e. scripts.startdev or scripts.start in the below json snippet.

package.json looked like:
  "scripts": {
    "start": "npm run server",
    "startdev": "concurrently \"npm run server\" \"npm run client\"",
    "server": "cross-env node --max-old-space-size=8192 ./bin/cherryshoeServer.js",
    "client": "node cherryshoe-client.js"
  }

To enable PM2 logging configuration, the ecosystem file had to be configured to call the custom npm scripts appropriately, depending on the environment.

// If you look at a default ecosystem.config.js, you'll see that
// an assumption is made that each npm start command is the same.  The app has
// different npm custom scripts defined in package.json to run depending
// on the environment - so because of this, decided to have multiple apps
// and to call them by "pm2 start --only <app-name>" to start.
module.exports = {
  apps : [
    // Local DEV app config
    {
      name: 'cherryshoe-dev',
      script: 'npm',
      // call custom start dev npm script from package.json
      args: 'run startdev',

      env: {
        NODE_ENV: 'development'
      }
    },
    // PROD app config
    {
      name: 'cherryshoe',
      script: 'npm',
      args: 'start',

      env: {
        NODE_ENV: 'production'
      }
    }
  ],
};

After this change, the command to have PM2 manage the start of the node process with npm became easier:
HOST=cherryshoe-dev.com pm2 start --only cherryshoe-dev
HOST=cherryshoe-dev.com pm2 start --only cherryshoe

The main differences between the original and the new way are:
1.  npm is now started with the ecosystem file with the "script" and "args" attributes in the ecosystem.config.js file.  Since the npm script startdev is custom, it needs to be prepended with "run" or "run-script"
2.  --name from the old command is now named within the ecosystem file
3.  to start a specific app vs starting all apps defined in the ecosystem.config.js file, the new start command now only has to add the "--only <app-name>" attribute
4.  NODE_ENV from the old command is now configured with the ecosystem file
5. Decided to have multiple apps and to call them by --only <app-name> to start, which can call different npm scripts to start for different environments

Additional notes:
1.  Additional app configs could be added to the ecosystem.config.js file for additional environments
2.  Don't need to specify path to ecosystem file if it's in the current directory as where you are running
3.  We could also make this even more generic and have the ecosystem.config.js be controlled by a tool like ansible to replace variables as needed for each environment, so we wouldn't have to have an app for development and a separate one for production.

These articles helped a lot:

Saturday, April 20, 2019

Changing Data Directory for MySQL on CentOS7

I was recently asked to move the DEV and TEST environment of the application I am working on to  new servers.  The old servers were running CentOS6, new servers running on CentOS7 with SELinux enabled.  I realized after the fact that the server that is hosting MySQL had a separate /data volume that had much disk space on it, so I needed to change the data directory after several databases had already been instantiated.

Environment:
CentOS Linux release 7.6.1810 (Core) with SELinux enabled with enforcing
Server version: 5.7.25 MySQL Community Server (GPL)

1.  Login as a user that has root privileges, sudo to root

2.  Verify the current data directory
mysql -u root -p <enter in the root password>

mysql> select @@datadir;
+-----------------+
| @@datadir       |
+-----------------+
| /var/lib/mysql/ |
+-----------------+
1 row in set (0.01 sec)

3. Stop mysqld and verify it is stopped
systemctl stop mysqld
systemctl status mysqld

Apr 19 08:39:51 <servername> systemd[1]: Stopping MySQL Server...
Apr 19 08:39:55 <servername> systemd[1]: Stopped MySQL Server.

4.  Check if you have cp with the -a option
man cp

       -a, --archive
              same as -dR --preserve=all
       -d     same as --no-dereference --preserve=links
       --preserve[=ATTR_LIST]
              preserve the specified attributes (default: mode,ownership,timestamps), if possible  additional  attributes:  context, links, xattr, all
       -R, -r, --recursive
              copy directories recursively

Copies a directory exactly as it is (preserves ownership and groups), the files retain all their attributes, and symlinks are not dereferenced (-d).

5.  The new folder/volume I want to copy to is /data..  Copy the files from the source /var/lib/mysql to /data with -a option
cp -a /var/lib/mysql /data

6. Rename the current folder /var/lib/mysql to a different name to avoid confusion
mv /var/lib/mysql /var/lib/mysql-OLD

7.  Take a backup of the my.cnf file
cp /etc/my.cnf ~/my.cnf.ORIG

8.  Configure MySQL data directory to new folder location, add in port=3306, and configure datadir and socket to the new location. Also add a [client] group to the bottom of the file (after every options in the [mysqld] group) with port and socket matching the [mysqld] group.

vi /etc/my.cnf

[mysqld]
port=3306
datadir=/data/mysql
socket=/data/mysql/mysql.sock

[client]
port=3306
socket=/data/mysql/mysql.sock

9.  Add SELinux security context to the new folder.  semanageutility is not installed by default and was missing, so installed policycoreutils-python.  Perform a listing with security context.

yum -y install policycoreutils-python
semanage fcontext -a -t mysqld_db_t "/data(/.*)?"
restorecon -Rv /data

ls -lZ /data/mysql/
ls -lZ /var/lib/mysql-OLD

NOTE:  If you don't perform this step, you will see the following warnings/errors in the /var/log/mysqld.log file.

2019-04-19T13:31:39.698773Z 0 [Warning] Can't create test file /data/mysql/<servername>.lower-test
2019-04-19T13:31:39.837948Z 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.

9. Restart mysql
systemctl start mysqld
systemctl status mysqld

10.  Verify the new data directory
mysql -u root -p <enter in the root password>

mysql> select @@datadir;
+-----------------+
| @@datadir       |
+-----------------+
| /data/mysql/ |
+-----------------+
1 row in set (0.01 sec)

These articles helped a lot:

Friday, March 22, 2019

Elastic Stack Multiple Index Search Examples

I am working with Elastic Stack (elasticsearch, logstash, and kibana) for a report where data needed to be joined with two indexes, where the level_X_id to level_Y_id could be matched upon.  The level_X_id to level_Y_id attributes exist in both indexes.  NOTE: You can have multiple documents in each index that have level_X_id to level_Y_id, not just one document that matches with an exact match.

Environment:
Elasticsearch 5.0.2
Logstash 5.0.2
Kibana 5.0.2

Here are multiple ways to do that:
1.  Use multiple indexes in your _search API.  This returns all documents where field name(s) match in both indexes.

GET /cherryshoe_primary_idx,cherryshoe_secondary_idx/_search
{
  "query": {
    "bool": {
      "must": {
        "query_string": {
          "analyze_wildcard": false,
          "query": "level_1_id:7268 AND level_2_id:7292"
        }
      }
    }
  }
}

2. Using Terms Query to specify primary index to match on, only returning the secondary index’s records.  This returns all documents where field name(s) match in both indexes.  The Elasticsearch API for Terms Query was not very clear, it took a while for me to get the query to work so I will explain it in detail below:
  • Retrieves cherryshoe_secondary_idx documents, where both cherryshoe_primary_idx and cherryshoe_secondary_idx documents have matching "level_1_id" value of "3629".  
  • query.terms.level_1_id json attribute refers to the cherryshoe_secondary_idx index attribute.
  • query.terms.level_1_id.path value of level_1_id refers to the query.terms.level_1_id.index document json structure "_source.level_1_id".  You can see this in Kibana -> Discover -> cherryshoe_primary_idx.  Expand one of the results -> and instead of the "Table" view look at the "JSON" view.  You'll notice the "_source" JSON object holds all the index attributes.
  • query.terms.level_1_id.type json attribute refers to the document json structure "_type".  You can see this in Kibana -> Discover -> cherryshoe_primary_idx.  Expand one of the results -> and instead of the "Table" view look at the "JSON" view.  You'll notice the "_type" JSON attribute has value "logs".

Single "terms":
GET cherryshoe_secondary_idx/_search
{
    "query" : {
        "terms" : {
            "level_1_id" : {
                "index" : "cherryshoe_primary_idx",
                "type" : "logs",
                "id" : 3629,
                "path" : "level_1_id"
            }
        }
    }
}

I thought I could immediately put multiple Terms in the query, to add additional attributes, but you can't have multiple Terms be defined and return the results you expect.  For example, the below runs with valid syntax, but doesn't return any data.  I haven't been able to find documentation to say that you cannot have multiple Terms in a query work.  Interesting because you can also have one Term.

Multiple "terms":
GET cherryshoe_secondary_idx/_search
{
    "query" : {
   "bool": {
    "must": 
    [{
     "terms" : {
      "level_1_id" : {
       "index" : "cherryshoe_primary_idx",
       "type" : "logs",
       "id" : 3629,
       "path" : "level_1_id"
      }
     }
    },
    {
     "terms" : {
      "level_2_id" : {
       "index" : "cherryshoe_primary_idx",
       "type" : "logs",
       "id" : 3719,
       "path" : "level_2_id"
      }
     }
    }]
   }
  }
}

3. Using the multi-search template, which allows you to execute several search template requests within the same API.  It returns records from either index depending on the query criteria that you want from each respective index.  NOTE:  each "index" and "query" json should not span multiple lines.

POST /_msearch
{"index": "cherryshoe_primary_idx" }
{"query":{"bool":{"must":{"query_string":{"analyze_wildcard":false,"query":"level_1_id:3629 AND tier_2_fa_id:level_2_id"}}}}}
{"index": "cherryshoe_secondary_idx" }
{"query":{"prefix":{"level_id":"_3629_3719_"}}}