How to debug a Node.js app running on a VM from local Windows

I'm working on an application where the frontend is React and the backend is Node running on CentOS 7.  Below I list steps on how to debug a PM2 managed clustered Node backend application with two different clustering methods:

1. Node cluster module is used to configure cluster processes
2. PM2 cluster mode is used to configure cluster processes

Environment:
Windows 10 Pro
CentOS Linux release 7.3.1611 (Core)
node v8.16.0

Node cluster module is used to configure cluster processes
NOTE: In this example PM2 starts the Node application via a npm script, and the backend code is spawning two cluster instances.

Update the command where you start node with the --inspect attribute. For us, we start node with the "startdev" npm script located in package.json. "startdev" uses the "server" script.

OLD

"scripts": {
  ..
  "startdev": "concurrently \"npm run server\" \"npm run client\"",
  "server": "cross-env node --max-old-space-size=8192 ./bin/cherryshoeServer.js",
  ..
}

NEW - You'll see that the "server" script is not changed as other npm scripts are also dependent on it. "startdev" uses a new "serverdev" script that was created to add the --inspect attribute.

"scripts": {
  ..
  "server": "cross-env node --max-old-space-size=8192 ./bin/cherryshoeServer.js",
  "startdev": "concurrently \"npm run serverdev\" \"npm run client\"",
  "serverdev": "cross-env node --max-old-space-size=8192 --inspect ./bin/cherryshoeServer.js",
  ..
}

A PM2 ecosystem configuration file is used; the config options of note are:

apps : [ {
  ..
  script: 'npm',
  // call appropriate npm script from package.json
  args: 'run startdev',
  ..
} ]

Start node with command "pm2 start", which calls the npm script "startdev", which calls npm script "serverdev", and runs cherryshoeServer.js on the local development environment.

Perform a ps command to verify the backend node processes are running. When cherryshoeServer.js is started, it spawns two worker processes on the local development environment (based on code to spawn two processes using Node cluster module).  Because of this, you'll see the first process below is the parent process with PID 5281, and the remaining two are worker processes with parent PPID 5281.

cs_admin  5281  5263  0 11:09 ?        00:00:00 node --max-old-space-
size=8192 --inspect ./bin/cherryshoeServer.js
cs_admin  5298  5281  0 11:09 ?        00:00:00 /usr/bin/node --max-old-
space-size=8192 --inspect --inspect-port=9230 /opt/cherryshoe/bin/cherryshoeServer.js
cs_admin  5303  5281  0 11:09 ?        00:00:00 /usr/bin/node --max-old-
space-size=8192 --inspect --inspect-port=9231 /opt/cherryshoe/bin/cherryshoeServer.js

Verify in the log file that debugger processes are listening. PM2 is used to manage logging, which is located in /var/log/cherryshoe/pm2/cs-out.log.  By default, the debugger listens on port 9229, then each additional debugger listener for each worker processeis incremented appropriately (9230 and 9231 respectively).

2020-01-26T11:09:22.657: [0] Debugger listening on ws://127.0.0.1:9229
/62dd92b9-a978-4cce-9e91-84b87835e014
2020-01-26T11:09:22.657: [0] For help see https://nodejs.org/en/docs
/inspector
2020-01-26T11:09:22.882: [1]
2020-01-26T11:09:22.882: [1] > origin-destination-client@0.2.0 start /opt
/cherryshoe/client
2020-01-26T11:09:22.882: [1] > concurrently "yarn watch-css" "cross-env
NODE_PATH=src/ react-scripts start"
2020-01-26T11:09:22.882: [1]
2020-01-26T11:09:22.965: [0] Running 2 processes
2020-01-26T11:09:22.975: [0] Debugger listening on ws://127.0.0.1:9230
/3e05b482-b186-4c7c-908d-7f5188353bb2
2020-01-26T11:09:22.975: [0] For help see https://nodejs.org/en/docs
/inspector
2020-01-26T11:09:22.978: [0] Debugger listening on ws://127.0.0.1:9231
/e3264361-6c0e-4843-8a4d-91b5ba9a8e4f
2020-01-26T11:09:22.978: [0] For help see https://nodejs.org/en/docs
/inspector

Back on your Windows local machine, open up your favorite app to ssh tunnel (I'm using git bash for this example but I am a big fan of MobaXterm) into the VM with appropriate ports to attach to the ports that the debuggers are listening on. This starts a ssh tunnel session where a connection to ports 8889-8891 (make sure these ports are not in use first) on your local machine will be forwarded to port 9229-9231 on the cherryshoe-dev.cherryshoe.com machine.  NOTES: I had to use 127.0.0.1 instead of localhost for this to work. Use a user account that has access to the VM. You may want to set up ssh passwordlessly so you don't have to enter passwords.

    ssh -L 8889:127.0.0.1:9229 cs_admin@cherryshoe-dev.cherryshoe.com
    ssh -L 8890:127.0.0.1:9230 cs_admin@cherryshoe-dev.cherryshoe.com
    ssh -L 8891:127.0.0.1:9231 cs_admin@cherryshoe-dev.cherryshoe.com

You can now attach a debugger client of choice to the X processes, as if the Node.js application was running locally. I will use Chrome DevTools as an example.

Open Chrome and enter "chrome://inspect" into the URL

Click "Discover network targets" Configure button and configure each of the ports to attach to:

• Enter "127.0.0.1:8889"
• Enter "127.0.0.1:8890"
• Enter "127.0.0.1:8891"

You should now see the 3 processes attached in the debugger client:

Click the "inspect" link for one of the processes, this will open up the DevTools for Node. Under "Sources" tab you can click "Ctrl-P" to open up a file of choice to debug that is attached to the process. You do NOT need to 'Add folder to workspace'.

Open up each remaining process by clicking the "inspect" link.

Invoke the Node application, i.e. call a REST endpoint that it responds to
One of the worker processes will process the request. Any breakpoints will be reached and any console logs will be printed.

You have to open a window for each worker process running because you don't know which process will get picked and if you don't have them all open you can miss it and think debugging isn't working!

If you restart the Node backend, the DevTools Targets will recognize the new process IDs, but the DevTools windows won't. Therefore, you need to open up the DevTools windows again for each process via the "inspect" link.

PM2 cluster mode is used to configure cluster processes
NOTE: It was discovered that PM2 cluster mode and starting node via npm do not play nicely. The first process would listen on port X, but each subsequent process would error out saying port X was in use. Because of this, the node application was changed to start directly by invoking the appropriate js script.

Using jest-when to support parameterized mock return values

The jest library by itself currently doesn't support parameterized mock return values for Node.js unit tests, but it can by integrating the jest-when library.

Environment:
CentOS Linux release 7.3.1611 (Core)
node v8.16.0
jest 24.8.0
jest-when version 2.7.0

The below jest unit test example uses jest-when to have two different parameterized mock return values for a method (db.runQuery) that is called twice but has different return values to simulate two different database calls.

const runDbQuery = async(sql, params) => {
  // this is the external dependency call that needs to be mocked
  const dataSet = await db.runQuery(sql, params);
  return dataSet;
};

const retrieveData = async(params) => {
  const data1 = await runDbQuery(QUERY1, [params]);
  const data2 = await runDbQuery(QUERY2, [data1]);

  // some manipulation of data...
  const returnData = <some manipulation of data>;
  return returnData;
};


Unit test for unit:
const { when } = require('jest-when')

// import external dependencies for mocking
const db = require('./db');

const unit = require(<unit under test js file>)(db);

test('Test retrieveData', async () => {

  ///////////////////
  // db call 1 setup
  ///////////////////
  // first db.runQuery mock setup
  const params1_var1 = 5;
  const params1_var2 = "five";

  const paramsJson1 = JSON.stringify({
    params1_var1: params1_var1,
    params1_var2: params1_var2,
  });

  const params1 = [paramsJson1];

  const returnData1_var1 = 99;
  const returnData1_var2 = "ninety-nine";
  const returnData1_var3 = true;

  const returnData1 = [
  {
    returnData1_var1: returnData1_var1,
    returnData1_var2: returnData1_var2,
    returnData1_var3: returnData1_var3,
  }

  ];

  // format returned from db call
  const dataSets1 = {
    rows: returnData1,
  };

  const query1 = QUERY1;

  ///////////////////
  // db call 2 setup
  ///////////////////
  // second db.runQuery mock setup
  const params2_var1 = 22;

  const params2 = [params2_var1];

  // county query is different
  const query2 = QUERY2;

  const returnData2_var1 = 100;
  const returnData2_var2 = "one-hundred";

  const returnData2 = [
  {
    returnData2_var1: returnData2_var1,
    returnData2_var2: returnData2_var2
  }

  // format returned from db call
  const dataSets2 = {
    rows: returnData2,
  };

  / external dependency method call that needs to be mocked
  const mockDbRunQuery = db.runQuery = jest.fn().mockName('mockDbRunQuery');

  // first call to db.runQuery
  when(db.runQuery).calledWith(query1, params1).mockResolvedValue(dataSets1);

  // second call to db.runQuery
  when(db.runQuery).calledWith(query2, params2).mockResolvedValue(dataSets2);

  const retrieveDataReturnVal = {
  ...
  };

  await expect(unit.retrieveData(params)).resolves.toStrictEqual(retrieveDataReturnVal);

  // verify that mock method(s) were expected to be called
  expect(mockDbRunQuery).toHaveBeenCalledTimes(2);
});

Example Jest test with an object instantiation inside method under test, then moved out into a utility class

Below are two examples of Node.js code and the corresponding unit tests for them. The first example includes an instantiation of an object inside it, the other breaks the instantiation of the object to a utility module.  Both are doable, but the latter way easier to unit test.

Environment:
CentOS Linux release 7.3.1611 (Core)
node v8.16.0
jest 24.8.0


Instantiating a new object inside method of test:
sampleOld.js

const cherryshoeAuthorization = require("cherry-shoe-auth");

const processValidRequest = async (requestType, jsonRequestData) => {
  const authService = new cherryshoeAuthorization.CherryshoeAuthorization("cherryshoe.com");
  const userData = await authService.getSessionData(jsonRequestData.token);
  const dbResponse = await requestTypeHandlers[requestType](jsonRequestData, userData.username);
};

sampleOld.test.js

const unit = require(./sampleOld);

// import external dependencies for mocking
const cherryshoeAuthorization = require('cherry-shoe-auth');

// mock cherry-shoe-auth
jest.genMockFromModule('cherry-shoe-auth');
jest.mock('cherry-shoe-auth');

test('Test processValidRequests', async () => {

  // mock external dependency method(s)
  const mockCherryshoeAuthorization = {
    getSessionData: jest.fn(),
  }

  cherryshoeAuthorization.CherryshoeAuthorization.mockImplementation(() => mockCherryshoeAuthorization);

  // mock return values that external dependency method(s) will return
  const userData = {
    firstname: "firstname",
    lastname: "lastname",
    username: "username",
  }

  mockCherryshoeAuthorization.getSessionData.mockReturnValue(userData);

  // unit under test
  // function returns nothing so nothing to expect with a return
  await expect(unit.processValidRequest(requestType, jsonRequest)).resolves;

  // verify that mock method(s) were expected to be called
  expect(mockCherryshoeAuthorization.getSessionData).toHaveBeenCalledTimes(1);
});

Break out instantiation of new object in a auth utility module:

sampleNew.js

const authUtils = require(./authUtils);

const processValidRequest = async (requestType, jsonRequestData) => {
  const userData = await authUtils.getSessionData(jsonRequestData.token);
  const dbResponse = await requestTypeHandlers[requestType](jsonRequestData, userData.username);
};

sampleNew.test.js

const unit = require(./sampleNew);

// import external dependencies for mocking
const authUtils = require(./authUtils);

test('Test processValidRequests', async () => {

  const mockLoginUtilsGetSessiondata = authUtils.getSessionData =
    jest.fn().mockName('mockLoginUtilsGetSessiondata');

  // mock return values that external dependency method(s) will return
  const userData = {
    firstname: "firstname",
    lastname: "lastname",
    username: "username",
  }

  mockLoginUtilsGetSessiondata.mockReturnValue(userData);

  // unit under test
  // function returns nothing so nothing to expect with a return
  await expect(unit.processValidRequest(requestType, jsonRequest)).resolves;

  // verify that mock method(s) were expected to be called

  expect(authUtils.getSessionData).toHaveBeenCalledTimes(1);
});

Websocket authorization using "ws", Node, and external tokening system

My project users an external system to manage a user's authorization, this external system returns a token which is stored in a cookie on the browser.  The backend Node.js application needs to make sure authorization is appropriate prior to letting a user make HTTP or WebSocket calls.  NOTE: User authentication across node instances will not be discussed in this post.

Environment:
CentOS Linux release 7.3.1611 (Core)
node v8.16.0
Backend ws version 7.0.0

HTTP:  Nothing additional needs to happen for the client code since the cookie Header is automatically sent with the HTTP request.  The backend code for HTTP calls include grabbing the cookie Header, parsing it for the token, and determine token's validity (return HTTP 401 Unauthorized if not valid, and continue on if valid).

WebSocket:
This ws issue and this article helped me a lot with this possible solution.

Client side code:
Since an external token system is being used (and not basic auth), the Authentication header cannot be used for my use case as well.  Adding custom headers are also not allowed for websocket connections.

Therefore, one workaround includes adding a URL param to the WebSocket URL, as seen in the below example.

const host = window.location.host;
const socket = new WebSocket(`wss://${host}/wsapi/?token=${token}`);

Backend code:

• The token needs to be retrieved from the WebSocket URL.  
• A subsequent call to determine if the token's validity needs to happen.  
• Valid:  For this example, I have assumed it is a valid token.  
• Look up any user info you want with the valid token, and add custom data "userInfo" to the request object
• The verifyClient callback function sets the result true.
• Not Valid: The verifyClient callback function sets the result as false, code as 401, and name as Unauthorized.

NOTE:  The ws documentation discourages the use of verifyClient, and instead recommends the upgrade event.  I'll have to try this sometime.

// During websocket handshake, validate the user is authorized
const wss = new WebSocketServer({
  server: server,
  verifyClient: (info, callback) => {
    const token = info.req.url.split('token=')[1];

    // check if this is a valid token, assume true
    const isValidToken = true;
    
    if (!isValidToken) {
        console.log("Error: token not valid");
        callback(false, 401, 'Unauthorized');
    } else {
      // NOTE: If we wanted to add custom user info here, we would look it up and add it.
      
      const userData = lookUpUserInfo(token);
      // add custom attributes to req object
      info.req.userInfo = {
        firstname: userData.firstname,
        lastname: userData.lastname,
        email: userData.email,
      }

      callback(true);
    }
  }
});

// From this point on we know the user is authorized since it was handled in the initial handshake
wss.on('connection', (ws, request) => {
  console.info("websocket connection open");

  // NOTE: If we wanted to access the custom user info here, we could access with request.user
  ...
  
  // Handle when message is sent from client
  ws.on('message', requestData => {
    // We don't have access to request.userInfo from when the connection was initially established.
    // So if you want access to the userInfo you would have to pass in the token again, and retrieve
    // the user data with lookUpUserInfo(token)
  
  });
  ...
});

pre-commit git hooks using husky

ESLint is already configured on my project.  There is a parent .eslintrc configuration file that is used for the backend code (uses Node.js).  There is also a child .eslintrc configuration file for the client code (uses React) that overrides/extends/adds rules that are different than for the backend.  I needed to run both npm scripts for linting during a git commit, using a pre-commit hook.

Other projects on my team uses manual written scripts that are installed in .git/hooks, but I decided to give husky a shot because:

• don't have to remember to install it the custom bash script to .git/hooks or upgrade it
• has over 1.5 million downloads and is actively maintained.  
• can be bypassed with usual --no-verify.
• git commit -m "TA-XXX:  Work and bypass git hooks" --no-verify

Environment:
CentOS Linux release 7.3.1611 (Core)
node v8.16.0
babel-eslint version "^10.0.2"
concurrently version "^3.1.0"
eslint version "^6.0.1"
eslint-plugin-flowtype version "^3.12.1"
eslint-plugin-import version "^2.18.2"
eslint-plugin-jsx-a11y version "^6.2.3"
eslint-plugin-react version "^7.14.2"
husky version "2.7.0"

Relevant npm scripts on the parent:
package.json

"clientlint": "cd client && npm run lint",
"lint": "node_modules/.bin/eslint -c .eslintrc .; exit 0",

Relevant npm script on the client child:
package.json

"lint": "node_modules/.bin/eslint -c .eslintrc .; exit 0"

Steps:

1. Add husky as a dev dependency on the parent package.json file, and run npm/yarn package manager to install

"devDependencies": {


  ...
  "husky": "2.7.0",
  ...
}



2. Configuring husky can be done in two different ways, both work great:
   • Using package.json:
     

     "husky": { 
       "hooks": { 
         "pre-commit": "npm run lint && npm run clientlint"
       }
     }
     

   • Using .huskyrc (DO NOT have "husky: {}" surrounding like it was in package.json file!)
     

     {
       "hooks": {
         "pre-commit": "npm run lint && npm run clientlint"
       }
     }
     

NOTE:  I submitted an issue Pre-commit fails with error when npm script calls "read with prompt" because I needed to add a npm script with a pause (I have now automated so this is OBE, but it is still a bug that has not been addressed by husky).

npm script in package.json:

"scripts": { 
  "clientlint": "cd client && npm run lint", 
  "clientcypress": "read -p Run Cypress and then press any key to continue",
  "lint": "node_modules/.bin/eslint -c .eslintrc .; exit 0"
}

.huskyrc file:

{
  "hooks": { 
    "pre-commit": "npm run clientcypress && npm run lint && npm run clientlint"
  }
}

Immutable.js examples with Map and List, fromJS, getIn, and get

I stumbled across Immutable.js in the code base today.  Here's a run down of how to set up an Immutable map/list, create a Immutable map/list using Immutable.fromJS() and how to use the getIn/get functions.

Environment:
immutable 3.8.2
assert 1.4.1
jest 20.0.3

describe('Immutable Map / List and fromJS Examples', function() {
  it('Immutable Map / List Example', function() {

    const immutableObj = Immutable.Map({
      property1: 'property1',
      property2: Immutable.List(['Cherry']),
      property3: 'property3',
      propertyMap: Immutable.Map({
        propertyMapProperty: 'Shoe',
      }),
    });
    let result = immutableObj.getIn(['propertyMap', 'propertyMapProperty'])
    let expectedResult = "Shoe";

    assert.equal(expectedResult, result);

    let immutableList = immutableObj.get('property2');
    result = immutableList.get(0);
    expectedResult = "Cherry";
    assert.equal(expectedResult, result);
  });

  it('Immutable fromJS Example', function() {

    const immutableObj = Immutable.fromJS({
      property1: 'property1',
      property2: ['Cherry'],
      property3: 'property3',
      propertyMap: {propertyMapProperty: "Shoe"},
    });

    let result = immutableObj.getIn(['propertyMap', 'propertyMapProperty'])
    let expectedResult = "Shoe";

    assert.equal(expectedResult, result);

    let immutableList = immutableObj.get('property2');
    result = immutableList.get(0);
    expectedResult = "Cherry";
    assert.equal(expectedResult, result);
  });
});

This article helped a lot.

Ansible - accessing .bashrc changes in subsequent task

My project uses ansible for local development setup, and I've found several examples where I needed to make updates to the .bashrc file:  updating the $PATH environment variable and creating a new environment variable for the VM's bridged IP address.  Accessing the changes in ~/.bashrc works fine when you logout and login again (or source it in the current shell) for anything done after the ansible playbook is run.  But, there's been a nuance I've encountered several times the past couple of months that I am sharing here to more easily remember the reasoning the next time it happens!

Environment: 
CentOS Linux release 7.3.1611 (Core)
ansible 2.8.2

After making updates to the ~/.bashrc file with an ansible task, the updates to the ~/.bashrc file is NOT available by default when performing the next ansible task.

For example, I needed to make updates to the ~/.bashrc file to update the $PATH environment variable to include another location where the yarn dependency was installed.  Another, I needed to export a new environment variable to more easily access the VM's bridged IP address.

To allow this to work as the intended user (vs root) I needed to switch user with su, but I also needed to have the command run inside a new login shell with an environment similar to a real login (the -l option for su).  Otherwise it wouldn't re-run the .bashrc file to get access to the just made changes in the ansible task that had just occurred.

NOTE:  {{user}} and {{project_path}} is set up in ansible's group_vars inventory.

- block:
  - name: export bridged ip address in .bashrc
    blockinfile:
      dest: '/home/{{user}}/.bashrc'
      block: |
       export BRIDGED_IP_ADDRESS=$(hostname -I | awk '{print $1}')
      marker: '# {mark} ANSIBLE MANAGED BLOCK - export BRIDGED_IP_ADDRESS'
      owner: '{{user}}'
      group: 'wheel'
      create: yes
  - name: command that needs access to $BRIDGED_IP_ADDRESS
    command: '<command>'
    args:
      chdir: '{{project_path}}'
    # The -l indicates to start shell as login shell with an environment similar to a real login
    become_method: su
    become_user: '{{user}}'
    become_flags: '-l'

This article helped a lot.