Saturday, November 23, 2019

Example Jest test with an object instantiation inside method under test, then moved out into a utility class

Below are two examples of code and the corresponding unit tests for them. The first example includes an instantiation of an object inside it, the other breaks the instantiation of the object to a utility module.  Both are doable, but the latter way easier to unit test.

Environment:
CentOS Linux release 7.3.1611 (Core)
node v8.16.0
jest 24.8.0


Instantiating a new object inside method of test:
sampleOld.js
const cherryshoeAuthorization = require("cherry-shoe-auth");

const processValidRequest = async (requestType, jsonRequestData) => {
  const authService = new cherryshoeAuthorization.CherryshoeAuthorization("cherryshoe.com");
  const userData = await authService.getSessionData(jsonRequestData.token);
  const dbResponse = await requestTypeHandlers[requestType](jsonRequestData, userData.username);
};

sampleOld.test.js

const unit = require(./sampleOld);

// import external dependencies for mocking
const cherryshoeAuthorization = require('cherry-shoe-auth');

// mock cherry-shoe-auth
jest.genMockFromModule('cherry-shoe-auth');
jest.mock('cherry-shoe-auth');

test('Test processValidRequests', async () => {

  // mock external dependency method(s)
  const mockCherryshoeAuthorization = {
    getSessionData: jest.fn(),
  }

  cherryshoeAuthorization.CherryshoeAuthorization.mockImplementation(() => mockCherryshoeAuthorization);

  // mock return values that external dependency method(s) will return
  const userData = {
    firstname: "firstname",
    lastname: "lastname",
    username: "username",
  }

  mockCherryshoeAuthorization.getSessionData.mockReturnValue(userData);

  // unit under test
  // function returns nothing so nothing to expect with a return
  await expect(unit.processValidRequest(requestType, jsonRequest)).resolves;

  // verify that mock method(s) were expected to be called
  expect(mockCherryshoeAuthorization.getSessionData).toHaveBeenCalledTimes(1);
});


Break out instantiation of new object in a auth utility module:

sampleNew.js
const authUtils = require(./authUtils);

const processValidRequest = async (requestType, jsonRequestData) => {
  const userData = await authUtils.getSessionData(jsonRequestData.token);
  const dbResponse = await requestTypeHandlers[requestType](jsonRequestData, userData.username);
};

sampleNew.test.js

const unit = require(./sampleNew);

// import external dependencies for mocking
const authUtils = require(./authUtils);

test('Test processValidRequests', async () => {

  const mockLoginUtilsGetSessiondata = authUtils.getSessionData =
    jest.fn().mockName('mockLoginUtilsGetSessiondata');

  // mock return values that external dependency method(s) will return
  const userData = {
    firstname: "firstname",
    lastname: "lastname",
    username: "username",
  }

  mockLoginUtilsGetSessiondata.mockReturnValue(userData);

  // unit under test
  // function returns nothing so nothing to expect with a return
  await expect(unit.processValidRequest(requestType, jsonRequest)).resolves;

  // verify that mock method(s) were expected to be called

  expect(authUtils.getSessionData).toHaveBeenCalledTimes(1);
});

Saturday, October 12, 2019

Websocket authorization using "ws", Node, and external tokening system

My project users an external system to manage a user's authorization, this external system returns a token which is stored in a cookie on the browser.  The backend Node.js application needs to make sure authorization is appropriate prior to letting a user make HTTP or WebSocket calls.  NOTE: User authentication across node instances will not be discussed in this post.

Environment:
CentOS Linux release 7.3.1611 (Core)
node v8.16.0
Backend ws version 7.0.0

HTTP:  Nothing additional needs to happen for the client code since the cookie Header is automatically sent with the HTTP request.  The backend code for HTTP calls include grabbing the cookie Header, parsing it for the token, and determine token's validity (return HTTP 401 Unauthorized if not valid, and continue on if valid).

WebSocket:
This ws issue and this article helped me a lot with this possible solution.

Client side code:
Since an external token system is being used (and not basic auth), the Authentication header cannot be used for my use case as well.  Adding custom headers are also not allowed for websocket connections.

Therefore, one workaround includes adding a URL param to the WebSocket URL, as seen in the below example.

const host = window.location.host;
const socket = new WebSocket(`wss://${host}/wsapi/?token=${token}`);

Backend code:

  • The token needs to be retrieved from the WebSocket URL.  
  • A subsequent call to determine if the token's validity needs to happen.  
    • Valid:  For this example, I have assumed it is a valid token.  
      • Look up any user info you want with the valid token, and add custom data "userInfo" to the request object
      • The verifyClient callback function sets the result true.
    • Not Valid: The verifyClient callback function sets the result as false, code as 401, and name as Unauthorized.

NOTE:  The ws documentation discourages the use of verifyClient, and instead recommends the upgrade event.  I'll have to try this sometime.


// During websocket handshake, validate the user is authorized
const wss = new WebSocketServer({
  server: server,
  verifyClient: (info, callback) => {
    const token = info.req.url.split('token=')[1];

    // check if this is a valid token, assume true
    const isValidToken = true;
    
    if (!isValidToken) {
        console.log("Error: token not valid");
        callback(false, 401, 'Unauthorized');
    } else {
      // NOTE: If we wanted to add custom user info here, we would look it up and add it.
      
      const userData = lookUpUserInfo(token);
      // add custom attributes to req object
      info.req.userInfo = {
        firstname: userData.firstname,
        lastname: userData.lastname,
        email: userData.email,
      }

      callback(true);
    }
  }
});

// From this point on we know the user is authorized since it was handled in the initial handshake
wss.on('connection', (ws, request) => {
  console.info("websocket connection open");

  // NOTE: If we wanted to access the custom user info here, we could access with request.user
  ...
  
  // Handle when message is sent from client
  ws.on('message', requestData => {
    // We don't have access to request.userInfo from when the connection was initially established.
    // So if you want access to the userInfo you would have to pass in the token again, and retrieve
    // the user data with lookUpUserInfo(token)
  
  });
  ...
});

Friday, September 13, 2019

pre-commit git hooks using husky

ESLint is already configured on my project.  There is a parent .eslintrc configuration file that is used for the backend code (uses Node.js).  There is also a child .eslintrc configuration file for the client code (uses React) that overrides/extends/adds rules that are different than for the backend.  I needed to run both npm scripts for linting during a git commit, using a pre-commit hook.

Other projects on my team uses manual written scripts that are installed in .git/hooks, but I decided to give husky a shot because:

  • don't have to remember to install it the custom bash script to .git/hooks or upgrade it
  • has over 1.5 million downloads and is actively maintained.  
  • can be bypassed with usual --no-verify.
    • git commit -m "TA-XXX:  Work and bypass git hooks" --no-verify
Environment:
CentOS Linux release 7.3.1611 (Core)
node v8.16.0
babel-eslint version "^10.0.2"
concurrently version "^3.1.0"
eslint version "^6.0.1"
eslint-plugin-flowtype version "^3.12.1"
eslint-plugin-import version "^2.18.2"
eslint-plugin-jsx-a11y version "^6.2.3"
eslint-plugin-react version "^7.14.2"
husky version "2.7.0"

Relevant npm scripts on the parent:
package.json

"clientlint": "cd client && npm run lint",
"lint": "node_modules/.bin/eslint -c .eslintrc .; exit 0",

Relevant npm script on the client child:
package.json

"lint": "node_modules/.bin/eslint -c .eslintrc .; exit 0"

Steps:
  1. Add husky as a dev dependency on the parent package.json file, and run npm/yarn package manager to install

  2. "devDependencies": {
      ...
      "husky": "2.7.0",
      ...
    }
    

  3. Configuring husky can be done in two different ways, both work great:
    • Using package.json:
      "husky": { 
        "hooks": { 
          "pre-commit": "npm run lint && npm run clientlint"
        }
      }
      
    • Using .huskyrc (DO NOT have "husky: {}" surrounding like it was in package.json file!)
      {
        "hooks": {
          "pre-commit": "npm run lint && npm run clientlint"
        }
      }
      

NOTE:  I submitted an issue Pre-commit fails with error when npm script calls "read with prompt" because I needed to add a npm script with a pause (I have now automated so this is OBE, but it is still a bug that has not been addressed by husky).

npm script in package.json:
"scripts": { 
  "clientlint": "cd client && npm run lint", 
  "clientcypress": "read -p Run Cypress and then press any key to continue",
  "lint": "node_modules/.bin/eslint -c .eslintrc .; exit 0"
}

.huskyrc file:
{
  "hooks": { 
    "pre-commit": "npm run clientcypress && npm run lint && npm run clientlint"
  }
}

Wednesday, August 14, 2019

Immutable.js examples with Map and List, fromJS, getIn, and get

I stumbled across Immutable.js in the code base today.  Here's a run down of how to set up an Immutable map/list, create a Immutable map/list using Immutable.fromJS() and how to use the getIn/get functions.

Environment:
immutable 3.8.2
assert 1.4.1
jest 20.0.3

describe('Immutable Map / List and fromJS Examples', function() {
  it('Immutable Map / List Example', function() {

    const immutableObj = Immutable.Map({
      property1: 'property1',
      property2: Immutable.List(['Cherry']),
      property3: 'property3',
      propertyMap: Immutable.Map({
        propertyMapProperty: 'Shoe',
      }),
    });
    let result = immutableObj.getIn(['propertyMap', 'propertyMapProperty'])
    let expectedResult = "Shoe";

    assert.equal(expectedResult, result);

    let immutableList = immutableObj.get('property2');
    result = immutableList.get(0);
    expectedResult = "Cherry";
    assert.equal(expectedResult, result);
  });

  it('Immutable fromJS Example', function() {

    const immutableObj = Immutable.fromJS({
      property1: 'property1',
      property2: ['Cherry'],
      property3: 'property3',
      propertyMap: {propertyMapProperty: "Shoe"},
    });

    let result = immutableObj.getIn(['propertyMap', 'propertyMapProperty'])
    let expectedResult = "Shoe";

    assert.equal(expectedResult, result);

    let immutableList = immutableObj.get('property2');
    result = immutableList.get(0);
    expectedResult = "Cherry";
    assert.equal(expectedResult, result);
  });
});


This article helped a lot.

Friday, August 9, 2019

Ansible - accessing .bashrc changes in subsequent task

My project uses ansible for local development setup, and I've found several examples where I needed to make updates to the .bashrc file:  updating the $PATH environment variable and creating a new environment variable for the VM's bridged IP address.  Accessing the changes in ~/.bashrc works fine when you logout and login again (or source it in the current shell) for anything done after the ansible playbook is run.  But, there's been a nuance I've encountered several times the past couple of months that I am sharing here to more easily remember the reasoning the next time it happens!

Environment: 
CentOS Linux release 7.3.1611 (Core)
ansible 2.8.2

After making updates to the ~/.bashrc file with an ansible task, the updates to the ~/.bashrc file is NOT available by default when performing the next ansible task.

For example, I needed to make updates to the ~/.bashrc file to update the $PATH environment variable to include another location where the yarn dependency was installed.  Another, I needed to export a new environment variable to more easily access the VM's bridged IP address.

To allow this to work as the intended user (vs root) I needed to switch user with su, but I also needed to have the command run inside a new login shell with an environment similar to a real login (the -l option for su).  Otherwise it wouldn't re-run the .bashrc file to get access to the just made changes in the ansible task that had just occurred.

NOTE:  {{user}} and {{project_path}} is set up in ansible's group_vars inventory.

- block:
  - name: export bridged ip address in .bashrc
    blockinfile:
      dest: '/home/{{user}}/.bashrc'
      block: |
       export BRIDGED_IP_ADDRESS=$(hostname -I | awk '{print $1}')
      marker: '# {mark} ANSIBLE MANAGED BLOCK - export BRIDGED_IP_ADDRESS'
      owner: '{{user}}'
      group: 'wheel'
      create: yes
  - name: command that needs access to $BRIDGED_IP_ADDRESS
    command: '<command>'
    args:
      chdir: '{{project_path}}'
    # The -l indicates to start shell as login shell with an environment similar to a real login
    become_method: su
    become_user: '{{user}}'
    become_flags: '-l'

This article helped a lot.

Friday, July 26, 2019

Culmination of hodgepodge realizations of WebSockets (Circa July 2019)

I've read many articles about WebSockets the past month, and each article gives one to several tidbits of information, but not enough to answer all my questions.

Environment:
CentOS Linux release 7.3.1611 (Core)
node v8.16.0
nginx version: nginx/1.12.2

Here's the culmination of what I've learned so far.  I am using Node.js with Express, "ws" for the Node WebSocket library, and native WebSocket javascript for the front-end.

  1. So many articles to read!  Some of the early articles I read taught how to code WebSocket connections, frames, listeners, etc yourself!  This is good reference but definitely not ideal to actually do! There's many libraries out there that can do this for you  I eventually chose "ws" as I discussed in my last post.
    • The only con I see right now it doesn't have a fallback method to long-polling like some libraries available (i.e. socket.io).   This doesn't concern me because WebSockets are supported by all major browsers - please reference number 7 below for more thoughts.
  2. Not explicitly clear if you are supposed to keep client connections open and re-use them as needed (or open a new one for each request).  
    • The typical use case for keeping client connection open and re-use them is for a chat type of application.
    • My use case (where I am opening a WebSocket connection each request) is that "occasionally" there will be requests that take "longer than usual", and where it used to be HTTP there would be a HTTP response timeout on the web server level (set at 5 mins).  To not have to keep increasing this timeout, a different solution was needed - polling, WebSockets, etc.  WebSockets were chosen for several reasons which I won't go into here.
  3. The Node backend is clustered with one instance per CPU, and a WebSocket Server is running alongside each Node Server instance, so it can handle many WebSocket client connections concurrently.  If you want a large number multiple concurrent WebSocket connections concurrently, take a look at this article.
  4. The WebSocket protocol discusses the concept of ping/pong messages for the server to know each client is still connected, as well as conversely each client connection can still communicate with the server (i.e. Wifi connection terminated unexpectedly):
    • If the WebSocket Server sends a ping or pong, then the client should have to be able to recognize a ping/pong was sent, and respond, but there is no client side javascript code that can send or recognize/receive ping/pong frames, it's either supported by the browser or not.  
      • As referenced from this article - “ping/pong frames” are used to check the connection, sent from the server, the browser responds to these automatically. 
      • This article was helpful in teaching me that "Unfortunately, the Websocket protocol does not provide a similar method for the client, to find out if it is still connected to the server. This can happen, if the connection simply disappears without further notification. In order to have the client recognize this, some Javascript code has to be added to the client code responsible for the Websocket".
      • Since my application controls both client and server code, adding in a specific header/metadata in a web socket message with `WebSocket#send()` vs using `WebSocket#ping()` can be used to implement a heartbeat. 
        • This article - helped me build a heartbeat system with messages.
      • During server development testing, I noticed that the heartbeat message was sometimes sent to the client and sometimes not:
        • Noticed that if paused in debugger, the setInterval function used for sending a heartbeat message from server -> client wasn't working, since the worker that is paused (so the heartbeat from server -> client is NOT sent).
        • If setTimeout(sleep) function was enabled (simulating actual work being perform), the setInterval check DOES RUN for the worker (so the heartbeat from server -> client is sent).
  5. Web Server configuration needs configuration to:
  6. This repo had a great explanation of all the various WebSocket libraries out there.
  7. WebSockets are supported by all the major browser,  but Internet Explorer (11) still has a default maximum number of client connections at 6.
  8. Easiest to understand guide were the WebSockets API documentation by mozilla.org.

Monday, June 24, 2019

Common HTTP Status Code Errors when getting started with WebSockets

I'm working on an application where the frontend is React and the backend is Node running on CentOS 7.  I am introducing WebSockets for some data requests that can be long-running queries.  I haven't used WebSockets in the past so wanted to share two HTTP error statuses I repeatedly got when starting out.  There are multiple reasons why I chose the Node "ws" library for WebSocket communication:

  • Actively maintained, roughly new version per month
  • Implements both WebSocket protocol versions 8 and 13 for Node
  • Straightforward to use, and easy to use with the Express Node web framework.  The server is used for both http and WebSockets communication
  • Over 13 million weekly downloads (most I found from any of the WebSocket libraries) on npmjs.com
  • Handles WebSocket frames for you

Environment:
CentOS Linux release 7.3.1611 (Core)
node v8.16.0
nginx version: nginx/1.12.2

Error 1 - HTTP Status Code 404
The first three browsers gave at least an HTTP Response error code, Firefox didn't provide much info on the error message.

  1. In IE: SCRIPT12008: WebSocket Error: Incorrect HTTP response. Status code 404, Not Found
  2. In Edge: SCRIPT12008: SCRIPT12008: WebSocket Error: Incorrect HTTP response. Status code 404, Not Found
  3. In Chrome: WebSocket connection to 'wss://cherryshoe.com/wsapi' failed: Error during WebSocket handshake: Unexpected response code: 404
  4. In Firefox:
    • The connection to wss://cherryshoe.com/sockjs-node/314/jv0jnivp/websocket was interrupted while the page was loading.
    • Firefox can’t establish a connection to the server at wss://cherryshoe.com/wsapi/.

Problem and Solution:  The nginx web server was not configured for WebSocket support, had to add the following to the /wsapi location
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

Error 2 - HTTP Status code 502

The first three browsers gave at least an HTTP Response error code, Firefox didn't provide much info on the error message.

  1. In IE: SCRIPT12008: WebSocket Error: Incorrect HTTP response. Status code 502, Bad Gateway
  2. In Edge: SCRIPT12008: SCRIPT12008: WebSocket Error: Incorrect HTTP response. Status code 502, Bad Gateway
  3. In Chrome: WebSocket connection to 'wss://cherryshoe.com/wsapi/' failed: Error during WebSocket handshake: Unexpected response code: 502
  4. In Firefox:
    • The connection to wss://cherryshoe.com/sockjs-node/957/mxbslcja/websocket was interrupted while the page was loading.
    • Firefox can’t establish a connection to the server at wss://cherryshoe.com/wsapi/.
Problem and Solution 2:
I would get multiple successful client connections (one for Firefox, two or three for Chrome/IE/Edge), but then the next connection consistently received Status Code 502.  This error essentially kills any communication with the WebSocket Server itself (not just that specific client connection), which would require a restart of the Node server.

The problem was that in close event handler ws.on('close') I was calling close on the WebSocketServer itself (wss.close()).  I didn't actually want to close the WebSocketServer itself, only wanted to close the connection, but in the close event handler ws.on('close') the connection is already closed!

The client code message event handler onmessage would close the socket after it received a 'Done' message, which caused the WebSocket Server ws.on('close') to be called.
    let socket;

    ...
    socket setup code
    ...

    // this is over https, so have to make it web socket over https "wss:"
    socket = new WebSocket('wss://cherryshoe.com/wsapi/');

    // message received from web socket server
    socket.onmessage = function(evt) {

      // If data request is 'Done'
      socket.close([1000], "Work complete");
    };


The server code web socket close event handler was closing the WebSocket server:

var WebSocketServer = require('ws').Server;

...
code to setup server
...


var wss = new WebSocketServer({server: server});
wss.on('connection', function (ws, socket) {
    console.info("websocket connection open");

    // send status back to client
    ws.send('Connection established');

    // Handle when message is sent from client
    ws.on('message', async (requestData) => {
        // process the request code here
        ws.send('Done');
    });

    ws.on('error', function (ws, code, reason) {
      console.info("websocket connection error");
    });

    ws.on('close', function (ws, code, reason) {
      console.info("websocket connection close");
      wss.close();
    });
});


The solution was to remove the wss.close from the ws.on('close') code:

wss.on('connection', function (ws, socket) { 

   ...

   ws.on('close', function (ws, code, reason) {
      // connection is closed, but you don't actually want to close down the
      // WebSocketServer so do not do wss.close()!
      console.info("websocket connection close");
   });
   
   ...

});