Showing posts with label ssh. Show all posts
Showing posts with label ssh. Show all posts

Sunday, January 26, 2020

How to debug a Node.js app running on a VM from local Windows

I'm working on an application where the frontend is React and the backend is Node running on CentOS 7.  Below I list steps on how to debug a PM2 managed clustered Node backend application with two different clustering methods:
  1. Node cluster module is used to configure cluster processes
  2. PM2 cluster mode is used to configure cluster processes
Environment:
Windows 10 Pro
CentOS Linux release 7.3.1611 (Core)
node v8.16.0

Node cluster module is used to configure cluster processes
NOTE: In this example PM2 starts the Node application via a npm script, and the backend code is spawning two cluster instances.

Update the command where you start node with the --inspect attribute. For us, we start node with the "startdev" npm script located in package.json. "startdev" uses the "server" script.

OLD
"scripts": {
  ..
  "startdev": "concurrently \"npm run server\" \"npm run client\"",
  "server": "cross-env node --max-old-space-size=8192 ./bin/cherryshoeServer.js",
  ..
}

NEW - You'll see that the "server" script is not changed as other npm scripts are also dependent on it. "startdev" uses a new "serverdev" script that was created to add the --inspect attribute.
"scripts": {
  ..
  "server": "cross-env node --max-old-space-size=8192 ./bin/cherryshoeServer.js",
  "startdev": "concurrently \"npm run serverdev\" \"npm run client\"",
  "serverdev": "cross-env node --max-old-space-size=8192 --inspect ./bin/cherryshoeServer.js",
  ..
}

A PM2 ecosystem configuration file is used; the config options of note are:

apps : [ {
  ..
  script: 'npm',
  // call appropriate npm script from package.json
  args: 'run startdev',
  ..
} ]

Start node with command "pm2 start", which calls the npm script "startdev", which calls npm script "serverdev", and runs cherryshoeServer.js on the local development environment.

Perform a ps command to verify the backend node processes are running. When cherryshoeServer.js is started, it spawns two worker processes on the local development environment (based on code to spawn two processes using Node cluster module).  Because of this, you'll see the first process below is the parent process with PID 5281, and the remaining two are worker processes with parent PPID 5281.

cs_admin  5281  5263  0 11:09 ?        00:00:00 node --max-old-space-
size=8192 --inspect ./bin/cherryshoeServer.js
cs_admin  5298  5281  0 11:09 ?        00:00:00 /usr/bin/node --max-old-
space-size=8192 --inspect --inspect-port=9230 /opt/cherryshoe/bin/cherryshoeServer.js
cs_admin  5303  5281  0 11:09 ?        00:00:00 /usr/bin/node --max-old-
space-size=8192 --inspect --inspect-port=9231 /opt/cherryshoe/bin/cherryshoeServer.js

Verify in the log file that debugger processes are listening. PM2 is used to manage logging, which is located in /var/log/cherryshoe/pm2/cs-out.log.  By default, the debugger listens on port 9229, then each additional debugger listener for each worker processeis incremented appropriately (9230 and 9231 respectively).
2020-01-26T11:09:22.657: [0] Debugger listening on ws://127.0.0.1:9229
/62dd92b9-a978-4cce-9e91-84b87835e014
2020-01-26T11:09:22.657: [0] For help see https://nodejs.org/en/docs
/inspector
2020-01-26T11:09:22.882: [1]
2020-01-26T11:09:22.882: [1] > origin-destination-client@0.2.0 start /opt
/cherryshoe/client
2020-01-26T11:09:22.882: [1] > concurrently "yarn watch-css" "cross-env
NODE_PATH=src/ react-scripts start"
2020-01-26T11:09:22.882: [1]
2020-01-26T11:09:22.965: [0] Running 2 processes
2020-01-26T11:09:22.975: [0] Debugger listening on ws://127.0.0.1:9230
/3e05b482-b186-4c7c-908d-7f5188353bb2
2020-01-26T11:09:22.975: [0] For help see https://nodejs.org/en/docs
/inspector
2020-01-26T11:09:22.978: [0] Debugger listening on ws://127.0.0.1:9231
/e3264361-6c0e-4843-8a4d-91b5ba9a8e4f
2020-01-26T11:09:22.978: [0] For help see https://nodejs.org/en/docs
/inspector

Back on your Windows local machine, open up your favorite app to ssh tunnel (I'm using git bash for this example but I am a big fan of MobaXterm) into the VM with appropriate ports to attach to the ports that the debuggers are listening on. This starts a ssh tunnel session where a connection to ports 8889-8891 (make sure these ports are not in use first) on your local machine will be forwarded to port 9229-9231 on the cherryshoe-dev.cherryshoe.com machine.  NOTES: I had to use 127.0.0.1 instead of localhost for this to work. Use a user account that has access to the VM. You may want to set up ssh passwordlessly so you don't have to enter passwords.
    ssh -L 8889:127.0.0.1:9229 cs_admin@cherryshoe-dev.cherryshoe.com
    ssh -L 8890:127.0.0.1:9230 cs_admin@cherryshoe-dev.cherryshoe.com
    ssh -L 8891:127.0.0.1:9231 cs_admin@cherryshoe-dev.cherryshoe.com

You can now attach a debugger client of choice to the X processes, as if the Node.js application was running locally. I will use Chrome DevTools as an example.

Open Chrome and enter "chrome://inspect" into the URL

Click "Discover network targets" Configure button and configure each of the ports to attach to:
  • Enter "127.0.0.1:8889"
  • Enter "127.0.0.1:8890"
  • Enter "127.0.0.1:8891"


You should now see the 3 processes attached in the debugger client:


Click the "inspect" link for one of the processes, this will open up the DevTools for Node. Under "Sources" tab you can click "Ctrl-P" to open up a file of choice to debug that is attached to the process. You do NOT need to 'Add folder to workspace'.

Open up each remaining process by clicking the "inspect" link.

Invoke the Node application, i.e. call a REST endpoint that it responds to
One of the worker processes will process the request. Any breakpoints will be reached and any console logs will be printed.

You have to open a window for each worker process running because you don't know which process will get picked and if you don't have them all open you can miss it and think debugging isn't working!

If you restart the Node backend, the DevTools Targets will recognize the new process IDs, but the DevTools windows won't. Therefore, you need to open up the DevTools windows again for each process via the "inspect" link.


PM2 cluster mode is used to configure cluster processes
NOTE: It was discovered that PM2 cluster mode and starting node via npm do not play nicely. The first process would listen on port X, but each subsequent process would error out saying port X was in use. Because of this, the node application was changed to start directly by invoking the appropriate js script.


Friday, August 17, 2018

Linux - How to remotely call a script with parameter over ssh

Below is an example on how to remotely call a bash script with a parameter over ssh.  I've done this multiples times in the past, but decided to document it this time so next time I can just reference this post.  This assumes that ssh to remote server passwordlessly already works with the intended user.

The example here performs an automated database script deployment for the sprint branch that is currently being worked.  Atlassian bamboo is being used for CI, but the DB script part was still being done manually.  This was a home-grown solution, the SQL for Bamboo add-on was not available and not used.

Environment: 
Source and Target DB server both centos-release-6-8.el6.centos.12.3.x86_64

The bamboo server was already configured with SSH Task's that called scripts to download the source code for the sprint branch on the source server.  So by the time the cherryshoe_db_copy_deploy.sh script is called, the database scripts files already reside on the source server.  We need the sprint number passed as a parameter because we have to know which sprint folder to copy.  This script then preps folders on the DB server (cherryshoe_db_prep.sh), copies the DB scripts, and runs scripts on the DB server (cherryshoe_db_deploy.sh).

cherryshoe_db_copy_deploy.sh
#!/bin/sh
usage () {
  echo "Usage (sprint number)"
}

SPRINT_NUMBER=$1
if [ -z "$1" ]
  then
   usage
   exit 1
fi

BUILD_HOME=/opt/app/cherryshoe/cherryshoe
BUILD_DIR=/opt/app/cherryshoe/Deploy
REMOTE_USER=cherryshoeuser
REMOTE_MACHINE=10.21.14.72
REMOTE_PATH_TO_SCRIPT=/opt/app/cherryshoe/build/database/automated_deployment/TEST

echo SPRINT_NUMBER $SPRINT_NUMBER
echo GIT_USER $GIT_USER
echo BUILD_HOME $BUILD_HOME
echo REMOTE_USER $REMOTE_USER
echo REMOTE_MACHINE $REMOTE_MACHINE
echo REMOTE_PATH_TO_SCRIPT $REMOTE_PATH_TO_SCRIPT
###########################################################

cd $BUILD_HOME

echo Call remote script to create sprint DB automated deployment folder if not exists
ssh $REMOTE_USER@$REMOTE_MACHINE "cd $REMOTE_PATH_TO_SCRIPT;./cherryshoe_db_prep.sh $SPRINT_NUMBER"

echo Copy current branch DB scripts to DB server
cd $BUILD_HOME
cd database/scripts/release_scripts/$SPRINT_NUMBER
scp *.sql $REMOTE_USER@$REMOTE_MACHINE:$REMOTE_PATH_TO_SCRIPT/$SPRINT_NUMBER
scp *.sh $REMOTE_USER@$REMOTE_MACHINE:$REMOTE_PATH_TO_SCRIPT/$SPRINT_NUMBER

echo Call remote script to perform deployment of db scripts
ssh $REMOTE_USER@$REMOTE_MACHINE "cd $REMOTE_PATH_TO_SCRIPT;./cherryshoe_db_deploy.sh $SPRINT_NUMBER"

echo DONE

cherryshoe_db_prep.sh
#!/bin/sh

if [ -z "$1" ]
  then
    echo "No sprint folder specified"
    exit 1
fi

SPRINT_FOLDER=$1
echo SPRINT_FOLDER: $SPRINT_FOLDER

DB_SCRIPT_DEPLOY_HOME=/opt/app/cherryshoe/build/database/automated_deployment/TEST

echo  Create sprint DB automated deployment folder if not exists.
cd $DB_SCRIPT_DEPLOY_HOME
mkdir -p $SPRINT_FOLDER

echo DONE

cherryshoe_db_deploy.sh
#!/bin/sh

if [ -z "$1" ]
  then
    echo "No sprint folder specified"
    exit 1
fi


SPRINT_FOLDER=$1
echo SPRINT_FOLDER: $SPRINT_FOLDER

DB_SCRIPT_DEPLOY_HOME=/opt/app/cherryshoe/build/database/automated_deployment/TEST
DB_USERNAME=cherryshoe
DB_PASSWORD=cherryshoe
DB_SCHEMA=cherryshoetest

echo changing to sprint folder: $DB_SCRIPT_DEPLOY_HOME/$SPRINT_FOLDER
cd $DB_SCRIPT_DEPLOY_HOME/$SPRINT_FOLDER

BASHFILE=$(ls | grep .sh)
echo BASHFILE: $BASHFILE

echo check if bash file exists
if [ -e "$BASHFILE" ]
then
  echo make sprint script file readable and executable
  chmod +rx *.sh

  echo executing bash file
  ./$BASHFILE $DB_SCHEMA $DB_USERNAME $DB_PASSWORD
else
  echo bash file does not exist, SKIPPING
fi

echo DONE



This article helped a lot.

Thursday, October 10, 2013

Issues with migrating data between oracle versions

A little background...
I implemented a complete backup and restore strategy for Alfresco and a custom application that have synchronized data (Alfresco holds repository documents, Application DB holds application specific data). This was done with Alfresco 4.1.2 on a clustered environment, Oracle backend for both database schemas, and SOLR for search subsystem; all running on two CentOS VMs. The database backups and recoveries were implemented with Oracle Datapump.   The database operations were all invoked remotely on the database server and used SSH Public Key Based Authentication for a completely automated solution.   The same backup and recovery scripts also work on a local Ubuntu environment with a default Alfresco 4.1.2 installation with Oracle; Oracle was installed on the same VM.

...and then I had an issue
I ran into a snag when I tried restoring the custom application database export on on a different server than where I took the backup.  The source expdp'ed dump file was created from "Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production".  I needed to import it into my destination local Ubuntu environment which is "Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production".

Notice the versioning difference.  I didn't think a sub-sub version would cause any problems, little did I know...

Steps to figure out the problem:
  1. I ran the following command on my Ubuntu destination server with the user that owned the schema.  I had to remap_schema and remap_tablespace since source and destination schema and tablespace names were different.
    impdp $SQL_USER/$SQL_PASSWORD@$SQL_CONN remap_schema=<source_schema>:$SQL_USER remap_tablespace=<source_tablespace>:$SQL_TABLESPACE full=N directory=datapump_dir dumpfile=$DUMP_FILENAME logfile=$DUMP_FILENAME.imp.log
    

  2. The import log split out the diagnoses below:
    ORA-39097: Data Pump job encountered unexpected error -1031
    ORA-39065: unexpected master process exception in DISPATCH
    ORA-01031: insufficient privileges
    THANKS A LOT ORACLE - sarcastically!  ORA-01031: insufficient privileges really tells me a whole lot (Mind you I had granted various permissions and my Alfresco restore had worked just fine with the same generic scripts).

  3. I then ran the import as sysdba.  I tried implementing this by granting the user dba priviledges but that did not work, perhaps I will try that again later...
    impdp \"/ as sysdba\" remap_schema=<source_schema>:$SQL_USER remap_tablespace=<source_tablespace>:$SQL_TABLESPACE full=N directory=datapump_dir dumpfile=$DUMP_FILENAME logfile=$DUMP_FILENAME.imp.log
    

  4. MUCH BETTER log file this time:
    ORA-39097: Data Pump job encountered unexpected error -30094
    ORA-39065: unexpected master process exception in DISPATCH
    ORA-30094: failed to find the time zone data file for version 11 in $ORACLE_HOME/oracore/zoneinfo

Solution:
This article helped me figure out how to solve the problem.  Since we are restoring from "Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production" to "Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production", the timezone versions were off.  Specifically the source database was on version 11, where the destination database was on version 13 (You can check with SELECT version FROM v$timezone_file;).  This matters only because the database schema has column types of TIMESTAMP(6) WITH TIME ZONE.
  • Copy over the timezone version pair that was missing from the source to the destination $ORACLE_HOME/oracore/zoneinfo folder.  In this particular example, the missing files were timezlrg_11.dat timezone_11.dat.
  • Change the permissions of the files to match the other .dat files in the directory.
  • Restart Oracle.
The import will now work with the impdp command from Step 2 above!

Tuesday, July 23, 2013

ssh to remote server passwordlessly

Often you'll want to run a script on serverB from serverA using ssh, one of those times is during an automated development build where you don't have the option of entering in a password.  Let's explore some options to do that passwordlessly.

Say serverA is where your automated build tool is installed.  serverB is where you want to deploy a war file to an application server using an automated script you want to invoke with ssh.

Both serverA and serverB were running CentOS for these set of instructions.

Solution
Step 1: On serverA machine, generate the authentication keys for ssh authentication.
  • For default rsa key filename, enter the following
    • [cherryshoe@serverA ~]$ ssh-keygen -t rsa
  • For specific rsa key filename for serverB, enter the following
    • [cherryshoe@serverA .ssh]$ ssh-keygen -t rsa -f ./id_rsa.serverB -C "Key for dev integration test server"
This will generate the authentication key pair -

  • id_rsa and id_rsa.pub OR 
  • id_rsa.serverB and id_rsa.serverB.pub

The result will be similar to:
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ./id_rsa.serverB.
Your public key has been saved in ./id_rsa.serverB.pub.
The key fingerprint is:
e9:d6:a6:ad:6d:89:15:31:59:be:ae:32:99:eb:10:e6 Key for dev integration test server

Step 2: NOTE: This step is not required.  Use ssh-agent to securely store the private key on serverA.  This will enable you to avoid continuing to type the pass-phrase when sshing from serverA to serverB.

[cherryshoe@serverA .ssh]$ ssh-agent $BASH
[cherryshoe@serverA .ssh]$ ssh-add
Identity added: /home/cherryshoe/.ssh/id_rsa
[cherryshoe@serverA .ssh]$ ssh-add id_rsa.serverB
Identity added: /home/cherryshoe/.ssh/id_rsa.serverB

NOTE: To list all private keys on serverA, issue ssh-add -l

Step 3: Copy public key to serverB

Step 3a: If sshpass is installed, otherwise use Step 3b.
  • Navigate to /<home>/.ssh - note that the id_rsa.pub already exists (default name from above step)
  • Run a single command with sshpass, if installed
    • sshpass -p <password of user on serverB> ssh-copy-id <server B IP address>
  • Test
    • ssh <server B IP address>
Step 3b: Copy the public key over to serverB <home>/.ssh/authorized_keys file.

Option a: manual
  • If the file doesn't exist yet, then copy over public key.pub file over as the authorized_keys file.
    • i.e: 
      • scp id_rsa.pub to the serverB AS <home>/.ssh/authorized_keys
  • If it does, then open up the contents of the public key .pub file on serverA, and append it to the existing authorized_keys file on serverB.
Option b: automated
Use ssh-copy-id to automate this step
[cherryshoe@serverA .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub cherryshoe@serverB
cherryshoe@serverB's password:

Step 4: Ensure serverB user's .ssh directory has permission 700 and the authorized
_keys file has permission 640

drwx------. 2 cherrryshoe wheel 4096 Jul 8 08:00 .
drwx------. 5 cherryshoe cherryshoe   4096 Jul 7 15:43 ..
-rw-r-----. 1 cherryshoe wheel  398 Jul 8 08:00 authorized_keys

You should now be able to ssh from serverA to serverB passwordlessly!  

NOTE:  The above instructions are for when the user on serverA and serverB are the same username. Take for example serverA cherryshoe wants to ssh to serverB as shoetech:

On serverA machine:

  • Generate the keys for user shoetech
  • ssh-keygen -t rsa -f id_serverB_shoetech -C "shoetech@serverB"
  • ssh-copy-id id_serverB_shoetech.pub shoetech@serverB
  • check the known_hosts file now has the serverB IP


Generate the keys for user cherryshoe

  • ssh-keygen -t rsa -f id_serverB_cherryshoe -C "cherryshoe@serverB"


On serverB machine:

  • Append the contents of id_serverB_cherryshoe.pub to shoetech's authorized_keys file.  The authorized_keys of shoetech user should have two .pub keys (one for cherryshoe user, second for shoetech user).  Make sure each .pub key has username@hostname at the end (this will already be at the end if you use the -C parameter when generating the key pair with ssh-keygen command).


From serverA, now you will need to specify the path to private key for user shoetech, to ssh into serverB as shoetech

  • ssh -i ~/.ssh/id_serverB_shoetech shoetech@serverB

These articles were a great help.
http://www.cyberciti.biz/tips/ssh-public-key-based-authentication-how-to.html
http://www.karan.org/blog/index.php/2009/08/25/multiple-ssh-private-keys
http://stackoverflow.com/questions/2419566/best-way-to-use-multiple-ssh-private-keys-on-one-client