Showing posts with label Cluster. Show all posts
Showing posts with label Cluster. Show all posts

Sunday, January 26, 2020

How to debug a Node.js app running on a VM from local Windows

I'm working on an application where the frontend is React and the backend is Node running on CentOS 7.  Below I list steps on how to debug a PM2 managed clustered Node backend application with two different clustering methods:
  1. Node cluster module is used to configure cluster processes
  2. PM2 cluster mode is used to configure cluster processes
Environment:
Windows 10 Pro
CentOS Linux release 7.3.1611 (Core)
node v8.16.0

Node cluster module is used to configure cluster processes
NOTE: In this example PM2 starts the Node application via a npm script, and the backend code is spawning two cluster instances.

Update the command where you start node with the --inspect attribute. For us, we start node with the "startdev" npm script located in package.json. "startdev" uses the "server" script.

OLD
"scripts": {
  ..
  "startdev": "concurrently \"npm run server\" \"npm run client\"",
  "server": "cross-env node --max-old-space-size=8192 ./bin/cherryshoeServer.js",
  ..
}

NEW - You'll see that the "server" script is not changed as other npm scripts are also dependent on it. "startdev" uses a new "serverdev" script that was created to add the --inspect attribute.
"scripts": {
  ..
  "server": "cross-env node --max-old-space-size=8192 ./bin/cherryshoeServer.js",
  "startdev": "concurrently \"npm run serverdev\" \"npm run client\"",
  "serverdev": "cross-env node --max-old-space-size=8192 --inspect ./bin/cherryshoeServer.js",
  ..
}

A PM2 ecosystem configuration file is used; the config options of note are:

apps : [ {
  ..
  script: 'npm',
  // call appropriate npm script from package.json
  args: 'run startdev',
  ..
} ]

Start node with command "pm2 start", which calls the npm script "startdev", which calls npm script "serverdev", and runs cherryshoeServer.js on the local development environment.

Perform a ps command to verify the backend node processes are running. When cherryshoeServer.js is started, it spawns two worker processes on the local development environment (based on code to spawn two processes using Node cluster module).  Because of this, you'll see the first process below is the parent process with PID 5281, and the remaining two are worker processes with parent PPID 5281.

cs_admin  5281  5263  0 11:09 ?        00:00:00 node --max-old-space-
size=8192 --inspect ./bin/cherryshoeServer.js
cs_admin  5298  5281  0 11:09 ?        00:00:00 /usr/bin/node --max-old-
space-size=8192 --inspect --inspect-port=9230 /opt/cherryshoe/bin/cherryshoeServer.js
cs_admin  5303  5281  0 11:09 ?        00:00:00 /usr/bin/node --max-old-
space-size=8192 --inspect --inspect-port=9231 /opt/cherryshoe/bin/cherryshoeServer.js

Verify in the log file that debugger processes are listening. PM2 is used to manage logging, which is located in /var/log/cherryshoe/pm2/cs-out.log.  By default, the debugger listens on port 9229, then each additional debugger listener for each worker processeis incremented appropriately (9230 and 9231 respectively).
2020-01-26T11:09:22.657: [0] Debugger listening on ws://127.0.0.1:9229
/62dd92b9-a978-4cce-9e91-84b87835e014
2020-01-26T11:09:22.657: [0] For help see https://nodejs.org/en/docs
/inspector
2020-01-26T11:09:22.882: [1]
2020-01-26T11:09:22.882: [1] > origin-destination-client@0.2.0 start /opt
/cherryshoe/client
2020-01-26T11:09:22.882: [1] > concurrently "yarn watch-css" "cross-env
NODE_PATH=src/ react-scripts start"
2020-01-26T11:09:22.882: [1]
2020-01-26T11:09:22.965: [0] Running 2 processes
2020-01-26T11:09:22.975: [0] Debugger listening on ws://127.0.0.1:9230
/3e05b482-b186-4c7c-908d-7f5188353bb2
2020-01-26T11:09:22.975: [0] For help see https://nodejs.org/en/docs
/inspector
2020-01-26T11:09:22.978: [0] Debugger listening on ws://127.0.0.1:9231
/e3264361-6c0e-4843-8a4d-91b5ba9a8e4f
2020-01-26T11:09:22.978: [0] For help see https://nodejs.org/en/docs
/inspector

Back on your Windows local machine, open up your favorite app to ssh tunnel (I'm using git bash for this example but I am a big fan of MobaXterm) into the VM with appropriate ports to attach to the ports that the debuggers are listening on. This starts a ssh tunnel session where a connection to ports 8889-8891 (make sure these ports are not in use first) on your local machine will be forwarded to port 9229-9231 on the cherryshoe-dev.cherryshoe.com machine.  NOTES: I had to use 127.0.0.1 instead of localhost for this to work. Use a user account that has access to the VM. You may want to set up ssh passwordlessly so you don't have to enter passwords.
    ssh -L 8889:127.0.0.1:9229 cs_admin@cherryshoe-dev.cherryshoe.com
    ssh -L 8890:127.0.0.1:9230 cs_admin@cherryshoe-dev.cherryshoe.com
    ssh -L 8891:127.0.0.1:9231 cs_admin@cherryshoe-dev.cherryshoe.com

You can now attach a debugger client of choice to the X processes, as if the Node.js application was running locally. I will use Chrome DevTools as an example.

Open Chrome and enter "chrome://inspect" into the URL

Click "Discover network targets" Configure button and configure each of the ports to attach to:
  • Enter "127.0.0.1:8889"
  • Enter "127.0.0.1:8890"
  • Enter "127.0.0.1:8891"


You should now see the 3 processes attached in the debugger client:


Click the "inspect" link for one of the processes, this will open up the DevTools for Node. Under "Sources" tab you can click "Ctrl-P" to open up a file of choice to debug that is attached to the process. You do NOT need to 'Add folder to workspace'.

Open up each remaining process by clicking the "inspect" link.

Invoke the Node application, i.e. call a REST endpoint that it responds to
One of the worker processes will process the request. Any breakpoints will be reached and any console logs will be printed.

You have to open a window for each worker process running because you don't know which process will get picked and if you don't have them all open you can miss it and think debugging isn't working!

If you restart the Node backend, the DevTools Targets will recognize the new process IDs, but the DevTools windows won't. Therefore, you need to open up the DevTools windows again for each process via the "inspect" link.


PM2 cluster mode is used to configure cluster processes
NOTE: It was discovered that PM2 cluster mode and starting node via npm do not play nicely. The first process would listen on port X, but each subsequent process would error out saying port X was in use. Because of this, the node application was changed to start directly by invoking the appropriate js script.


Sunday, November 17, 2013

Alfresco auto-update failed error message

In my last blog, I introduced a complete backup and restore strategy for Alfresco and a custom application that has synchronized data (Alfresco holds repository documents, Application DB holds application specific data).

PROBLEM BACKGROUND:
Another issue I ran into during that mini-project was, often, when starting Alfresco 4.1.2 on a clustered environment I would see the following in the tomcat log:

ERROR [web.context.ContextLoader] [main] Context initialization failed
 org.alfresco.error.AlfrescoRuntimeException: 09100000 Schema auto-update failed
        at org.alfresco.repo.domain.schema.SchemaBootstrap.onBootstrap(SchemaBootstrap.java:1671)
        at org.springframework.extensions.surf.util.AbstractLifecycleBean.onApplicationEvent(AbstractLifecycleBean.java:56)

The key issue from the above error is "Schema auto-update failed".  I didn't find much that could help me on internet forums, so wanted to share what worked.  It turns out that error happens because when the Alfresco database schema is restored, Alfresco thinks that the database schema is updated.  I cannot completely explain this nuance, but in this particular case, the schema had in fact, NOT been updated at all.  Just completely restored from a backup.

SOLUTION:
The clustered Alfresco configuration is using tomcat datasource configuration, so this was the solution that "tricked" Alfresco into thinking that the database schema had NOT changed.

  1. For the type="javax.sql.DataSource" resource attribute, add schema.update="false" THE FIRST TIME you start Alfresco after restoring the schema from the backup.
  2. After Alfresco completely starts up successfully, stop it.
  3. Change schema.update="true" again since Alfresco already started up successfully, so it doesn't think the database got updated.
  4. Start Alfresco again.

Details below...
  • Configure datasource properties in <tomcathome>/conf/server.xml. 
  After the <GlobalNamingResources><Resource /> element, add the following.
  Keep each attribute on its own line.
<Resource auth="Container"
 defaultAutoCommit="false"
 defaultReadOnly="false"
 debug="10"
          driverClassName="oracle.jdbc.OracleDriver"
          factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
 fairQueue="false"
 initSQL="alter session set nls_timestamp_tz_format = &apos;DD-MON-RR HH24.MI.SSXFF TZR&apos;"
 initialSize="3"
 jdbcInterceptors="ConnectionState;StatementFinalizer"
 jmxEnabled="true"
 logAbandoned="false"
 maxActive="15"
 maxIdle="10"
 maxWait="10000"
 minEvictableIdleTimeMillis="5000"
 minIdle="5"
 name="alfresco-datasource"
 password="<db_password>"
 removeAbandoned="true"
 removeAbandonedTimeout="60"
          testOnBorrow="true"
 testOnReturn="false"
 testWhileIdle="true"
 timeBetweenEvictionRunsMillis="7000"
 schema.update="<true|false>" type="javax.sql.DataSource"
 url="jdbc:oracle:thin:@127.0.0.1:1521:XE"
 useEquals="false"
 username="<db_username>"
 validationInterval="30000"
 validationQuery="select 1 from dual"/>
       
The following were also configured...
- Configure resource in <tomcathome>/conf/context.xml.
  Before the </Context> element, add the following.
  Keep each attribute on its own line.
         
    <ResourceLink global="alfresco-datasource"
                  name="jdbc/dataSource"
                  type="org.apache.tomcat.jdbc.pool.DataSource"/>
               
- Since we're running Tomcat 6, need to add tomcat-jdbc.jar to the <tomcathome>/lib folder.

** UPDATE:  Since we are not upgrading the alfresco version, you can safely always set schema.update="false" in server.xml until an upgrade actually needs to be performed.  Reference http://wiki.alfresco.com/wiki/Schema_Upgrade_Scripts.

Thursday, October 10, 2013

Issues with migrating data between oracle versions

A little background...
I implemented a complete backup and restore strategy for Alfresco and a custom application that have synchronized data (Alfresco holds repository documents, Application DB holds application specific data). This was done with Alfresco 4.1.2 on a clustered environment, Oracle backend for both database schemas, and SOLR for search subsystem; all running on two CentOS VMs. The database backups and recoveries were implemented with Oracle Datapump.   The database operations were all invoked remotely on the database server and used SSH Public Key Based Authentication for a completely automated solution.   The same backup and recovery scripts also work on a local Ubuntu environment with a default Alfresco 4.1.2 installation with Oracle; Oracle was installed on the same VM.

...and then I had an issue
I ran into a snag when I tried restoring the custom application database export on on a different server than where I took the backup.  The source expdp'ed dump file was created from "Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production".  I needed to import it into my destination local Ubuntu environment which is "Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production".

Notice the versioning difference.  I didn't think a sub-sub version would cause any problems, little did I know...

Steps to figure out the problem:
  1. I ran the following command on my Ubuntu destination server with the user that owned the schema.  I had to remap_schema and remap_tablespace since source and destination schema and tablespace names were different.
    impdp $SQL_USER/$SQL_PASSWORD@$SQL_CONN remap_schema=<source_schema>:$SQL_USER remap_tablespace=<source_tablespace>:$SQL_TABLESPACE full=N directory=datapump_dir dumpfile=$DUMP_FILENAME logfile=$DUMP_FILENAME.imp.log
    

  2. The import log split out the diagnoses below:
    ORA-39097: Data Pump job encountered unexpected error -1031
    ORA-39065: unexpected master process exception in DISPATCH
    ORA-01031: insufficient privileges
    THANKS A LOT ORACLE - sarcastically!  ORA-01031: insufficient privileges really tells me a whole lot (Mind you I had granted various permissions and my Alfresco restore had worked just fine with the same generic scripts).

  3. I then ran the import as sysdba.  I tried implementing this by granting the user dba priviledges but that did not work, perhaps I will try that again later...
    impdp \"/ as sysdba\" remap_schema=<source_schema>:$SQL_USER remap_tablespace=<source_tablespace>:$SQL_TABLESPACE full=N directory=datapump_dir dumpfile=$DUMP_FILENAME logfile=$DUMP_FILENAME.imp.log
    

  4. MUCH BETTER log file this time:
    ORA-39097: Data Pump job encountered unexpected error -30094
    ORA-39065: unexpected master process exception in DISPATCH
    ORA-30094: failed to find the time zone data file for version 11 in $ORACLE_HOME/oracore/zoneinfo

Solution:
This article helped me figure out how to solve the problem.  Since we are restoring from "Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production" to "Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production", the timezone versions were off.  Specifically the source database was on version 11, where the destination database was on version 13 (You can check with SELECT version FROM v$timezone_file;).  This matters only because the database schema has column types of TIMESTAMP(6) WITH TIME ZONE.
  • Copy over the timezone version pair that was missing from the source to the destination $ORACLE_HOME/oracore/zoneinfo folder.  In this particular example, the missing files were timezlrg_11.dat timezone_11.dat.
  • Change the permissions of the files to match the other .dat files in the directory.
  • Restart Oracle.
The import will now work with the impdp command from Step 2 above!