Bash scripting - replace URLs for different environments

Unfortunately, due to limitations of the project, a better solution for environment specific settings was not available.  I wrote a bash script to perform automated deployment of a front end angular web application.  There was a client-side javascript file that needed a specific environment specific URL.

Below is the sed command I used inside the bash script to do this.  Instead of using a backslash as the delimiter, I used #, you can use another non-character as well.

Environments tested
CentOS release 6.7

To replace "https://localhost:8080" with "http://dev.cherryshoetech.com" in the cherryshoetech.js file:

• When you don't need to use shell variables, the following works:

sed -i 's#https://localhost:8080#http://dev.cherryshoetech.com#g' cherryshoetech.js

• When you need to use shell variables, the following works.  Notice the double quotes are used for interpolation:

sed -i "s#$OLD_ENVIRONMENT_URL#$NEW_ENVIRONMENT_URL#g" cherryshoetech.js

Also used the following to grep for specific process "gulp" and kill in the same line.  This article was of great help.
kill $(ps aux | grep '[g]ulp' | awk '{print $2}')

Splitting of Large Text Files

Opening up large log files from a production environment can be difficult (or rather impossible depending on your system resources) with apps or editors like notepad or vi.  There are plenty of free apps out there to split text files up, but you can't install them if you don't have admin priviledges (which was my case for a Windows 7 laptop).

Luckily, I did have cygwin installed, so I could run some unix commands to help with my dilemna.  I found some great articles online, but ultimately the following was the command that I ended up using
that helped me a great deal.

awk '{outfile=sprintf("HUGEFILE.log.2015-04-28-%02d.txt",NR/2000000+1);print > outfile}' HUGEFILE.log.2015-04-28

The HUGEFILE.log.2015-04-28 needs to be replaced twice.  The first one is the output file(s) name, which is the filename + a number indicator if you will have multiple output files.  The second is the filename you are splitting.

NR is an awk built-in variable for the number of records.  For example: if HUGEFILE.log.2015-04-28 file had 162 lines in it, and we replaced 2000000 with 50, we would get four output files.  The first three would have 50 lines in it (with an additional empty line at the end from the +1), and the last one 12 lines (with an additional empty line at the end from the +1).

The 2000000 is the number of lines that I determined my editor (cygwin vi) could open in a relatively reasonable amount of time.  The 2000000 number of lines is approximately a 500 MB size file.

So after running the above command on my 4.2GB original file, I got 9 output files of ~500MB each, except the last file which was 140MB.  A lot easier to open those files to peruse the logs!

Issues with migrating data between oracle versions

A little background...
I implemented a complete backup and restore strategy for Alfresco and a custom application that have synchronized data (Alfresco holds repository documents, Application DB holds application specific data). This was done with Alfresco 4.1.2 on a clustered environment, Oracle backend for both database schemas, and SOLR for search subsystem; all running on two CentOS VMs. The database backups and recoveries were implemented with Oracle Datapump.   The database operations were all invoked remotely on the database server and used SSH Public Key Based Authentication for a completely automated solution.   The same backup and recovery scripts also work on a local Ubuntu environment with a default Alfresco 4.1.2 installation with Oracle; Oracle was installed on the same VM.

...and then I had an issue
I ran into a snag when I tried restoring the custom application database export on on a different server than where I took the backup.  The source expdp'ed dump file was created from "Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production".  I needed to import it into my destination local Ubuntu environment which is "Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production".

Notice the versioning difference.  I didn't think a sub-sub version would cause any problems, little did I know...

Steps to figure out the problem:

1. I ran the following command on my Ubuntu destination server with the user that owned the schema.  I had to remap_schema and remap_tablespace since source and destination schema and tablespace names were different.

   impdp $SQL_USER/$SQL_PASSWORD@$SQL_CONN remap_schema=<source_schema>:$SQL_USER remap_tablespace=<source_tablespace>:$SQL_TABLESPACE full=N directory=datapump_dir dumpfile=$DUMP_FILENAME logfile=$DUMP_FILENAME.imp.log
   


The import log split out the diagnoses below:
ORA-39097: Data Pump job encountered unexpected error -1031
ORA-39065: unexpected master process exception in DISPATCH
ORA-01031: insufficient privileges
THANKS A LOT ORACLE - sarcastically!  ORA-01031: insufficient privileges really tells me a whole lot (Mind you I had granted various permissions and my Alfresco restore had worked just fine with the same generic scripts).


2. I then ran the import as sysdba.  I tried implementing this by granting the user dba priviledges but that did not work, perhaps I will try that again later...

   impdp \"/ as sysdba\" remap_schema=<source_schema>:$SQL_USER remap_tablespace=<source_tablespace>:$SQL_TABLESPACE full=N directory=datapump_dir dumpfile=$DUMP_FILENAME logfile=$DUMP_FILENAME.imp.log
   


MUCH BETTER log file this time:
ORA-39097: Data Pump job encountered unexpected error -30094
ORA-39065: unexpected master process exception in DISPATCH
ORA-30094: failed to find the time zone data file for version 11 in $ORACLE_HOME/oracore/zoneinfo

Solution:
This article helped me figure out how to solve the problem.  Since we are restoring from "Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production" to "Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production", the timezone versions were off.  Specifically the source database was on version 11, where the destination database was on version 13 (You can check with SELECT version FROM v$timezone_file;).  This matters only because the database schema has column types of TIMESTAMP(6) WITH TIME ZONE.

• Copy over the timezone version pair that was missing from the source to the destination $ORACLE_HOME/oracore/zoneinfo folder.  In this particular example, the missing files were timezlrg_11.dat timezone_11.dat.
• Change the permissions of the files to match the other .dat files in the directory.
• Restart Oracle.

The import will now work with the impdp command from Step 2 above!

pushd and popd directory stack commands

Traversing between multiple directories during software development can be irritating when the directory paths get longer than a few levels deep.  You'll likely need to move between where your source code is, a target directory where the source code is built, and then the deployment directory.

Instead of using the tried and true 'cd' and 'cd -' commands to only get back to the last directory you were in, pushd and popd are two easy commands for directory stack manipulation.  It's exactly what it sounds like - a stack, last in/first out.  It works in unix and dos, but my simple example here will be with cygwin installed on Windows 7.

$ pushd /home/cherryshoe/workspaces/project1
/home/cherryshoe/workspaces/project1 <wherever you were when you started>
$ mvn clean install
$ pushd target
/home/cherryshoe/workspaces/project1/target /home/cherryshoe/workspaces/project1 <wherever you were when you started>
$ cp SNAPSHOT.war /opt/app/apache-tomcat/webapp
$ pushd /opt/app/apache-tomcat/bin
/opt/app/apache-tomcat/bin /home/cherryshoe/workspaces/project1/target /home/cherryshoe/workspaces/project1 <wherever you were when you started>
$ startup.sh&
$ popd
/home/cherryshoe/workspaces/project1/target /home/cherryshoe/workspaces/project1 <wherever you were when you started>
$ popd
/home/cherryshoe/workspaces/project1 <wherever you were when you started>
$ popd
<wherever you were when you started>
$ dirs -v
 0 <wherever you were when you started>

You'll notice between each pushd and popd command, it'll show you what is currently on the stack.  If you want to suppress that you can redirect it to /dev/null.  You can then use 'dirs -v' to see there is no paths left on the stack

You can also save build and deployment time by using an class-reloader like JRebel, which I discussed a few posts ago.