Continuous Integration in Pipeline as Code Environment with Jenkins, JaCoCo, Nexus and SonarQube

Github Link for the source code: https://github.com/vishwakarmarhl/jenkinstest

Here we discuss the setup for a Continuous integration pipeline. This is for mavenized Spring boot build with JaCoCo coverage reports and Sonar metrics. I used a windows machine with Tomcat 8 for hosting jenkins, but similar setup can be done on any OS where Sonar server can run on the same system.

A. Get the following artifacts on the system

  1. Tomcat server with Java JDK – Configure the server.xml to run on port 8099
  2. Setup Maven & other build utilities on your machine
  3. Access to Github source code
  4. Source code should have the Jenkinsfile in project root to be used by the pipeline
  5. Source should have the sonar-project.properties in project root for the SonarQube project linkage & source paths

JenkinsFile

Jenkinsfile and sonar-project.properties snapshot

B. Setup & Startup SonarQube

  1. Download the SonarQube package from https://www.sonarqube.org/#downloads
  2. Start sonar server: SONAR_HOME\bin\windows-x86-32\StartSonar.bat (for 32 bit Windows)
  3. Open Sonar admin page “http://localhost:9000“. Default credentials – admin/admin
  4. Create user in security tab and generate an access token, 50997f4a8c26d5698cccee30cf398c0ed9b98de0
  5. Create a project SPRINGBOOT with a key
  6. Download SonarQube scanner from https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner
  7. Additional configuration from https://docs.sonarqube.org/display/SCAN/Advanced+SonarQube+Scanner+Usages

C. Setup & Startup Tomcat

  1. Download jenkins.war from https://jenkins.io/download
  2. Put the jenkins.war file in webapps folder of Tomcat home
  3. Set Environment Variables as follows,
  4. SET JENKINS_HOME=”C:/Users/vishwaka/Documents/Workspace/git/jenkinstest/cisetup/jenkins_home”
  5. SET CATALINA_OPTS=”-DJENKINS_HOME=C:/Users/vishwaka/Documents/Workspace/git/jenkinstest/cisetup/jenkins_home”
  6. Start the server using startup.bat

JenkinsHome

Initial launch of Jenkins

D. Initialize Jenkins

  1. Access Jenkins at http://localhost:8099/jenkins
  2. Provide the initial credentials from jenkins_home/secrets/initialPassword*
  3. Install the default set of plugins and proceed
  4. Create a user for this installation
  5. Use “New Item” for creating a pipeline and provide the Jenkinsfile pipeline script from Git SCM for this

JenkinsCreatePipeline

Create pipeline project

E. Plugin & Configuration to Jenkins

  1. Add the “JaCoCo plugin” through the Manage Jenkins > Manage Plugins and install without restart
  2. Add “SonarQube Scanner for Jenkins” through the same Plugin Manager as above
  3. Go to the Manage Jenkins > Configure system and provide the credentials for Sonar Server
  4. Add the “SonarQube Server” name running on URL http://localhost:9000 alongwith user authentication key generated in SonarQube Server user administration page
  5. Remove the auto install option and add the “Sonar Scanner” env variable SONAR_RUNNER_HOME installation path as $JENKINS_HOME/sonar-scanner-3.0.3.778-windows through “Global Tool Configuration”
  6. Make sure the Sonar scanner path is configured properly as its path is hard coded in Jenkinsfile.

JenkinsGlobalProperties

Global Tool Configuration

F. Run the Build now for this pipeline

  1. The pipeline is at http://localhost:8099/jenkins/job/JENKINS-BOOT/JenkinsStatusPipeline
  2. Checkout the coverage report within the pipeline reports JenkinsJacoco
  3. You can also look at the Sonar reports at http://localhost:9000/dashboard?id=JENKINSBOOT JenkinsToSonar
  4. If you have many such projects then its better to execute all your Job Pipelines from a parent Job Pipeline. You can create one and call it “BUILD-ALL-JOBS”. It can be configured using the below pipeline script to run your JENKINS-BOOT job described in the example above as well as any other fictitious job call JENKINS-BOOT-XXX.
node {
    stage('JENKINS-BOOT-STAGE-A') {
        build job: 'JENKINS-BOOT'
    }
    stage('JENKINS-BOOT-STAGE-B') {
        build job: 'JENKINS-BOOT-XXX'
    }
}

There are plugins to build jobs in parallel as well but that depends on what workflow you want to build in your system.

G. Adding Nexus repository management capability to your CI environment from my blog

Click on the text link below:

Repository Management with Nexus 3 for your Mavenized project, including release and snapshot distribution

H. Finally put everything into a script that can run it all

Pardon my naive & careless script, considering my setup is on a local windows development workstation.

@echo off
echo "--------------------------------------------------------------------------"
echo "------------------------- CI STARTUP SCRIPT ------------------------------"
echo "--------------------------------------------------------------------------"

echo "Startup SonarQube Server"
echo "------------------------"
START CMD /C "cd c:\Dock\ci\sonar\sonarqube-6.4\bin\windows-x86-64 & CALL StartSonar.bat"
echo "Sonar may be up on http://localhost:9000/"

echo "Startup Nexus Repository Manager"
echo "--------------------------------"
START CMD /C "cd c:\Dock\ci\nexus\nexus-3.3.1-01\bin & nexus.exe /run"
echo "Nexus may be up on http://localhost:8081/"

echo "Startup Jenkins on Tomcat"
echo "-------------------------"
START CMD /C "cd c:\Dock\ci\jenkins\apache-tomcat-8.5.15\bin & startup.bat"
echo "Jenkins may be up on http://localhost:8099/jenkins"

echo "-------------------------------- END -------------------------------------"

 

Thanks.

 

Reduce Insecurity for free, HTTPS now democratized by LetsEncrypt

 

how-to-install-lets-encrypt-ssl-certificate

I am typing this with my fingers crossed, that I could just get someone to acknowledge that HTTPS is not only prudent but damn easy to setup. Security is not my primary focus, yet I align with most of the InfoSec’s paranoia out there today. A dumb hacker millions of years ago said that, the minimum you could do in security is to use SSL encryption in your communication.

Now that I have uncrossed my fingers. Below is my rough note for setting up a secure instance for which I assume you have an elastic IP in EC2 instance and a DNS pointing using the A-host configuration to this IP. Below is a totally fake xyzminime.org domain name which I do not own and is just used for example. No offence to anyone who owns it, I just think its an awesome name.

URL: https://xyzminime.org

Email: info@xyzminime.org

# This is how I used to generate my insecure self-signed certificate earlier
keytool -genkeypair -dname "CN=xyzminime.org, OU=XYZ, O=XYZ, L=PaloAlto, ST=CA, C=US" -alias xyzminime -keyalg RSA -ext san=ip:xyzminime.org -keystore /opt/tomcat7/.keystore

Since I am not an authorized certificate signing authority, all the browsers just flags my certificate as unsecure and block it by default.

This is where LetsEncrypt came for help with their democratic certificate authority. There were some references that I drew inspiration from, to do this thing as a rough note and not a tutorial.

Ref:    https://certbot.eff.org/#centosrhel6-other
        https://certbot.eff.org/docs/using.html#webroot
        https://melo.myds.me/wordpress/lets-encrypt-for-tomcat-7-on-ds/

 

1.) Pre-requisite is to get the certbot client

# Installation taken care by the certbot-auto client
sudo yum install epel-release wget
wget https://dl.eff.org/certbot-auto
chmod a+x certbot-auto

# Install the certbot on your instance
sudo ./certbot-auto

Certbot dumps its contents in a folder like below, in my ec2-user local path,
/home/ec2-user/.local/share/letsencrypt/bin/letsencrypt certonly

 

2.) Generate a certificate

Keep an email address for notification and validation handy for the enrolment with ACME

If you have a functional webserver that needs to be SSLified then use the webroot way otherwise --standalone is preffered
sudo ./certbot-auto certonly -n --rsa-key-size 2048 --agree-tos --email info@xyzminime.org --webroot -w /opt/tomcat7/webapps/ -d xyzminime.org
IMPORTANT NOTES:
– Congratulations! Your certificate and chain have been saved at
/etc/letsencrypt/live/xyzminime.org/fullchain.pem.
Your cert will expire on 2016-12-22. To obtain a new or tweaked
version of this certificate in the future, simply run certbot-auto
again. To non-interactively renew *all* of your certificates, run
“certbot-auto renew”
– If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let’s Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le
The certificates are written to /etc/letsencrypt/live/xyzminime.org/
export CERT_PATH="/etc/letsencrypt/live/xyzminime.org/"
 - cert.pem
 - chain.pem
 - fullchain.pem
 - privkey.pem

 

3.) Create a keystore for Tomcat

Basically there are only two steps required to get our fullchain.pem and privkey.pem inside a JKS. 
First we bundle both our fullchain and the private key in a PKCS12 keystore. 
We do this, because apparently Java’s keytool (which we use to create our JKS),
is not able to import pre-existing keys and certificates into a JKS, as described here.
sudo openssl pkcs12 -export -in "$CERT_PATH"fullchain.pem -inkey "$CERT_PATH"privkey.pem -out fullchain_and_key.p12 -name xyzminime -password pass:mini#123

Now that we have our PKCS12 keystore, we can use Java’s keytool to generate a JKS,
from our PKCS12 file like;
keytool -importkeystore -deststorepass mini#123 -destkeypass mini#123 -destkeystore xyzminime.jks -srckeystore fullchain_and_key.p12 -srcstoretype PKCS12 -srcstorepass mini#123 -alias xyzminime
 
# Backup and place your self-signed keystore in the tomcat home
mv /opt/tomcat7/.keystore .keystore_backup_1
sudo cp xyzminime.jks /opt/tomcat7/.keystore
 
Make sure that the 8443 conector configuration in the conf/server.xml is as follows
<Connector port="8443" 
           keystoreFile="${user.home}/.keystore" 
           keystorePass="mini#123" 
           keyAlias="xyzminime" 
           ...
Run the InstallCert utility for java security ca cert 
Compile the InstallCert using javac
java InstallCert xyzminime.org
sudo cp jssecacerts /usr/java/jdk1.8.0_73/jre/lib/security/

 

4.) Automating renewal

# A test run for renewal
certbot-auto renew --dry-run

# Add the following to the cron or systemmd that should run twice daily in case of any certificate invalidation
certbot-auto renew --quiet

 

Now your tomcat will be able to serve the content over SSL. Verify this by accessing the server on the below URL.

https://xyzminime.org

 

 

 

A quick setup for tomcat 7 on CentOS 6. Also, added the SSL configuration with self-signed certificates to run tomcat 7 on HTTPS secured SSL layer

Setup tomcat

1.) Pre-requisite:

Since Java is a major requirement

$ yum install java-1.7.0-openjdk-devel.x86_64

Add the JAVA_HOME environment variable to ~/.bashrc file 
  #Env variables for java
  export JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk.x86_64
  export CATALINA_HOME=/opt/tomcat7
  export PATH=$PATH:$JAVA_HOME/bin

Open the ports that will be used by tomcat for service

Flush the tables before config
$ iptables -F
$ iptables -t nat -F

Now setup INPUT ports
$ iptables -I INPUT -p tcp --dport 8443 -j ACCEPT
$ iptables -I INPUT -p tcp --dport 8080 -j ACCEPT
$ service iptables save
$ service iptables restart

In case we want to route the access from port 80 to tomcats 8080

$ iptables -t nat -I PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8080
$ iptables -t nat -I OUTPUT -p tcp --dport 80 -j REDIRECT --to-ports 8080

2.) Download and setup tomcat 7

$ wget http://mirrors.gigenet.com/apache/tomcat/tomcat-7/v7.0.62/bin/apache-tomcat-7.0.62.tar.gz
$ tar -xvzf apache-tomcat-7.0.62.tar.gz
$ mv apache-tomcat-7.0.62 tomcat7
$ mv tomcat7/ /opt/

3.) Create a tomcat specific user and user group. Since the tomcat would be running from a script it should not be root user.

$ groupadd tomcat
$ useradd -g 99 -s /sbin/nologin -d /opt/tomcat7 tomcat
$ passwd tomcat
Adjust Ownership For New Users And Groups. Give the new user access to the tomcat directories. 
$ chown -R tomcat:tomcat /opt/tomcat7
$ chmod 775 /opt/tomcat7/webapps
$ chmod +x /opt/tomcat7/bin/*.sh

4.) Create a startup service script

$ vim /etc/init.d/tomcat
Add the following content to this script
#!/bin/bash
# description: Tomcat Start Stop Restart
# processname: tomcat
# chkconfig: 234 20 80
PATH=$JAVA_HOME/bin:$PATH
export PATH
CATALINA_HOME=/opt/tomcat7
export CATALINA_HOME

case $1 in
start)
   cd $CATALINA_HOME/bin
   /bin/su -s /bin/bash tomcat ./startup.sh
   ;;
stop)
   cd $CATALINA_HOME/bin/
   /bin/su -s /bin/bash tomcat ./shutdown.sh
   ;;
restart)
   cd $CATALINA_HOME/bin/
   /bin/su -s /bin/bash tomcat ./shutdown.sh
   cd $CATALINA_HOME/bin/
   /bin/su -s /bin/bash tomcat ./startup.sh
   ;;
esac
exit 0

5.) Add the tomcat script as a service

$ chmod 755 /etc/init.d/tomcat
$ chkconfig --add tomcat
$ chkconfig --level 234 tomcat on
$ chkconfig --list tomcat

6.) Start/Stop the tomcat service

 $ service tomcat start
 $ service tomcat stop

SSL security with self-signed certificates on tomcat

In order to setup this tomcat on SSL Use the following configuration steps,

1.) Generate a keystore file for this server

This will be used as a self-signed certificate for secured connectivity. 
Default path: /home/%user.home%/.keystore
keytool -genkeypair -dname "CN=127.0.0.1, OU=Rahul, O=Luhar, L=Vishwakarma, ST=Karnataka, C=IN" -alias mysslsecuredserver -keyalg RSA -ext san=ip:127.0.0.1

2.) Add the relevant configuration to the tomcats https connector in conf/server.xml

 maxThreads="150" scheme="https" secure="true"
 clientAuth="false" sslProtocol="TLS" keystoreFile="${user.home}/.keystore" keystorePass="mypassman"/>

3.) Add the server IP to the truststore in order to allow for this self signed certificate

Use the InstallCert.java to add the IP to the trusted store
https://github.com/vishwakarmarhl/javahelper/blob/master/InstallCert.java
Compile InstallCert.java. Run the following two commands to generate jssecacerts binary. 127.0.0.1 is the web servers IP.
$ java InstallCert 127.0.0.1:8443
Copy the generated jssecacerts in this path to %JAVA_HOME%\jre\lib\security

You can also export and import the generated certificate from the keystore with the password and share it with other systems on the network that negotiates with this server.

$ keytool -export -alias mysslsecuredserver -file mysslsecuredserver.cer
  $ keytool -import -trustcacerts -alias mysslsecuredserver -file mysslsecuredserver.cer

Verify the tomcat running and secured via HTTPS.

For a proper SSL shared from a hosting provider. Look at the import into the java cacerts

keytool -import -trustcacerts -file NewRootCACertificate.crt -keystore "%JAVA_HOME%\jre\lib\security\cacert"

http://stackoverflow.com/questions/28521266/caused-by-sun-security-provider-certpath-suncertpathbuilderexception-unable-to

Test Link: https://127.0.0.1:8443