Continuous Integration in Pipeline as Code Environment with Jenkins, JaCoCo, Nexus and SonarQube

Github Link for the source code: https://github.com/vishwakarmarhl/jenkinstest

Here we discuss the setup for a Continuous integration pipeline. This is for mavenized Spring boot build with JaCoCo coverage reports and Sonar metrics. I used a windows machine with Tomcat 8 for hosting jenkins, but similar setup can be done on any OS where Sonar server can run on the same system.

A. Get the following artifacts on the system

  1. Tomcat server with Java JDK – Configure the server.xml to run on port 8099
  2. Setup Maven & other build utilities on your machine
  3. Access to Github source code
  4. Source code should have the Jenkinsfile in project root to be used by the pipeline
  5. Source should have the sonar-project.properties in project root for the SonarQube project linkage & source paths

JenkinsFile

Jenkinsfile and sonar-project.properties snapshot

B. Setup & Startup SonarQube

  1. Download the SonarQube package from https://www.sonarqube.org/#downloads
  2. Start sonar server: SONAR_HOME\bin\windows-x86-32\StartSonar.bat (for 32 bit Windows)
  3. Open Sonar admin page “http://localhost:9000“. Default credentials – admin/admin
  4. Create user in security tab and generate an access token, 50997f4a8c26d5698cccee30cf398c0ed9b98de0
  5. Create a project SPRINGBOOT with a key
  6. Download SonarQube scanner from https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner
  7. Additional configuration from https://docs.sonarqube.org/display/SCAN/Advanced+SonarQube+Scanner+Usages

C. Setup & Startup Tomcat

  1. Download jenkins.war from https://jenkins.io/download
  2. Put the jenkins.war file in webapps folder of Tomcat home
  3. Set Environment Variables as follows,
  4. SET JENKINS_HOME=”C:/Users/vishwaka/Documents/Workspace/git/jenkinstest/cisetup/jenkins_home”
  5. SET CATALINA_OPTS=”-DJENKINS_HOME=C:/Users/vishwaka/Documents/Workspace/git/jenkinstest/cisetup/jenkins_home”
  6. Start the server using startup.bat

JenkinsHome

Initial launch of Jenkins

D. Initialize Jenkins

  1. Access Jenkins at http://localhost:8099/jenkins
  2. Provide the initial credentials from jenkins_home/secrets/initialPassword*
  3. Install the default set of plugins and proceed
  4. Create a user for this installation
  5. Use “New Item” for creating a pipeline and provide the Jenkinsfile pipeline script from Git SCM for this

JenkinsCreatePipeline

Create pipeline project

E. Plugin & Configuration to Jenkins

  1. Add the “JaCoCo plugin” through the Manage Jenkins > Manage Plugins and install without restart
  2. Add “SonarQube Scanner for Jenkins” through the same Plugin Manager as above
  3. Go to the Manage Jenkins > Configure system and provide the credentials for Sonar Server
  4. Add the “SonarQube Server” name running on URL http://localhost:9000 alongwith user authentication key generated in SonarQube Server user administration page
  5. Remove the auto install option and add the “Sonar Scanner” env variable SONAR_RUNNER_HOME installation path as $JENKINS_HOME/sonar-scanner-3.0.3.778-windows through “Global Tool Configuration”
  6. Make sure the Sonar scanner path is configured properly as its path is hard coded in Jenkinsfile.

JenkinsGlobalProperties

Global Tool Configuration

F. Run the Build now for this pipeline

  1. The pipeline is at http://localhost:8099/jenkins/job/JENKINS-BOOT/JenkinsStatusPipeline
  2. Checkout the coverage report within the pipeline reports JenkinsJacoco
  3. You can also look at the Sonar reports at http://localhost:9000/dashboard?id=JENKINSBOOT JenkinsToSonar
  4. If you have many such projects then its better to execute all your Job Pipelines from a parent Job Pipeline. You can create one and call it “BUILD-ALL-JOBS”. It can be configured using the below pipeline script to run your JENKINS-BOOT job described in the example above as well as any other fictitious job call JENKINS-BOOT-XXX.
node {
    stage('JENKINS-BOOT-STAGE-A') {
        build job: 'JENKINS-BOOT'
    }
    stage('JENKINS-BOOT-STAGE-B') {
        build job: 'JENKINS-BOOT-XXX'
    }
}

There are plugins to build jobs in parallel as well but that depends on what workflow you want to build in your system.

G. Adding Nexus repository management capability to your CI environment from my blog

Click on the text link below:

Repository Management with Nexus 3 for your Mavenized project, including release and snapshot distribution

H. Finally put everything into a script that can run it all

Pardon my naive & careless script, considering my setup is on a local windows development workstation.

@echo off
echo "--------------------------------------------------------------------------"
echo "------------------------- CI STARTUP SCRIPT ------------------------------"
echo "--------------------------------------------------------------------------"

echo "Startup SonarQube Server"
echo "------------------------"
START CMD /C "cd c:\Dock\ci\sonar\sonarqube-6.4\bin\windows-x86-64 & CALL StartSonar.bat"
echo "Sonar may be up on http://localhost:9000/"

echo "Startup Nexus Repository Manager"
echo "--------------------------------"
START CMD /C "cd c:\Dock\ci\nexus\nexus-3.3.1-01\bin & nexus.exe /run"
echo "Nexus may be up on http://localhost:8081/"

echo "Startup Jenkins on Tomcat"
echo "-------------------------"
START CMD /C "cd c:\Dock\ci\jenkins\apache-tomcat-8.5.15\bin & startup.bat"
echo "Jenkins may be up on http://localhost:8099/jenkins"

echo "-------------------------------- END -------------------------------------"

 

Thanks.

 

Advertisements

A command list for GIT based distributed version control and development

GIT is an important distributed version control discipline that any developer should adhere to. It gives you a command line based perspective to development and organizes the basic operations. As a developer one should be familiar with the daily chores of pull, commit, push local as well as on remote repositories.

Git-Logo-1788C

Just before jumping into any learning its mandatory that you really understand the terminology and githubs help page is really helpful there. https://help.github.com/articles/github-glossary/

When you start working with Git then there is a standard sequence that one tends to follow. I found http://gitref.org useful. Anyways, here is a link which shows similar workflow as below and was edited on http://www.draw.io

Overall GIT versioning workflow diagram

Overall GIT versioning workflow diagram

General Guideline

  1. In order to achieve a sanitized version control we should always maintain a master and a develop branch in the repository in https://github.com/rawnics/ships.git, as is shown in the figure above with their respective git commands in a table below. The PROJ-R1 has a master stable branch and a development branch which is constantly in progress.
  2. The development proceeds in the develop branch and the code here is unit tested, verified and committed by the developer (3):Figure
  3. The develop branch is stable in capacity of nightly build and snapshot development release
  4. The master branch is the production release branch which contains stable release quality code
  5. According to plan as and when development matures into a release we merge that branch to master (4):Figure
  6. In case of a feature product say PROJ-R6, that we want to develop from the PROJ-R1 codebase we create a branch dev_r6 for that (5):Figure
  7. Once this feature product matures and there remains no commonality. We may move it to a separate repository altogether for further development with similar master/develop branch structure
  8. Tag is the most important identifier in the laundry list of commits & push. It should definitely be used for production/master branch to track and identify releases

GIT Commands

A. Workflow and commands for branching to develop and merging back to master repository
(1) Initialize repository
      Initialize a repo       : git init
      Add a remote repo : git remote add origin https://github.com/rawnics/ships.git
      Pull the master       : git pull origin master
(2) Branch checkout,
     List branches           : git ls-remote origin
     Checkout branch     : git checkout -b develop master
     Select branch           : git checkout develop
(3)  Code Changes      : echo “Development branch code base changed” >> Ships\README.md
       (3.1) Index              : git add .
       (3.2) Commit          : git commit -a -m “Development branch code base”
       (3.3) Remote push : git push origin develop
(4) Merge
     (4.1) Compare branch : git diff master develop
     (4.2) Merge develop    : git checkout master
                                            git merge develop
     (4.3) Remote push      : git push
     (4.4) Delete branch     : git branch -d develop
(5) Create feature product branch
(6) Tag branch version
    (6.1) Tag annotated    : git tag “1.0.0” -m “Beta 1.0.0”
    (6.2) Push to tag         : git push origin tag “1.0.0”

    (6.3) Checkout tag      : git checkout “1.0.0”

B. Workflow to understand the basic checkout from and to master repo without branching

Initialize a repo          : git init
Add a remote repo    : git remote add origin https://github.com/rawnics/ships.git
Pull the master          : git pull origin master
Do your changes      : echo “Test Entry for demonstrating change” > Ships\README.md
Stage files                 : git add .
Commit file                : git commit -a -m “new buggy commit of README.md”
Pushed the change   : git push origin master
Check status            : git status
C. Workflow to contribute to a repository by Fork and Pull Request scenario

Fork the repository you want to contribute to in your github account. This setup will have an upstream repository which you just forked, an origin repository which is pointing to your github account and your local master versioned repository branch.
(1) Setup
Initialize a repo             : git init
Add a remote repo        : git remote add origin https://github.com/rawnics/ships.git
Add a upstream repo    : git remote add upstream https://github.com/AWNICS/ships.git
Merge the master          : git pull origin master
Merge the upstream     : git pull upstream master
(2) Checkout Branch
# Start working with the develop branch with a pull-merge
Checkout a branch        : git checkout develop
Merge/Pull a branch     : git pull upstream develop
(3) Commit & Push
# Commit and push branches to your origin like in Section B
git add .
git commit -m “Updated the code” –author=”Rahul Vishwakarma <rahul@awnics.com>”
git push origin develop
git push origin master
(4) Pull Request
 # Github repository
 Create a pull request for AWNICS repository from your Github page

Now there are some jargons that one needs to be familiar with and that can be pulled from the help. I was scouring the net and this video looked really promising.

In case you want an exhaustive description with some fundamentals.