Quick Apt Repository way – NVIDIA CUDA 9.x on Ubuntu 18.04 LST installation

The same NVIDIA CUDA 9.1 setup on Ubuntu 18.04 LST using the aptitude repository. However this appears to work and is simple to work with. Reference is taken from this askubuntu discussion.

Lookup the solution to the Nouveau issue from this blogpost

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo ubuntu-drivers autoinstall
sudo reboot

Now install the CUDA toolkit

sudo apt install g++-6
sudo apt install gcc-6
sudo apt install nvidia-cuda-toolkit gcc-6

Screenshot from 2018-07-13 14-18-16

Screenshot from 2018-07-13 14-16-00

Run the installer

root@wind:~/Downloads# ./cuda_9.1.85_387.26_linux --override

Screenshot from 2018-07-13 14-27-36.png

Screenshot from 2018-07-13 14-28-43

Setup the environment variables

# Environment variables
export PATH=/usr/local/cuda-9.1/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda-9.1/lib64 
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda-9.1/lib

Provide the soft link for the gcc-6 compiler

sudo ln -s /usr/bin/gcc-6 /usr/local/cuda/bin/gcc
sudo ln -s /usr/bin/g++-6 /usr/local/cuda/bin/g++
sudo reboot

Test

cd ~/NVIDIA_CUDA-9.1_Samples/
make -j4

Upon completion of the compilation test using device query binary

$ cd ~/NVIDIA_CUDA-9.1_Samples/bin/x86_64/linux/release
$ ./deviceQuery

Screenshot from 2018-07-13 14-41-49.png

$ sudo bash -c "echo /usr/local/cuda/lib64/ > /etc/ld.so.conf.d/cuda.conf"
$ sudo ldconfig

DONE

Advertisements

NVIDIA CUDA 9.x on Ubuntu 18.04 LST installation

Guide

An installation guide to take you through the NVIDIA graphics driver as well as CUDA toolkit setup on an Ubuntu 18.04 LTS.

A. Know your cards

Verify what graphics card you have on your machine

rahul@karma:~$ lspci | grep VGA
04:00.0 VGA compatible controller: 
NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1)
rahul@karma:~$ sudo lshw -C video
 *-display 
 description: VGA compatible controller
 product: GM204 [GeForce GTX 970]
 vendor: NVIDIA Corporation
 physical id: 0
 bus info: pci@0000:04:00.0
 version: a1
 width: 64 bits
 clock: 33MHz
 capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
 configuration: driver=nouveau latency=0
 resources: irq:30 memory:f2000000-f2ffffff memory:e0000000-efffffff memory:f0000000-f1ffffff ioport:2000(size=128) memory:f3080000-f30fffff

Download the right driver

downloaded the Version 390.67 for GeForce GTX 970

Screenshot from 2018-07-12 17-15-34.png

B. Nouveau problem kills your GPU rush

Hoever there are solutions available

Here is what worked for me

  1. remove all nvidia packages ,skip this if your system is fresh installed
    sudo apt-get remove nvidia* && sudo apt autoremove
    
  2. install some packages for build kernel:
    sudo apt-get install dkms build-essential linux-headers-generic
    
  3. now block and disable nouveau kernel driver:
    sudo vim /etc/modprobe.d/nvidia-installer-disable-nouveau.conf
    

Insert follow lines to the nvidia-installer-disable-nouveau.conf:

blacklist nouveau
blacklist lbm-nouveau
options nouveau modeset=0
alias nouveau off
alias lbm-nouveau off

save and exit.

  1. Disable the Kernel nouveau by typing the following commands(nouveau-kms.conf may not exist,it is ok):
    rahul@wind:~$ echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
    options nouveau modeset=0
    
  2. build the new kernel by:
    rahul@wind:~$ sudo update-initramfs -u
    update-initramfs: Generating /boot/initrd.img-4.15.0-23-generic
    
  3. reboot
Run the Installer in run-level 3
$ sudo init 3 
$ sudo bash
$ ./NVIDIA-Linux-x86_64-390.67.run

Uninstall

More instruction on how to stop using the driver before uninstallation
sudo nvidia-installer –uninstall

C. NVIDIA X Server Settings

Install this from the ubuntu software center.
Screenshot from 2018-07-12 17-23-43.png

D. Start the CUDA related setup

We will need the CUDA toolkit 9.1 which is supported for the GTX 970 version with compute 3.0 capability. So download the local installer for Ubuntu.

Screenshot from 2018-07-13 13-55-24.png

Downloaded the “cuda_9.1.85_387.26_linux.run*” local installation file.

$ sudo add-apt-repository ppa:graphics-drivers/ppa
$ sudo apt install nvidia-cuda-toolkit gcc-6

Steps are taken from the CUDA 9.1 official documentation

  1. Perform the pre-installation actions.
  2.  Disable the Nouveau drivers. We did this in the above driver installation
  3. Reboot into text mode (runlevel 3). This can usually be accomplished by adding the number “3” to the end of the system’s kernel boot parameters. Change the runlevel ‘sudo init 3’, refer
  4. Verify that the Nouveau drivers are not loaded. If the Nouveau drivers are still loaded, consult your distribution’s documentation to see if further steps are needed to disable Nouveau.
  5. Run the installer and follow the on-screen prompts:
$ chmod +x cuda_9.1.85_387.26_linux
$ rahul@wind:~/Downloads$ ./cuda_9.1.85_387.26_linux --override

Screenshot from 2018-07-13 13-52-19.png

Since we already installed the Driver above we say NO in the NVIDIA accelerated graphic driver installation question.

Screenshot from 2018-07-13 13-54-20.png

This will install the CUDA stuff in the following locations

  • CUDA Toolkit /usr/local/cuda-9.1
  • CUDA Samples $(HOME)/NVIDIA_CUDA-9.1_Samples

We can verify the graphic card using the NVIDIA-SMI command.

Screenshot from 2018-07-12 20-02-08

Uninstallation

cd /usr/local/cuda-9.1/bin
sudo ./uninstall_cuda_9.1.pl

 

E. Environment Variables

rahul@wind:~$ vim ~/.bashrc

# Add the following to the environment variables
export PATH=/usr/local/cuda-9.1/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda-9.1/lib64 
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda-9.1/lib

rahul@wind:~$ source ~/.bashrc
rahul@wind:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Tue_Jun_12_23:07:04_CDT_2018
Cuda compilation tools, release 9.1, 

 

F. Test

Ensure you have the right driver versions

rahul@wind:$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 390.67 Fri Jun 1 04:04:27 PDT 2018
GCC version: gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)

Change directory to the NVIDIA CUDA Samples and compile them

rahul@wind:~/NVIDIA_CUDA-9.1_Samples$ make

Now run the device query test

rahul@wind:~/NVIDIA_CUDA-9.1_Samples/bin/x86_64/linux/release$ ./deviceQuery
./deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

 

END

 

Compile and Setup OpenCV 3.4.x on Ubuntu 18.04 LTS with Python Virtualenv for Image processing with Ceres, VTK, PCL

OpenCV: Open Source Computer Vision Library

Links

Documentation: https://docs.opencv.org/3.4.2/

OpenCV Source: https://github.com/opencv/opencv

OpenCV_Logo

A. Setup an external HDD/SSD for this setup

filesystem-partition-ubuntu-external-ssd

B. Environment (Ubuntu 18.04 LTS)

 

Python3 setup

Install the needed packages in a python virtualenv. Refer similar windows Anaconda setup or look at the ubuntu based info here

sudo apt-get install build-essential cmake unzip pkg-config 
sudo apt-get install ubuntu-restricted-extras
sudo apt-get install python3-dev python3-numpy
sudo apt-get install git python3-pip virtualenv
sudo pip3 install virtualenv
rahul@karma:~$ virtualenv -p /usr/bin/python3 cv3
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/rahul/cv3/bin/python3
Also creating executable in /home/rahul/cv3/bin/python
Installing setuptools, pkg_resources, pip, wheel...
done.

Activate and Deactivate the python Environment

rahul@karma:~$ source ~/cv3/bin/activate
(cv3) rahul@karma:~$ python
Python 3.6.5 (default, Apr 1 2018, 05:46:30) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Test",4*5)
Test 20
>>> exit()
(cv3) rahul@karma:~$ deactivate
rahul@karma:~$ 

Alternatively, a great way to use virtualenv is to use Virtualenvwrappers

sudo pip3 install virtualenv virtualenvwrapper

Add these to your ~/.bashrc file

# virtualenv and virtualenvwrapper
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/virtualenvwrapper.sh

Now, run “source ~/.bashrc” to set the environment

Create a Virtual environment
rahul@karma:~$ mkvirtualenv cv3 -p python3
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/rahul/.virtualenvs/cv3/bin/python3
Also creating executable in /home/rahul/.virtualenvs/cv3/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/rahul/.virtualenvs/cv3/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/rahul/.virtualenvs/cv3/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/rahul/.virtualenvs/cv3/bin/preactivate
virtualenvwrapper.user_scripts creating /home/rahul/.virtualenvs/cv3/bin/postactivate
virtualenvwrapper.user_scripts creating /home/rahul/.virtualenvs/cv3/bin/get_env_details
(cv3) rahul@karma:~$
Activate/Deactivate virtual env
rahul@karma:~$ workon cv3
(cv3) rahul@karma:~$ deactivate 
rahul@karma:~$

Install basic packages for the vision work.

(cv3) rahul@karma: pip install numpy scipy matplotlib scikit-image scikit-learn ipython

Java installation from this blog

sudo add-apt-repository ppa:linuxuprising/java
sudo apt update
sudo apt install oracle-java10-installer
sudo apt install oracle-java10-set-default

VTK for SFM Modules

SFM setup: https://docs.opencv.org/3.4.2/db/db8/tutorial_sfm_installation.html

sudo apt-get install libxt-dev libglew-dev libsuitesparse-dev
sudo apt-get install tk8.5 tcl8.5 tcl8.5-dev tcl-dev

Ceres-Solver: http://ceres-solver.org/installation.html

# However, if you want to build Ceres as a *shared* library, 
# You must, add the following PPA:
sudo add-apt-repository ppa:bzindovic/suitesparse-bugfix-1319687
sudo apt-get update
sudo apt-get install libsuitesparse-dev
git clone https://ceres-solver.googlesource.com/ceres-solver
cd ceres-solver
mkdir build && cd build
cmake ..
make -j4
make test
sudo make install

VTK Setup, https://gitlab.kitware.com/vtk/vtk.git

git clone git://vtk.org/VTK.git VTK
cd VTK
mkdir VTK-build
cd VTK-build
cmake ../ -DBUILD_SHARED_LIBS=ON -DBUILD_TESTING=ON \ 
      -DCMAKE_BUILD_TYPE=Release -DVTK_WRAP_PYTHON=ON
make -j4
sudo make install
$ cp -r ~/cv/VTK/VTK-build/lib/python3.6/site-packages/* ~/.virtualenvs/cv3/lib/python3.6/site-packages/
$ export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib:/usr/local/lib"
$ sudo ldconfig

FLANN

http://www.cs.ubc.ca/research/flann/#download
cd flann-1.8.4-src/ && mkdir build && cd build
cmake ..
make -j4 
sudo make install

PCL

Download: http://www.pointclouds.org/downloads/linux.html

sudo apt-get install libusb-1.0-0-dev libusb-dev libudev-dev
sudo apt-get install mpi-default-dev openmpi-bin openmpi-common 
sudo apt-get install libboost-all-dev
sudo apt-get install libqhull* libgtest-dev
sudo apt-get install freeglut3-dev pkg-config
sudo apt-get install libxmu-dev libxi-dev 
sudo apt-get install mono-complete
sudo apt-get install openjdk-8-jdk openjdk-8-jre


git clone https://github.com/PointCloudLibrary/pcl
# https://github.com/PointCloudLibrary/pcl/archive/pcl-1.8.1.tar.gz
cd pcl && mkdir build && cd build

cmake -DBUILD_apps=ON -DBUILD_examples=ON ..
make -j8
sudo make install

Official OpenCV installation

wget -O opencv.zip https://github.com/opencv/opencv/archive/3.4.2.zip
wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/3.4.2.zip
unzip opencv.zip
unzip opencv_contrib.zip
Packages needed for OpenCV
GTK support for GUI features, Camera support (libv4l), Media Support (ffmpeg, gstreamer) etc. Additional packages for image formats mostly downloaded form the ubuntu-restricted-extra repository
sudo apt-get install libjpeg-dev libpng-dev libtiff-dev
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
sudo apt-get install libxvidcore-dev libx264-dev libvorbis-dev
sudo apt-get install libgtk-3-dev
sudo apt-get install qt5-default libgtk2.0-dev libtbb-dev
sudo apt-get install libatlas-base-dev gfortran libblas-dev liblapack-dev 
sudo apt-get install libdvd-pkg libgstreamer-plugins-base1.0-dev
sudo apt-get install libfaac-dev libmp3lame-dev libtheora-dev
sudo apt-get install libxine2-dev libv4l-dev x264 v4l-utils
sudo apt-get install libopencore-amrnb-dev libopencore-amrwb-dev

# Optional dependencies
sudo apt-get install libprotobuf-dev protobuf-compiler
sudo apt-get install libgoogle-glog-dev libgflags-dev
sudo apt-get install libgphoto2-dev libeigen3-dev libhdf5-dev doxygen
Configure OpenCV with CMake
$ cd ~/cv/opencv-3.4.2/
$ mkdir build
$ cd build
$ cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D INSTALL_C_EXAMPLES=OFF \
-D OPENCV_EXTRA_MODULES_PATH=~/cv/opencv_contrib-3.4.2/modules \
-D WITH_TBB=ON \
-D WITH_V4L=ON \
-D WITH_QT=ON \
-D WITH_OPENGL=ON \
-D WITH_VTK=ON \
-D PYTHON_EXECUTABLE=~/.virtualenvs/cv3/bin/python \
-D BUILD_EXAMPLES=ON ..
Screenshot from 2018-07-12 13-18-55

Make sure the Python 3 interpreter and other dependencies are configured correctly.

Compile, Install and Verify
(cv3) rahul@karma:~/cv/opencv-3.4.2/build$ make -j4
$ sudo make install
$ sudo sh -c 'echo "/usr/local/lib" >> /etc/ld.so.conf.d/opencv.conf'
$ sudo ldconfig
$ pkg-config --modversion opencv
Setup the cv shared libraries
(cv3) rahul@karma$ ls -l /usr/local/lib/python3.6/site-packages
total 5172
-rw-r--r-- 1 root staff 5292240 Jul 12 13:32 cv2.cpython-36m-x86_64-linux-gnu.so
# or use the find command 
$ find /usr/local/lib/ -type f -name "cv2*.so"
$ cd /usr/local/lib/python3.6/site-packages/
$ mv cv2.cpython-36m-x86_64-linux-gnu.so cv2.so
$ cd ~/.virtualenvs/cv3/lib/python3.6/site-packages/
$ ln -s /usr/local/lib/python3.6/site-packages/cv2.so cv2.so

C. Test

(cv3) rahul@karma:~$ python
Python 3.6.5 (default, Apr 1 2018, 05:46:30) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information
>>> import cv2
>>> cv2.__version__
'3.4.2'
>>> exit()
(cv3) rahul@karma:~$

Done.

Anaconda for your Image Processing, Machine Learning, Neural Networks, Computer Vision development environment using VS Code

Python is a great language and I will not go into explaining why it is so. Here is a brief setup for your development environment in case you are tinkering with computer vision problems and looking at learning neural network on your windows laptop.

Anaconda3 5.0

64 bit Download: https://www.anaconda.com/download

Install Anaconda with the default options.

  • Anaconda Navigator is a great place to look at your environment and activate them as per your need.
  • In case you want to have a Python 2x and 3x environment side by side, then you can create them in navigator. Here I have a base(root) setup with Python 3.6 and an additional Python 2.7 environment.
  • In order to use a particular environment you can click on that environment in the navigator or go to the Anaconda prompt and execute the following command
"(base)C:\Users\Karma>activate Py27"
  • To deactivate use
deactivate
  • To create a new environment use the following command:
(base)C:\Users\Karma>conda create -n Py27 python=2.7 anaconda

Anaconda-Navigator

Whenever you want to use a particular environment just go to the environments section and activate it. This will setup your python with the packages and version as configured in that environment.  In the screenshot above I have tensorflow in my base environment while its always better to have a separate environment for this.

In case you are using Cmder like me then go for this:

Considering where you have installed your Anaconda
> C:\Anaconda3\Scripts\activate.bat C:\Anaconda3
or
> C:\Users\Karma\Anaconda3\Scripts\activate.bat C:\Users\Karma\Anaconda3
> conda info --envs
> conda activate py27
> conda deactivate

Lets try to use package manager “conda” for the setup.

Run the following installation command on Anaconda Command Prompt which will open up showing prompt as (C:\Anaconda3) C:\Users\Karma>:

In order to find packages, you should look at the Anaconda repository ( https://anaconda.org/anaconda/repo )

# Adding the menpo channels and install opencv
conda install -c https://conda.binstar.org/menpo opencv
conda config --add channels menpo
conda install -c menpo opencv

# or directly use conda-forge
conda install -c conda-forge opencv

# Install packages
conda install numpy
conda install scipy
conda install matplotlib

# List packages
conda list

OpenCV

If the OpenCV installation did not go through then we can use the pre-built windows binaries maintained by,

Christoph Gohlke at https://www.lfd.uci.edu/~gohlke/pythonlibs/#opencv

Download File: You can remove these modules by using “pip uninstall <package>”

(base)λ pip install opencv_python-3.4.0-cp36-cp36m-win_amd64.whl
Processing c:\users\karma\downloads\opencv_python-3.4.0-cp36-cp36m-win_amd64.whl
Installing collected packages: opencv-python
Successfully installed opencv-python-3.4.0
(base)λ pip install opencv_python-3.4.0+contrib-cp36-cp36m-win_amd64.whl
Processing c:\users\karma\downloads\opencv_python-3.4.0+contrib-cp36-cp36m-win_amd64.whl
Installing collected packages: opencv-python
Successfully installed opencv-python-3.4.0+contrib

In my case I used SIFT and SURF implementations which were made available in the contrib packages.

Now, that we have packages set, lets test it out on the python interpreter interface,
Use the following commands on the python CLI.

import numpy as np
import cv2

TensorFlow

Instructions: https://www.tensorflow.org/install/install_windows

To install this package with conda run:
conda install -c conda-forge tensorflow

Version changes based on the repository you are trying to download from.

I typically use VS Code but if you like smooth scrolling go for Sublime.

In VS Code I use ms-python.python, tht13.python extensions to simplify my workspace.

VSCode-Python

Debugging is critical to work with any kind of code. So here is some configuration to get you started here.

  • Verify that the workspace settings.json file has the right python path
"python.pythonPath ": "C:\\Anaconda3\\python.exe"
  • Add a launch.json in your project .vscode folder with the following values
{
   "name": "Python",
   "type": "python",
   "pythonPath":"${config:python.pythonPath}", "request": "launch", "stopOnEntry": true, "console": "none", "program": "${file}", "cwd": "${workspaceFolder}", "debugOptions": [ "WaitOnAbnormalExit", "WaitOnNormalExit", "RedirectOutput" ] }
This will get you setup for debugging and here is how the debug interface would look like when you have put the breakpoints and stepped through the code.
VSCode-Python-Debug

Good Luck.

Repository Management with Nexus 3 for your Mavenized project, including release and snapshot distribution

Like the Nexus documentation says;

Stop developing in the Dark Ages, read this book, and start using a repository manager. Trust us, once you start using a Nexus Repository Manager, you’ll wonder how you ever functioned without it.

Reference:

A. Download the archive from https://www.sonatype.com/download-oss-sonatype

B. Unzip it into a folder and run it as follows

cd ~\nexus\nexus-3.3.1-01\bin
nexus.exe /run 

If the log shows the following that means the server is up  
-------------------------------------------------
 Started Sonatype Nexus OSS 3.3.1-01
-------------------------------------------------

C. Server starts by default on http://localhost:8081

username: admin  
password: admin123 
Use the above credentials to login as the default administrator

D. Add the following configuration to the ~\USER_HOME\.m2\settings.xml

Make sure you remove the code tags before using this configuration, which is used here for wordpress content formatting only.

<settings>
	  <mirrors>
		<mirror>
		  <!--This sends everything else to /public -->
		  <id>nexus</id>
		  <mirrorOf>*</mirrorOf>
		  <url>http://localhost:8081/repository/maven-public/</url>
		</mirror>
	  </mirrors>
	  <profiles>
		<profile>
		  <id>nexus</id>
		  <!--Enable snapshots for the built in central repo to direct -->
		  <!--all requests to nexus via the mirror -->
		  <repositories>
			<repository>
			  <id>central</id>
			  <url>http://central</url>
			  <releases><enabled>true</enabled></releases>
			  <snapshots><enabled>true</enabled></snapshots>
			</repository>
		  </repositories>
		 <pluginRepositories>
			<pluginRepository>
			  <id>central</id>
			  <url>http://central</url>
			  <releases><enabled>true</enabled></releases>
			  <snapshots><enabled>true</enabled></snapshots>
			</pluginRepository>
		  </pluginRepositories>
		</profile>
	  </profiles>
	  <activeProfiles>
		<!--make the profile active all the time -->
		<activeProfile>nexus</activeProfile>
	  </activeProfiles>
	   <servers>
		<server>
		  <id>nexus</id>
		  <username>admin</username>
		  <password>admin123</password>
		</server>
	  </servers>
	</settings>

E. Release and snapshot artifacts should be configured in the projects pom as distributionManagement

  <distributionManagement>
    <repository>
      <id>nexus</id>
      <name>Releases</name>
      <url>http://localhost:8081/repository/maven-releases</url>
    </repository>
    <snapshotRepository>
      <id>nexus</id>
      <name>Snapshot</name>
      <url>http://localhost:8081/repository/maven-snapshots</url>
    </snapshotRepository>
  </distributionManagement>

F. The clean and deploy goal in your Java project will build and upload the artifacts to the repository using the server credentials tag from settings.xml

mvn clean deploy -DskipTests

NexusRepository

G. Add a proxy repository

You can add a new proxy repository to your Nexus instance using the following steps

  1. Create a repository from the repositories admin page
  2. Select the maven2 recipe since JBOSS is a maven like repository
  3. Provide a name like “jboss-nexus-repository”
  4. Add this repository to the group you have defaulted your maven to, so that maven can use this as a part of the group it is defaulted to.

NexusProxyRepository

H. Adding your custom jars into the repository

  1. Create a repository with maven2 hosted recipe
  2. Obtain the created repository URL and run the following maven deploy command on your jar file
 mvn deploy:deploy-file 
-Durl=http://localhost:8081/repository/project-customs/ 
-DrepositoryId=nexus -DgroupId=com.oracle 
-DartifactId=ojdbc6 -Dversion=11.2.0.4 
-Dpackaging=jar -Dfile=C:/Users/vishwaka/.m2/ojdbc6.jar 
-DgeneratePom=true
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-deploy-plugin:2.7:deploy-file (default-cli) @ standalone-pom ---
Uploading: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.jar
Uploaded: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.jar (1942 KB at 583.2 KB/sec)
Uploading: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.pom
Uploaded: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.pom (392 B at 0.1 KB/sec)
Downloading: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/maven-metadata.xml
Downloaded: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/maven-metadata.xml (302 B at 0.2 KB/sec)
Uploading: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/maven-metadata.xml
Uploaded: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/maven-metadata.xml (302 B at 0.1 KB/sec)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.768 s
[INFO] Finished at: 2017-06-15T11:29:59-07:00
[INFO] Final Memory: 11M/245M
[INFO] ------------------------------------------------------------------------

NexusHostedRepository

You should be able to see this in your repositories assets once the upload is successful. The upload deploy uses credentials from your server.xml configuration so make sure that is available.

3. Upon doing this we need to add the project-custom repository as a member to the maven-public group of repositories

NexusMemberRepository

I. Test by running a clean build of your maven project

  • Delete the folder containing the jar files in the path \.m2\repository\com\oracle\ojdbc6\11.2.0.4
  • Rerun the maven build using mvn clean compile
  • Verify the following logs in the build
Downloading: http://localhost:8081/repository/maven-public/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.pom
Downloaded: http://localhost:8081/repository/maven-public/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.pom (392 B at 2.8 KB/sec)
Downloading: http://localhost:8081/repository/maven-public/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.jar
Downloaded: http://localhost:8081/repository/maven-public/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.jar (1942 KB at 10846.1 KB/sec)

Continuous Integration in Pipeline as Code Environment with Jenkins, JaCoCo, Nexus and SonarQube

Github Link for the source code: https://github.com/vishwakarmarhl/jenkinstest

Here we discuss the setup for a Continuous integration pipeline. This is for mavenized Spring boot build with JaCoCo coverage reports and Sonar metrics. I used a windows machine with Tomcat 8 for hosting jenkins, but similar setup can be done on any OS where Sonar server can run on the same system.

A. Get the following artifacts on the system

  1. Tomcat server with Java JDK – Configure the server.xml to run on port 8099
  2. Setup Maven & other build utilities on your machine
  3. Access to Github source code
  4. Source code should have the Jenkinsfile in project root to be used by the pipeline
  5. Source should have the sonar-project.properties in project root for the SonarQube project linkage & source paths

JenkinsFile

Jenkinsfile and sonar-project.properties snapshot

B. Setup & Startup SonarQube

  1. Download the SonarQube package from https://www.sonarqube.org/#downloads
  2. Start sonar server: SONAR_HOME\bin\windows-x86-32\StartSonar.bat (for 32 bit Windows)
  3. Open Sonar admin page “http://localhost:9000“. Default credentials – admin/admin
  4. Create user in security tab and generate an access token, 50997f4a8c26d5698cccee30cf398c0ed9b98de0
  5. Create a project SPRINGBOOT with a key
  6. Download SonarQube scanner from https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner
  7. Additional configuration from https://docs.sonarqube.org/display/SCAN/Advanced+SonarQube+Scanner+Usages

C. Setup & Startup Tomcat

  1. Download jenkins.war from https://jenkins.io/download
  2. Put the jenkins.war file in webapps folder of Tomcat home
  3. Set Environment Variables as follows,
  4. SET JENKINS_HOME=”C:/Users/vishwaka/Documents/Workspace/git/jenkinstest/cisetup/jenkins_home”
  5. SET CATALINA_OPTS=”-DJENKINS_HOME=C:/Users/vishwaka/Documents/Workspace/git/jenkinstest/cisetup/jenkins_home”
  6. Start the server using startup.bat

JenkinsHome

Initial launch of Jenkins

D. Initialize Jenkins

  1. Access Jenkins at http://localhost:8099/jenkins
  2. Provide the initial credentials from jenkins_home/secrets/initialPassword*
  3. Install the default set of plugins and proceed
  4. Create a user for this installation
  5. Use “New Item” for creating a pipeline and provide the Jenkinsfile pipeline script from Git SCM for this

JenkinsCreatePipeline

Create pipeline project

E. Plugin & Configuration to Jenkins

  1. Add the “JaCoCo plugin” through the Manage Jenkins > Manage Plugins and install without restart
  2. Add “SonarQube Scanner for Jenkins” through the same Plugin Manager as above
  3. Go to the Manage Jenkins > Configure system and provide the credentials for Sonar Server
  4. Add the “SonarQube Server” name running on URL http://localhost:9000 alongwith user authentication key generated in SonarQube Server user administration page
  5. Remove the auto install option and add the “Sonar Scanner” env variable SONAR_RUNNER_HOME installation path as $JENKINS_HOME/sonar-scanner-3.0.3.778-windows through “Global Tool Configuration”
  6. Make sure the Sonar scanner path is configured properly as its path is hard coded in Jenkinsfile.

JenkinsGlobalProperties

Global Tool Configuration

F. Run the Build now for this pipeline

  1. The pipeline is at http://localhost:8099/jenkins/job/JENKINS-BOOT/JenkinsStatusPipeline
  2. Checkout the coverage report within the pipeline reports JenkinsJacoco
  3. You can also look at the Sonar reports at http://localhost:9000/dashboard?id=JENKINSBOOT JenkinsToSonar
  4. If you have many such projects then its better to execute all your Job Pipelines from a parent Job Pipeline. You can create one and call it “BUILD-ALL-JOBS”. It can be configured using the below pipeline script to run your JENKINS-BOOT job described in the example above as well as any other fictitious job call JENKINS-BOOT-XXX.
node {
    stage('JENKINS-BOOT-STAGE-A') {
        build job: 'JENKINS-BOOT'
    }
    stage('JENKINS-BOOT-STAGE-B') {
        build job: 'JENKINS-BOOT-XXX'
    }
}

There are plugins to build jobs in parallel as well but that depends on what workflow you want to build in your system.

G. Adding Nexus repository management capability to your CI environment from my blog

Click on the text link below:

Repository Management with Nexus 3 for your Mavenized project, including release and snapshot distribution

H. Finally put everything into a script that can run it all

Pardon my naive & careless script, considering my setup is on a local windows development workstation.

@echo off
echo "--------------------------------------------------------------------------"
echo "------------------------- CI STARTUP SCRIPT ------------------------------"
echo "--------------------------------------------------------------------------"

echo "Startup SonarQube Server"
echo "------------------------"
START CMD /C "cd c:\Dock\ci\sonar\sonarqube-6.4\bin\windows-x86-64 & CALL StartSonar.bat"
echo "Sonar may be up on http://localhost:9000/"

echo "Startup Nexus Repository Manager"
echo "--------------------------------"
START CMD /C "cd c:\Dock\ci\nexus\nexus-3.3.1-01\bin & nexus.exe /run"
echo "Nexus may be up on http://localhost:8081/"

echo "Startup Jenkins on Tomcat"
echo "-------------------------"
START CMD /C "cd c:\Dock\ci\jenkins\apache-tomcat-8.5.15\bin & startup.bat"
echo "Jenkins may be up on http://localhost:8099/jenkins"

echo "-------------------------------- END -------------------------------------"

 

Thanks.

 

Dockerification of your local virtual instance with SSH, XFCE & VNC

1. Docker: the client-server application made up of the Docker daemon, a REST API that specifies interfaces for interacting with the daemon, and a command line interface (CLI) client that talks to the daemon (through the REST API wrapper). Docker Engine accepts docker commands from the CLI, such as docker run , docker ps to list running containers, docker images to list images, and so on.

2. Docker Machine: a tool for provisioning and managing your Dockerized hosts (hosts with Docker Engine on them). Typically, you install Docker Machine on your local system. Docker Machine has its own command line client docker-machine and the Docker Engine client, docker. You can use Machine to install Docker Engine on one or more virtual systems.

engine-components-flow

We will be using Virtualbox based virtualization in docker which is supported on windows and mac.

A setup for linux instance is also available

I will be using docker toolbox as for my docker installation on windows.

 

1. Install the docker-toolbox using the default options and verify the versions from the Docker Quickstart Terminal.

$ docker-machine --version
  docker-machine.exe version 0.8.2, build e18a919

$ docker --version
  Docker version 1.12.2, build bb80604

 

2. Perfect, now we move to the docker machine toolbox which hosts the docker engine and give it a kickoff.

$ docker-machine rm default
$ docker-machine create --driver virtualbox --virtualbox-disk-size "500100" default
$ docker-machine start default
$ docker-machine env default
$ eval $("C:\Program Files\Docker Toolbox\docker-machine.exe" env default)

 

3. Lets work with docker engine CLI for management of the docker image

$ docker ps
$ docker info
$ docker --help
$ docker-machine ls
  NAME    ACTIVE DRIVER     STATE   URL                       SWARM DOCKER ERRORS
  default *      virtualbox Running tcp://192.168.99.100:2376       v1.12.3

 

4. Pull an image from docker hub and set it up locally

-- Verify the images that are downloaded, which  will be empty initially
$ docker images
-- Search for an image in docker hub and pull it
$ docker search ubuntu
$ docker pull ubuntu 
$ docker run ubuntu

 

5. Initialize and connect to bash for this image

-- Run a container and connect to its term
-- Also expose the ports and maps it to relevant exposed port in the image
$ docker run -it -p 52022:22 -p 52023:5900 -p 52021:80 ubuntu /bin/bash
-- Verify the version 
root@c:/# cat /etc/lsb-release
          apt-get update
          apt-get install -y build-essential openssh-server
          ip addr show
          service ssh restart

-- Exit from virtual host. This will also drop the changes if its uncommitted.
-- Keep this instance alive and go through the step 6 for persisting the changes.
root@c:/# exit

-- Get the IP from another console while keeping the image running 
$ docker ps
  CONTAINER ID IMAGE  COMMAND     CREATED       STATUS       PORTS NAMES
  dee57b8bba0e ubuntu "/bin/bash" 5 minutes ago Up 5 minutes 0.0.0.0:52022->22/tcp, 0.0.0.0:52021->80/tcp, 0.0.0.0:52023->5900/tcp sick_darwin 

$ docker inspect 
-- Look for the network configuration in the result
Obtained "IPAddress": "172.17.0.2"

 

6. Setup your new Ubuntu for vnc based desktop access

# apt-get update
-- Install the xfce desktop environment
# apt-get install -y build-essential xfonts-base xfce4 xfce4-goodies xubuntu-desktop
-- Install the vncserver
# apt-get install -y build-essential tightvncserver sudo vim openssh-server
# service ssh restart
-- Add a user for access to the instance
# adduser crusader
# usermod -aG sudo crusader
# su crusader
-- Initialize the VNC server for access
# export USER=crusader
# vncserver -geometry 1440x900 -rfbport 5900
# ps -eaf | grep vnc

 

7. Commit, persist and manage the image changes

-- Makes sure you commit the changes to docker and add a tag to it.
$ docker commit <container_id> vishwakarmarhl/ubshinydev:v01

-- Run a container and connect to its term
$ docker run -it -p 52022:22 -p 52023:5900 -p 52021:80 vishwakarmarhl/ubshinydev:v01 /bin/bash

-- Run the committed container image as a daemon, restart sshd and open bash
$ docker run -d -p 52022:22 -p 52023:5900 -p 52021:80 vishwakarmarhl/ubshinydev:v01 /bin/sh -c "while true; do echo hello world; sleep 10; done"
$ docker exec -it <container_id> /bin/bash
  # service ssh restart
 
-- Now you shoule be able to connect to the instance from the host
$ ssh -p 52022 crusader@192.168.99.100

-- Stop a container
$ docker stop

-- Remove Image by name. This will permanently delete your image
$ docker rmi -f ubuntu

 

8. Push Image to docker hub

Docker Hub Link : https://hub.docker.com/u/vishwakarmarhl

$ docker images
$ docker commit 162f8f8c5f19  vishwakarmarhl/ubunitydesk:v01
$ docker login
$ docker push vishwakarmarhl/ubunitydesk

-- Pull the images from docker hub
$ docker pull vishwakarmarhl/ubunitydesk

You should make sure you commit the changes done on this instance to the docker repository for persistence. This will be used to share at the docker hub repository.

 

9. Dockerfile for similar setup on Github

Download the docker file from the provided github link

Github Link: https://github.com/vishwakarmarhl/dockers/blob/master/Xubuntu-16-Desk-DockerFile

-- Build and run the docker container
$ docker build -t vishwakarmarhl/ubunitydesk:v01 . -f DockerFile 
$ docker run -it -p 52022:22 -p 52023:5900 -p 52021:80 vishwakarmarhl/ubunitydesk:v01 /bin/bash

Here upon you can configure your machine with any package. Will continue to describe how to use this environment for development purposes. This may as well be my notes but helps all the time for a quick reference.