Repository Management with Nexus 3 for your Mavenized project, including release and snapshot distribution

Like the Nexus documentation says;

Stop developing in the Dark Ages, read this book, and start using a repository manager. Trust us, once you start using a Nexus Repository Manager, you’ll wonder how you ever functioned without it.

Reference:

A. Download the archive from https://www.sonatype.com/download-oss-sonatype

B. Unzip it into a folder and run it as follows

cd ~\nexus\nexus-3.3.1-01\bin
nexus.exe /run 

If the log shows the following that means the server is up  
-------------------------------------------------
 Started Sonatype Nexus OSS 3.3.1-01
-------------------------------------------------

C. Server starts by default on http://localhost:8081

username: admin  
password: admin123 
Use the above credentials to login as the default administrator

D. Add the following configuration to the ~\USER_HOME\.m2\settings.xml

Make sure you remove the code tags before using this configuration, which is used here for wordpress content formatting only.

<settings>
	  <mirrors>
		<mirror>
		  <!--This sends everything else to /public -->
		  <id>nexus</id>
		  <mirrorOf>*</mirrorOf>
		  <url>http://localhost:8081/repository/maven-public/</url>
		</mirror>
	  </mirrors>
	  <profiles>
		<profile>
		  <id>nexus</id>
		  <!--Enable snapshots for the built in central repo to direct -->
		  <!--all requests to nexus via the mirror -->
		  <repositories>
			<repository>
			  <id>central</id>
			  <url>http://central</url>
			  <releases><enabled>true</enabled></releases>
			  <snapshots><enabled>true</enabled></snapshots>
			</repository>
		  </repositories>
		 <pluginRepositories>
			<pluginRepository>
			  <id>central</id>
			  <url>http://central</url>
			  <releases><enabled>true</enabled></releases>
			  <snapshots><enabled>true</enabled></snapshots>
			</pluginRepository>
		  </pluginRepositories>
		</profile>
	  </profiles>
	  <activeProfiles>
		<!--make the profile active all the time -->
		<activeProfile>nexus</activeProfile>
	  </activeProfiles>
	   <servers>
		<server>
		  <id>nexus</id>
		  <username>admin</username>
		  <password>admin123</password>
		</server>
	  </servers>
	</settings>

E. Release and snapshot artifacts should be configured in the projects pom as distributionManagement

  <distributionManagement>
    <repository>
      <id>nexus</id>
      <name>Releases</name>
      <url>http://localhost:8081/repository/maven-releases</url>
    </repository>
    <snapshotRepository>
      <id>nexus</id>
      <name>Snapshot</name>
      <url>http://localhost:8081/repository/maven-snapshots</url>
    </snapshotRepository>
  </distributionManagement>

F. The clean and deploy goal in your Java project will build and upload the artifacts to the repository using the server credentials tag from settings.xml

mvn clean deploy -DskipTests

NexusRepository

G. Add a proxy repository

You can add a new proxy repository to your Nexus instance using the following steps

  1. Create a repository from the repositories admin page
  2. Select the maven2 recipe since JBOSS is a maven like repository
  3. Provide a name like “jboss-nexus-repository”
  4. Add this repository to the group you have defaulted your maven to, so that maven can use this as a part of the group it is defaulted to.

NexusProxyRepository

H. Adding your custom jars into the repository

  1. Create a repository with maven2 hosted recipe
  2. Obtain the created repository URL and run the following maven deploy command on your jar file
 mvn deploy:deploy-file 
-Durl=http://localhost:8081/repository/project-customs/ 
-DrepositoryId=nexus -DgroupId=com.oracle 
-DartifactId=ojdbc6 -Dversion=11.2.0.4 
-Dpackaging=jar -Dfile=C:/Users/vishwaka/.m2/ojdbc6.jar 
-DgeneratePom=true
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-deploy-plugin:2.7:deploy-file (default-cli) @ standalone-pom ---
Uploading: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.jar
Uploaded: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.jar (1942 KB at 583.2 KB/sec)
Uploading: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.pom
Uploaded: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.pom (392 B at 0.1 KB/sec)
Downloading: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/maven-metadata.xml
Downloaded: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/maven-metadata.xml (302 B at 0.2 KB/sec)
Uploading: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/maven-metadata.xml
Uploaded: http://localhost:8081/repository/project-customs/com/oracle/ojdbc6/maven-metadata.xml (302 B at 0.1 KB/sec)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.768 s
[INFO] Finished at: 2017-06-15T11:29:59-07:00
[INFO] Final Memory: 11M/245M
[INFO] ------------------------------------------------------------------------

NexusHostedRepository

You should be able to see this in your repositories assets once the upload is successful. The upload deploy uses credentials from your server.xml configuration so make sure that is available.

3. Upon doing this we need to add the project-custom repository as a member to the maven-public group of repositories

NexusMemberRepository

I. Test by running a clean build of your maven project

  • Delete the folder containing the jar files in the path \.m2\repository\com\oracle\ojdbc6\11.2.0.4
  • Rerun the maven build using mvn clean compile
  • Verify the following logs in the build
Downloading: http://localhost:8081/repository/maven-public/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.pom
Downloaded: http://localhost:8081/repository/maven-public/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.pom (392 B at 2.8 KB/sec)
Downloading: http://localhost:8081/repository/maven-public/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.jar
Downloaded: http://localhost:8081/repository/maven-public/com/oracle/ojdbc6/11.2.0.4/ojdbc6-11.2.0.4.jar (1942 KB at 10846.1 KB/sec)

Java Integration with Kafka distributed message broker

Kafka is a major distributed, partitioned, replicated, commit log service used as a message  broker in the current tech industry open-sourced by Linked-In. It functions as a central messaging bus for various domains and frameworks specializing in big-data systems.

It works as a broker between the producers and consumers with a strong order guarantee than any traditional messaging system. It provides a single consumer abstraction model that generalizes the queuing and publish-subscribe model as a Consumer-Group.

Reference: http://kafka.apache.org/documentation.html#introduction

Github: https://github.com/vishwakarmarhl/kafka-sample

The API is provided by Kafka-clients version 0.9.0.0 

Here I am going through a sample that is capable of publishing a message over the kafka topic. There is a listener constantly waiting with a consumer group that works as a thread pool subscribed to the topic.

A. Setting up kafka is as simple as extracting the archive and starting the server.

Reference: http://kafka.apache.org/090/documentation.html

Download: https://www.apache.org/dyn/closer.cgi?path=/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz

Say you have a virtual machine as per my setup and the host IP as 192.168.56.101

  1. Download and extract the above kafka-2.11-0.9 build in /opt folder, and set the KAFKA_HOME variable
  2. Configure $KAFKA_HOME/config/server.properties
    advertised.host.name=192.168.56.101

kafka broker internals

If you are into spring and java then this is a great introduction resource from SpringDeveloper. However in this tutorial I am covering a very basic implementation.

 

B. Walk-through of the Producer and Listener

producer_consumer

The code has two main classes responsible for the send (KafkaSpeaker.java) and listen (KafkaListener.java) functionality. KafkaMain is responsible for launching the KafkaListener in a pre-configured consumer threadpool.

KafkaProducer.java

The producer is configured with the broker url and client identifier.

//The kafka producer is configured with the topic and broker address for sending the message as follows: 

 Properties props = new Properties();
 props.put("bootstrap.servers", prop.getProperty("producer.url"));
 props.put("client.id", prop.getProperty("producer.client.id"));
 props.put("acks", "all");
 props.put("retries", 0);
 props.put("batch.size", 16384);
 props.put("linger.ms", 1);
 props.put("buffer.memory", 33554432);
 props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
 props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
 this.producer = new KafkaProducer<>(props);

The producer can send the message on a topic using the send api

producer.send(new ProducerRecord<String, String>(topic,
  String.format("{\"type\":\"sample\",\"t\":%.3f,\"k\":%d}",
                System.nanoTime()*1e-9,i)
  ));

 

KafkaListener.java

The listener has the concept of consumer group that can listen on each partition and transact on the messages queued for consumption and further processing.

consumer-groups

The consumer group is created using the threadpool executors as follows,

final ExecutorService executor = Executors.newFixedThreadPool(numConsumers);
final List<KafkaListener> consumers = new ArrayList<>();
for (int i = 0; i < numConsumers; i++) {
 KafkaListener consumer = new KafkaListener(i, prop, topics);
 consumers.add(consumer);
 executor.submit(consumer);
}

Configure the listener and create a consumer

props.put("bootstrap.servers", prop.getProperty("consumer.url"));
props.put("group.id", prop.getProperty("consumer.group.id")); 
props.put("client.id", prop.getProperty("consumer.client.id"));
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "2000");
props.put("session.timeout.ms", "30000");
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());
this.consumer = new KafkaConsumer<>(props);
this.timeOut = Integer.parseInt( prop.getProperty("consumer.poll.timeout"));

Subscribe and listen infinitely on the topic until the shutdown hook is executed for this thread pool.

consumer.subscribe(topics);
while (true) {
 ConsumerRecords<String, String> records = consumer.poll(this.timeOut);
 for (ConsumerRecord<String, String> record : records) {
    log.info(record.topic()+" - "+this.id + ", Data: " + record.value());
 }
}

 

The code available in github is self-sufficient and simple enough and was very simple to create.

Thanks to the great article that set me of on this trial project.