Monitoring memory usage for Bigdata processes using attach API and notifying users

Recently I have been asked by customers for a way to monitor heap memory growth and alert if it exceeds a certain thresholds. Ambari provides robust alerting and monitoring capabilities but there is limited support in monitoring heap sizes for services like HiveServer2. Although there are several monitoring and notification applications available in the market, I have made an attempt to highlight how to build our own monitoring and notification application with unlimited customization.

Technologies used

1. Java attach API

2. Apache Commons Email

3. Maven

4. OpenJDK 8

The project is uploaded to my git repo and can be accessed using the following link https://github.com/bharathdcs/attach_api_notification

Java Attach API’s allows user program to attach to running JVM’s. The JVM could be hosting NodeManagers , HiveServer2 or any other process. One of the advantages of attach API is you can retrieve the JMX url from JVM and fetch various statistics, one of them is memory consumption as shown in following code snippet

VirtualMachine virtualMachine = VirtualMachine.attach(virtualMachineDescriptor);
String propertyValue = agentProperties.getProperty("com.sun.management.jmxremote.localConnectorAddress");
c.getMBeanServerConnection().getAttribute(new ObjectName("java.lang:type=Memory"), "HeapMemoryUsage");

Using the memory object (bean) returned we can retrieve stats like used, allocated and max memory. More details about MemoryMXBean can be found in the java documentation

Properties File for Parameters

Users specify the memory usage thresholds for alerts and smtp server details to send the notifications using the property file.

Following is a snippet from a sample properties file

procId=3124
smtp_host=smtp.gmail.com
smtp_port=465
smpt_user=test@gmail.com
stmp_password=testing123
threshold=10
email=test@gmail.com

The procId is the process Id of JVM to be attached and monitored**.

provide your SMTP server details using smtp_* variables.

Threshold specifies the threshold for triggering alert when memory usage crosses (threshold/100)*totalHeap.

Steps to run the Appliation

The application uses maven build framework.

1. Compile using “mvn clean compile assembly:single”

2. Run the application using following syntax

java -cp ${JAVA_HOME}/lib/tools.jar:heap-0.0.1-SNAPSHOT-jar-with-dependencies.jar monitor.heap.App sample.properties

The application monitors memory usage continuously and when it exceeds a certain threshold , following email is sent to your configured Inbox.

Possible Future Enhancements

1. Integrating with mobile notification APIs

2. Publish the memory units as JSON enabling java script widgets to chart them.

** – Users can only attach to JVMs owned by them.For example, Hive user can only attach to hive JVM’s .

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s