GrayLog Demo Overview

gl-logo_gray-whiteOne of my favorite new tools in the IT Operations Analytics space is GrayLog.  In the process of gearing up to perform my initial full blown product review I ran across a well done initial overview by Jonah Kowall, Research VP for Gartner.   The overview, published on January 27, 2015 and titled, “Cool Vendor Pick: Graylog,” provides a brief overview and several screen shots of GrayLog.

Following is the full text of Mr. Kowall’s article:

There has been a lot of interest over the last 12 months in products based on open source for monitoring and management. In the area of log analysis, Elasticsearch has been a player which has strengthened with the growing investments in the space. The awareness has been greatly increased in the past year. While the popular Kibana frontend to Elasticsearch has been the main GUI. These two projects are paired with Logstash for ingest, combined these make up the ELK stack. There is another great open source project to take a look at. The focus of this weeks write-up is on this alternative to ELK.

The company behind Graylog is Torch out of Hamburg Germany (https://www.torch.sh/) they do consulting around the product. The open source site is https://www.graylog2.org/ the project is an ElasticSearch based product, but unlike Kibana it also has additional features:

  • Take inputs directly into the Graylog server processes
  • Output from the server to multiple backends based on output plugins, right now the main one is for ElasticSearch
  • Alerting based on matching or other criteria are integrated into the Graylog project along with a stream processing capability

The supported data comes in the form of plugins which include syslog or GELF (Graylog Extended Log Format) or other plugins. GELF allows for several enhancement from typical syslog.

  • No length limitations for messages (syslog is 1024 bytes)
  • Data types (string, number)
  • Variation in syslog implementation
  • Compression via gzip or zlib

The nice thing is that you don’t need to do any extractions once the messages have been added via GELF. They have 72 such plugins including many GELF libraries (See: https://www.graylog2.org/supported-sources?perPage=100)

On the site you can sign up for a self-service trial of the software, I did this in early November, there has been another release since then. These screenshots may be a little out of date:

Image1

 

Image2

There can be multiple backend nodes connected to the frontend. There is some good management within the GUI of the connections. The main dashboard when you login shows you information about the cluster, components, and the status. There is a query box.

Image4

 

Some other administrative views. Many of the log management tools, especially in open source neglect the day to day maintenance and administration. Being a systems and operations person myself I always dig into the internals needed for day to day administration. Graylog has a lot of what’s been missing across open source ElasticSearch management tools. Some additional views:

 

Image19

 

Image20

They have a data generator in the demo so you’ll see there are plenty of events in the data store.

Image14

Image4

Here is a query for smtp in the last 30 minutes.

Image5

 

You can also see inside the queries being sent to ElasticSearch, here are the JSON objects being passed to the engine:

Image6

Value breakdowns of the results quickly

Image7

 

Graylog has the notion of stream as illustrated below

Image8

 

What these are is a way to pass realtime rules against the data coming into the Graylog server before they are committed to elasticsearch, this real time processing provides a differentiator to Kibana based systems

Image9

Image10

 

Image17

 

Some sample sinks of what you can do with a proper eventing system, such as alerting:

Image11

The requisite dashboarding for any monitoring tool. Everyone loves dashboards, users are always asking for more dashboards, and they clearly do sell monitoring products. The value they provide are typically pretty limited. If the actual analytics in our software were better the computer would be doing the analysis versus a user looking at graphical displays of data. I digress…

Image12

You cannot share the same backend between Kibana/Logstash and Graylog since they use a different schema for the log data in ElasticSearch. Hence you’ll have to make a decision which tool you want to use when setting up ElasicSearch.

 

About Charles Skamser
Charles Skamser is an internationally recognized technology sales, marketing and product management leader with over 25 years of experience in Information Governance, eDiscovery, Machine Learning, Computer Assisted Analytics, Cloud Computing, Big Data Analytics, IT Automation and ITOA. Charles is the founder and Senior Analyst for eDiscovery Solutions Group, a global provider of information management consulting, market intelligence and advisory services specializing in information governance, eDiscovery, Big Data analytics and cloud computing solutions. Previously, Charles served in various executive roles with disruptive technology start ups and well known industry technology providers. Charles is a prolific author and a regular speaker on the technology that the Global 2000 require to manage the accelerating increase in Electronically Stored Information (ESI). Charles holds a BA in Political Science and Economics from Macalester College.