Jay Taylor's notes

back to listing index

Garage made, self hosted NewRelic alternative with Ruby, Sinatra, Grafana and InfluxDB

[web search]
Original source (mic-kul.com)
Tags: ruby diy influxdb grafana new-relic sinatra mic-kul.com
Clipped on: 2015-12-11

Garage made, self hosted NewRelic alternative with Ruby, Sinatra, Grafana and InfluxDB

24 October 2015 on ruby, newrelic

So you realised that for your hobby app free tier of NewRelic is not enough but still you would like to have prolonged history?
You don't want to use anything else. Maybe you have Premium version for your production app but need some monitoring for staging environment? If so continue further. It's going to be a strictly technical post with a tutorial on setting up metrics collector.


  • Ubuntu 14.04 running on VPS with 2 vCores and 2GB RAM
  • Sinatra - Ruby micro framework
  • InfluxDB - time series database
  • Grafana - graph and dashboard builder for data visualisation

Install Sinatra

Having Ruby installed, installing Sinatra is as simple as:

gem install sinatra  

but it might complicate things later so I created directory 'collector' and placed new file Gemfile with following content:

source 'https://rubygems.org'

gem 'sinatra'  
gem 'sinatra-contrib'

gem 'influxdb' # InfluxDB Ruby Client  

and run bundle install in that directory.

Install InfluxDB

To install my database I've followed official guide:

# for 64-bit systems
wget https://s3.amazonaws.com/influxdb/influxdb_0.6.5_amd64.deb  
sudo dpkg -i influxdb_latest_amd64.deb

# for 32-bit systems
wget https://s3.amazonaws.com/influxdb/influxdb_0.6.5_i386.deb  
sudo dpkg -i influxdb_latest_i386.deb  

UPDATE: Paul Dix, CEO of InfluxDB, mentioned me on Twitter (yay!) to point, that they don't use latest anymore and instead nightly or one of releases should be used:

#64-bit system install instructions
wget https://s3.amazonaws.com/influxdb/influxdb_0.9.4.2_amd64.deb  
sudo dpkg -i influxdb_0.9.4.2_amd64.deb  

Install Grafana

Same goes to Grafana installation:
Add following line to /etc/apt/sources.list:

deb https://packagecloud.io/grafana/stable/debian/ wheezy main  

and install package:

apt-get update  
apt-get install grafana  

and start the service:

sudo service grafana-server start  

Create collector

In collector directory create file named... collector.rb (Gist):

require 'sinatra'  
require 'sinatra/namespace'  
require 'sinatra/json'

require 'base64'  
require 'json'  
require 'zlib'  
require 'stringio'

require 'pry'

require 'influxdb'

use Rack::Deflater

namespace '/agent_listener/:api_version' do |api_version|  
  namespace '/:license_key' do |license_key|
    get_redirect_host = { return_value: 'localhost' }
    get '/get_redirect_host' do
      json get_redirect_host

    post '/get_redirect_host' do
      json get_redirect_host

    connect =
      { return_value: { browser_key: 'xx', application_id: 1, js_agent_loader: '' } }
    post '/connect' do
      json connect

    metric_data = {}
    post '/metric_data' do
      body = request.body.read
      body = Zlib::Inflate.inflate(body) if request.env["HTTP_CONTENT_ENCODING"] == "deflate"
      metrics = JSON.parse body
      data = []
      metrics[3].each do |meta, values|
        data = {
          tags: { metric_name: meta['name'] },
          values: {
            cnt: values[0],
            val: values[1], own: values[2], min: values[3], max: values[4], sqr: values[5]
        single_to_influx('metric_data', data)
      json metric_data

    analytic_event_data = {}
    post '/analytic_event_data' do
      p request.env
      body = request.body.read
      body = Zlib::Inflate.inflate(body) if request.env["HTTP_CONTENT_ENCODING"] == "deflate"
      json_analytics = JSON.parse body
      data = []
      json_analytics[1].each do |meta, _wtf|
        data << {
          series: 'analytics_data',
          tags: { metric_name: meta['name'], mtype: meta['type'] },
          values: { mduration: meta['duration'] },
      json analytic_event_data

    error_data = { return_value: 'ok' }
    post '/error_data' do
      # p request.body.read
      raw_body = request.body.read
      raw_body = Zlib::Inflate.inflate(raw_body) if request.env["HTTP_CONTENT_ENCODING"] == "deflate"
      json_error = JSON.parse raw_body
      data = []

      json_error[1].each do |err|
        data = {
          tags: { error_method: err[1], request_uri: err[4]['request_uri'] },
          values: { message: err[2] },
          # timestamp: err[0],
        single_to_influx('errors_data', data)
      json error_data

    get_agent_commands = { return_value: [] }
    post '/get_agent_commands' do
      json get_agent_commands

def unblob(blob)  
  return unless blob

def inflate(string)


def to_influx(data)  
  rescue StandardError => e
    p "----------"
    p data
    p "----------"
    raise e

def single_to_influx(name, data)  
    influxdb.write_point(name, data)
  rescue StandardError => e
    p "----------"
    p data
    p "----------"
    raise e

def influxdb  
  @influxdb ||= InfluxDB::Client.new 'collector', host: 'localhost'

As you can see this file parses newrelic/rpm gem messages and events, and saves them to InfluxDB.

Run collector

To run the collector use following command:
bundle exec ruby collector.rb

Your collector is now listening on default Sinatra port (4567).

Change your Rails app configuration

In your existing Rails application find rpm configuration file newrelic.yml and add following values:

  monitor_mode: true
  host: localhost
  port: 4567
  api_host: localhost
  api_port: 4567

Connect Grafana to InfluxDB

Navigate to your Grafana instance (http://localhost:3000 by default) and login with default admininstrator account (login: admin@localhost pw: admin).

Add Dashboard

From sidebar select 'Data Sources' and click 'Add new'.
Give it some name, from Type dropdown select 'InfluxDB' and set URL of your Datasource to http://localhost:8086. Your database name is 'collector' and default login credentials are again admin with password admin.

Hit Save and from top menu select '+ New' to create new dashboard.

Add Graph to Dashboard

To add panel to Dashboard, locate tiny green rectangle and click it - it will open the Row menu where you can add Graph Panel.
Image (Asset 1/10) alt=

Click on graph and select 'Edit' from menu that will appear.
On Metrics tab set 'Source' to created data source and write query as:

SELECT max(mduration) FROM "analytics_data" WHERE "mtype" = 'Transaction' AND $timeFilter GROUP BY time($interval), "metric_name"  

Set 'Group by time interval' to > 60s.

Play with other tabs or fill Axis and Grids with following values:
Image (Asset 2/10) alt= and Display Styles like this:
Image (Asset 3/10) alt=

Remember to click 'Save dashboard' icon on top menu.

Enjoy analysing your metrics

If everything is configured properly after a while your dashboard should be filled with colorful metrics:
Image (Asset 4/10) alt=

More metrics

There are more metrics collected presented, so you can add more panels and graphs to present CPU/memory usage, Ruby Garbage Collector metrics and Transactions to Errors statistics.
These examples are created using the Query Builder:

  • CPU/Memory Image (Asset 5/10) alt= Image (Asset 6/10) alt=
  • Garbage Collector Image (Asset 7/10) alt= Image (Asset 8/10) alt=
  • Transactions count Image (Asset 9/10) alt= Image (Asset 10/10) alt=

Known issues

  • timestamp is saved as metrics receive time, not the real timestamp coming from metrics payload
  • InfluxDB database should be configured with some data retention options (now data is stored forever)

Ideas for further development

  • create a proxy server for rpm gem out of this tiny app that will allow to simultaneously store metrics in InfluxDB and push them to NewRelic.

Please note that I'm not connected with NewRelic in any manner and this code is more like proof-of-concept and is not intended for usage on production environment.

Michal Kulesza

IT student & web developer, interested in DevOps, web applications security and automatization of daily routines.

Warsaw, Poland https://twitter.com/mic_kul

Share this post

Proudly published with Ghost