Navigate back to the homepage

Building a Prometheus Exporter

Nancy Chauhan
May 4th, 2021 · 3 min read

Prometheus is an open-source monitoring tool for collecting metrics from your application and infrastructure. As one of the foundations of the cloud-native environment, Prometheus has become the de-facto standard for visibility in the cloud-native landscape.

How Prometheus Works?

Prometheus is a time-series database and a pull-based monitoring system. It periodically scrapes HTTP endpoints (targets) to retrieve metrics. It can monitor targets such as servers, databases, standalone virtual machines, etc. Prometheus read metrics exposed by target using a simple text-based exposition format. There are client libraries that help your application to expose metrics in Prometheus format.

How Prometheus Works?

Prometheus Metrics

While working with Prometheus it is important to know about Prometheus metrics. These are the four types of metrics that will help in instrumenting your application:

  • Counter (the only way is up): Use counters for counting events, jobs, money, HTTP request, etc. where a cumulative value is useful.
  • Gauges (the current picture): Use where the current value is important — CPU, RAM, JVM memory usage, queue levels, etc.
  • Histograms (Sampling Observations): Generally use with timings, where an overall picture over a time frame is required — query times, HTTP response times.
  • Summaries (client-side quantiles): Similar in spirit to the Histogram, with the difference being that quantiles are calculated on the client-side as well. Use when you start using quantile values frequently with one or more histogram metrics.

Using Prometheus

  • Prometheus provides client libraries that you can use to add instrumentation to your applications.
  • The client library exposes your metrics at URLs such as http://localhost:8000/metrics
  • Configure the URL as one of the targets in Prometheus. Prometheus will now scrape metrics in periodic intervals. You can use visualization tools such as Grafana to view your metrics or configure alerts using Alertmanager via custom rules defined in configuration files.

Prometheus Exporters

Exporter

Prometheus has a huge ecosystem of exporters. Prometheus exporters bridge the gap between Prometheus and applications that don’t export metrics in the Prometheus format. For example, Linux does not expose Prometheus-formatted metrics. That’s why Prometheus exporters, like the node exporter, exist.

Some applications like Spring Boot, Kubernetes, etc. expose Prometheus metrics out of the box. On the other hand, exporters consume metrics from an existing source and utilize the Prometheus client library to export metrics to Prometheus.

Prometheus exporters can be stateful or stateless. A stateful exporter is responsible for gathering data and exports them using the general metrics format such as counter, gauge, etc. Stateless exporters are exporters that translate metrics from one format to Prometheus metrics format using counter metric family, gauge metric family, etc. They do not maintain any local state instead they show a view derived from another metric source such as JMX. For example, Jenkins Jobmon is a Prometheus exporter for Jenkins which calls Jenkins API to fetch the metrics on every scrape.

https://github.com/grofers/jenkins-jobmon

Let’s build a generic HTTP server metrics exporter!

We will build a Prometheus exporter for monitoring HTTP servers from logs. It extracts data from HTTP logs and exports it to Prometheus. We will be using a python client library, prometheus_client, to define and expose metrics via an HTTP endpoint.

One of the metrics from httpd_exporter

Our HTTP exporter will repeatedly follow server logs to extract useful information such as HTTP requests, status codes, bytes transferred, and requests timing information. HTTP logs are structured and standardized across different servers such as Apache, Nginx, etc. You can read more about it from here.

1127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
  • We will use a counter metric to store the HTTP requests using status code as a label.
  • We will use a counter metric to store bytes transferred.

Here is the script which collects data from apache logs indefinitely and exposes metrics to Prometheus :

https://gist.github.com/Nancy-Chauhan/105d9db52a06fe37d0039cad6a037a93

The follow_log function tails apache logs stored var/log/apache in your system infinitely. gather_metrics() uses a regular expression to fetch the useful information from logs like status_code and total_bytes_sent and accordingly increments the counters.

If you run the script, it will start the server at http://localhost:8000 The collected metrics will show up there. Setup Prometheus to scrape the endpoint. Over time, Prometheus will build the time-series for the metrics collected. Setup Grafana to visualize the data within Prometheus.

You can find the code here and run the exporter:

https://github.com/Nancy-Chauhan/httpd_exporter

Originally Posted at https://medium.com/@_nancychauhan/building-a-prometheus-exporter-8a4bbc3825f5

More articles from Nancy Chauhan

Introduction to Message Queue:Build a newsletter app using Django, Celery, and RabbitMQ in 30 min

Messaging Queues are widely used in asynchronous systems. In a data-intensive application using queues makes sure users have a fast experience while still completing complicated tasks. For instance, you can show a progress bar in your UI while your task is being completed in the background. This allows the user to relieve themselves from waiting for a task to complete and, hence, can do other jobs during that time.

April 26th, 2021 · 4 min read

Top 10 Productive Hacks for Software Developers

Recently, I posted a tweet asking all the amazing software developers in my network to tell about the hacks they use to keep themselves productive. I have compiled the wonderful and helpful solutions I received. Hope you find them useful as well.

April 17th, 2021 · 4 min read
© 2018–2021 Nancy Chauhan
Link to $https://twitter.com/_nancychauhanLink to $https://github.com/Nancy-ChauhanLink to $https://www.linkedin.com/in/nancy-chauhan/Link to $https://www.instagram.com/_nancychauhan/Link to $https://medium.com/@_nancychauhan