How to use Prometheus for metrics

Introduction

Light-4j provide Prometheus metrics handler as MiddlewareHandler to collects the API runtime information and save it to Prometheus Metric data module. And then use pre-defined HttpHandler (PrometheusGetHandler) to expose the prometheus Metric data module; Prometheus server will pull the metric from metrics_path which is configurable on

Prometheus config yml file (for exampe: /v1/prometheus) ; The pulling interval is second base on the config file (scrape_interval: 30s).

Configuration

Configuration file (prometheus.yml) for the MiddlewareHandler (com.networknt.metrics.prometheus.PrometheusHandler)

# Metrics handler configuration

# If prometheus metrics handler is enabled or not
enabled: false

Change the enabled value to true to enable the Prometheus metrics handler;

Configuration for services

If user want to use Prometheus metrics handler as metrics MiddlewareHandler handler for the service, simply set the config “prometheusMetrics” as true

Config to generate new service by light-codegen

Here is an example of config.json for openapi generator.

{
  "name": "petstore",
  "version": "1.0.1",
  "groupId": "com.networknt",
  "artifactId": "petstore",
  "rootPackage": "com.networknt.petstore",
  "handlerPackage":"com.networknt.petstore.handler",
  "modelPackage":"com.networknt.petstore.model",
  "overwriteHandler": true,
  "overwriteHandlerTest": true,
  "overwriteModel": true,
  "httpPort": 8080,
  "enableHttp": true,
  "httpsPort": 8443,
  "enableHttps": false,
  "enableRegistry": false,
  "supportDb": true,
  "dbInfo": {
    "name": "mysql",
    "driverClassName": "com.mysql.jdbc.Driver",
    "jdbcUrl": "jdbc:mysql://mysqldb:3306/oauth2?useSSL=false",
    "username": "root",
    "password": "my-secret-pw"
  },
  "supportH2ForTest": false,
  "supportClient": false,
  "prometheusMetrics": true
}

Then following the step to generate service API.

The generated service will use Prometheus metrics handler as metrics MiddlewareHandler handler and the the API runtime information will be save to prometheus Metric data module.

Change existing service to use light-4j Prometheus MiddlewareHandler handler

If user want to change existing service which created before based on light-4j palteform (default use [InfluxDB metrics handler]: /concern/metrics/) to use Prometheus metrics handler, please change the service by following steps:

Change the service POM to add light-4j Prometheus MiddlewareHandler handler dependency:

        <dependency>
            <groupId>com.networknt</groupId>
            <artifactId>prometheus</artifactId>
            <version>${version.light-4j}</version>
        </dependency>

Change the service.yml file to use Prometheus MiddlewareHandler handler:

- com.networknt.handler.MiddlewareHandler:
  # Exception Global exception handler that needs to be called first to wrap all middleware handlers and business handlers
  - com.networknt.exception.ExceptionHandler
  # Metrics handler to calculate response time accurately, this needs to be the second handler in the chain.
  - com.networknt.metrics.prometheus.PrometheusHandler
  ...

Change the api spec (openapi.json or swagger.json) by adding Prometheus metrics_path:

"/prometheus":{"get":{"responses":{"200":{"description":"successful operation"}},"parameters":[]}}

Change the

import com.networknt.metrics.prometheus.PrometheusGetHandler;

public class PathHandlerProvider implements HandlerProvider {
    @Override
    public HttpHandler getHandler() {
        return Handlers.routing()
                .add(Methods.GET, "/v1/prometheus", new PrometheusGetHandler());
    }
}

The metrics_path is configurable. In light-codegen, we set it as /prometheus, but it can be change to any url user want. Then user just to to set the same “metrics_path” in the Prometheus config file.

Configuration for Prometheus (config with Consul )

Prometheus is configured via a configuration file. The file is written in YAML format, defined by the Prometheus scheme. By default, the targets ( nodes to monitor ) are provided statically in the prometheus config file. Static targets are simple when we know in advance our infrastructure. But for the microservice architecture, service API will have dynamic target on the cloud. So our solution is use consul to provide targets for prometheus to monitor dynamically

In prometheus, we need to configure it use consul_sd_targets. Prometheus will then query consul http interface for the catalog of targets to monitor.

Here is the sample config file. It can be found from link:

  - job_name: 'services'
    scrape_interval: 15s
    consul_sd_configs:
      - server: 'consul:8500'
        services:
          - 'multimodule_tram-todolist-command-service'
          - 'multimodule_tram-todolist-view-service'

    relabel_configs:
      - source_labels: ['__meta_consul_service']
        regex:         '(.*)'
        target_label:  'service'
        replacement:   '$1'
      - source_labels: ['__meta_consul_tags']
        target_label: 'slave_name'
        regex: '.*slave_name=([^,]*).*'
        replacement: '${1}'
      - source_labels: ['__meta_consul_tags']
        target_label: 'public_hostname'
        regex: '.*public_hostname=([^,]*).*'
        replacement: '${1}'

    metrics_path: /v1/prometheus
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

Verification:

The latest version of light-tram-4j example todo-list has been change to use prometheus metrics handler; Please follow the step in the tutrial to start the sample service api:

light-tram todo-list service API

Starts consul service discovery by docker-compose:


 cd ~/networknt/light-docker
 docker-compose -f docker-compose-consul.yml up

Starts registrator service by docker-compose:


 cd ~/networknt/light-docker
 docker-compose -f docker-compose-registrator.yml up

Starts Prometheus server by docker-compose (the Prometheus server will start based on the config on ~/networknt/light-docker/prometheus/prometheus.yml. User can change it for different api):


 cd ~/networknt/light-docker
 docker-compose -f docker-compose-prometheus.yml up

Open a browser to verify Consul:

http://127.0.0.1:8500/

Open a browser to verify Prometheus:

http://localhost:9090/

Using Prometheus in Grafana:

Light framwork provided docker-compose file (docker-compose-prometheus.yml) includes the Grafana image: image: grafana/grafana:4.6.3

So user can open browser to view the Prometheus from Grafana

http://127.0.0.1:3000/

user: admin
passwork: admin

Please follow the following step to set the Prometheus in Grafana:

Adding the data source to Grafana:

Open the side menu by clicking the Grafana icon in the top header.
In the side menu under the Dashboards link you should find a link named Data Sources.
Click the + Add data source button in the top header.
Select Prometheus from the Type dropdown.

You can get the detail from: http://docs.grafana.org/features/datasources/prometheus/