OpenShift 4: Query Prometheus API for Firing Alert

Introduction

Prometheus is a robust, efficient and cloud native monitoring tools, graduated from CNCF incubator. It was part of the OpenShift since OpenShift 3.11. In OpenShift 4.x it become part of the platform core component.

In this quick blog, we will show you how to query OpenShift Prometheus API that already being exposed by route. This is very useful for external monitoring integration solution like Nagios active check that can query Prometheus for firing alert.

How to get this done?

First you need an authorization bearer token. In this blog we will use short-lived user token, however for proper integration, we can use ServiceAccount long-lived token with assigned RBAC.

# export TOKEN=$( oc whoami -t)
# export URL=https://prometheus-k8s-openshift-monitoring.apps.ocp4.local.bytewise.my

#curl -s -g -k -X GET -H "Authorization: Bearer $TOKEN" -H 'Accept: application/json' -H 'Content-Type: application/json' "$URL/api/v1/alerts" | jq  -c '.data.alerts[] | select( .state == "firing" and .labels.severity == "critical")'

OUTPUT:
{"labels":{"alertname":"KubeAPIErrorsHigh","resource":"events","severity":"critical","verb":"LIST"},"annotations":{"message":"API server is returning errors for 45.45% of requests for LIST events ."},"state":"firing","activeAt":"2020-11-04T04:12:20.907806203Z","value":"4.545454545454545e-01"}

For full and official Prometheus API documentation, we may refer to https://prometheus.io/docs/prometheus/latest/querying/api/.