Setup Amazon Managed Service for Prometheus (AMP) on AWS EKS

Tan Kai Jian

3 years ago

https://aws.amazon.com/prometheus

Hi everyone! Here i am setting up Amazon Managed Service for Prometheus (AMP) to monitor my AWS EKS. Sharing the journey to and steps to provision AMP below:

Amazon Managed Service for Prometheus (AMP) is an AWS fully-managed monitoring and alerting service which is 100% compatible with Prometheus that makes it easy to monitor containerized applications and infrastructure at scale.

AMP supports the same metrics, the same PromQL queries, and can also make use of the 150+ Prometheus exporters. AMP runs across multiple Availability Zones for high availability, and is powered by CNCF Cortex for horizontal scalability. AMP will easily scale to ingest, store, and query millions of time series metrics.

The unique architecture for Amazon Managed Prometheus (AMP) is that you will still require a Prometheus server (deployment pod) which retrieve metrics/data from AWS EKS API and pushes those data to Amazon Managed Prometheus (AMP).

Amazon Managed Service for Prometheus Documentation: https://docs.aws.amazon.com/prometheus/

Deployment of Amazon Managed Prometheus (AMP)

Prerequisite:

Provision VPC, Subnets, Route Tables and Security Groups
Existing or new AWS EKS Cluster
Visual Studio Code IDE (Optional)

Step1: Configure Permission for Prometheus & Grafana

Enable service accounts to access AWS resources in three steps

1. Create an IAM OIDC provider for your cluster – You only need to do this once for a cluster.

2. Create an IAM role and attach an IAM policy to it with the permissions that your service accounts need – We recommend creating separate roles for each unique collection of permissions that pods need.

3. Associate an IAM role with a service account – Complete this task for each Kubernetes service account that needs access to AWS resources.

From the steps above, what you are performing is to create an AWS IAM role with attached IAM policy to be associated to a Kubernetes service account.

Step2: Create an AWS Prometheus Workspace

AWS Console > AWS AMP > Create a AWS Prometheus workspace (eg. Prometheus-Workspace )

Step3: Deploy Prometheus server

Amazon Managed Service for Prometheus does not directly scrape operational metrics from containerized workloads in a Kubernetes cluster. It requires users to deploy and manage a standard Prometheus server.

Set the following environment variables before deploying the helm chart :

IAM_PROXY_PROMETHEUS_ROLE_ARN=arn:aws:iam::<AWS-Account-ID>:role/<EKS-AMP-ServiceAccount-Role>

WORKSPACE_ID=<AWS Prometheus Workspace ID>

AWS_REGION=<your AWS account region>

‘IAM_PROXY_PROMETHEUS_ROLE_ARN‘ – is the IAM role name you have created in Step1
‘WorkSPACE_ID‘ – is the Prometheus Workspace ID which you have created a Prometheus Workspace in Step2
‘AWS_Region‘ – is the region in where your AWS Prometheus is deployed, i have input ‘ap-southeast-1‘ which is the Singapore region.

Now create a file called ‘amp_ingest_override_values.yaml‘ with the following content in it:

‘iamproxy-service-account‘ is the newly created Kubernetes service account

The following code below is a set of default values for prometheus server helm chart which enable remoteWrite to AMP:

#amp_ingest_override_values.yaml
serviceAccounts:
  ## Disable alert manager roles
  ##
  server:
        name: "iamproxy-service-account"
  alertmanager:
    create: false


  ## Disable pushgateway
  ##
  pushgateway:
    create: false


server:
  remoteWrite:
    -
      queue_config:
        max_samples_per_send: 1000
        max_shards: 200
        capacity: 2500


  ## Use a statefulset instead of a deployment for resiliency
  ##
  statefulSet:
    enabled: true


  ## Store blocks locally for short time period only
  ##
  retention: 1h
  
## Disable alert manager
##
alertmanager:
  enabled: false


## Disable pushgateway
##
pushgateway:
  enabled: false

Deploy the helm chart:

kubectl apply -f amp_ingest_override_values.yaml

Now we can deploy the Prometheus server (Pod) after you have set the configuration:

helm repo add prometheus-community https://prometheus-community.github.io/helm-chart
kubectl create ns prometheus

helm install prometheus-for-amp prometheus-community/prometheus -n prometheus -f ./amp_ingest_override_values.yaml \
--set serviceAccounts.server.annotations."eks\.amazonaws\.com/role-arn"="${IAM_PROXY_PROMETHEUS_ROLE_ARN}" \
--set server.remoteWrite[0].url="https://aps-workspaces.${AWS_REGION}.amazonaws.com/workspaces/${WORKSPACE_ID}/api/v1/remote_write" \
--set server.remoteWrite[0].sigv4.region=${AWS_REGION}

Now check the Service account in Kubernetes (AWS EKS):

kubectl get ServiceAccount -A

kubectl get pods --all-namespaces

Step4: Deploy Grafana

Enter the following code to deploy Grafana pods in your AWS EKS cluster :

#Add Grafana helm chart rep
helm repo add grafana https://grafana.github.io/helm-charts


#Create a Grafana namespace
kubectl create ns grafana


#Install Grafana for AMP
helm install grafana-for-amp grafana/grafana -n grafana


#Check the Grafana pods
kubectl get pods --all-namespaces

Now Update your Grafana server to use the AWS signing proxy.
Create a new file and name it ‘amp_query_override_values.yaml‘. (This file will be used to update your Grafana deployment to enable the Sigv4 protocol which the AWS signing proxy uses to authenticate)

#amp_query_override_values.yaml
serviceAccount
    name: "iamproxy-service-account"
    annotations:
        eks.amazonaws.com/role-arn: "${IAM_PROXY_PROMETHEUS_ROLE_ARN}"
grafana.ini:
  auth:
    sigv4_auth_enabled: true:

Now execute the following command to update yourGrafana environment

helm upgrade --install grafana-for-amp grafana/grafana -n grafana -f ./amp_query_override_values.yaml

Port forward Grafana deployment so we can access the UI from our browser:

#Check Grafana pods
kubectl get pods --all-namespaces


#Port forward Grafana so we can access the dashboard UI from our browser port 5001
kubectl port-forward -n grafana pods/<grafana pod name> 5001:3000

Get the default Grafana admin password:

kubectl get secrets grafana-for-amp -n grafana -o jsonpath='{.data.admin-password}'|base64 --decode

Now access via your URL browser: http://localhost:5001
Username: admin
Password: <What you generated secrets from above>

Step5: Configure Grafana with Amazon Managed Service for Prometheus

Before we can visualise the metrics in Grafana, its has to be configured with 1 or more data sources. Lets specify the workspace within Amazon Managed Service for Prometheus (AMP).