AWS / Cloud / How To / Kubernetes
Setup Amazon Managed Service for Prometheus (AMP) on AWS EKS
https://aws.amazon.com/prometheus
Hi everyone! Here i am setting up Amazon Managed Service for Prometheus (AMP) to monitor my AWS EKS. Sharing the journey to and steps to provision AMP below:
Amazon Managed Service for Prometheus (AMP) is an AWS fully-managed monitoring and alerting service which is 100% compatible with Prometheus that makes it easy to monitor containerized applications and infrastructure at scale.
AMP supports the same metrics, the same PromQL queries, and can also make use of the 150+ Prometheus exporters. AMP runs across multiple Availability Zones for high availability, and is powered by CNCF Cortex for horizontal scalability. AMP will easily scale to ingest, store, and query millions of time series metrics.
The unique architecture for Amazon Managed Prometheus (AMP) is that you will still require a Prometheus server (deployment pod) which retrieve metrics/data from AWS EKS API and pushes those data to Amazon Managed Prometheus (AMP).
- Amazon Managed Service for Prometheus Documentation: https://docs.aws.amazon.com/prometheus/
Deployment of Amazon Managed Prometheus (AMP)
Prerequisite:
- Provision VPC, Subnets, Route Tables and Security Groups
- Existing or new AWS EKS Cluster
- Visual Studio Code IDE (Optional)
Step1: Configure Permission for Prometheus & Grafana
Enable service accounts to access AWS resources in three steps
1. Create an IAM OIDC provider for your cluster – You only need to do this once for a cluster.
2. Create an IAM role and attach an IAM policy to it with the permissions that your service accounts need – We recommend creating separate roles for each unique collection of permissions that pods need.
3. Associate an IAM role with a service account – Complete this task for each Kubernetes service account that needs access to AWS resources.
From the steps above, what you are performing is to create an AWS IAM role with attached IAM policy to be associated to a Kubernetes service account.
Step2: Create an AWS Prometheus Workspace
AWS Console > AWS AMP > Create a AWS Prometheus workspace (eg. Prometheus-Workspace )
Step3: Deploy Prometheus server
Amazon Managed Service for Prometheus does not directly scrape operational metrics from containerized workloads in a Kubernetes cluster. It requires users to deploy and manage a standard Prometheus server.
Set the following environment variables before deploying the helm chart :
IAM_PROXY_PROMETHEUS_ROLE_ARN=arn:aws:iam::<AWS-Account-ID>:role/<EKS-AMP-ServiceAccount-Role>
WORKSPACE_ID=<AWS Prometheus Workspace ID>
AWS_REGION=<your AWS account region>- ‘IAM_PROXY_PROMETHEUS_ROLE_ARN‘ – is the IAM role name you have created in Step1
- ‘WorkSPACE_ID‘ – is the Prometheus Workspace ID which you have created a Prometheus Workspace in Step2
- ‘AWS_Region‘ – is the region in where your AWS Prometheus is deployed, i have input ‘ap-southeast-1‘ which is the Singapore region.
Now create a file called ‘amp_ingest_override_values.yaml‘ with the following content in it:
‘iamproxy-service-account‘ is the newly created Kubernetes service account
The following code below is a set of default values for prometheus server helm chart which enable remoteWrite to AMP:
#amp_ingest_override_values.yaml
serviceAccounts:
## Disable alert manager roles
##
server:
name: "iamproxy-service-account"
alertmanager:
create: false
## Disable pushgateway
##
pushgateway:
create: false
server:
remoteWrite:
-
queue_config:
max_samples_per_send: 1000
max_shards: 200
capacity: 2500
## Use a statefulset instead of a deployment for resiliency
##
statefulSet:
enabled: true
## Store blocks locally for short time period only
##
retention: 1h
## Disable alert manager
##
alertmanager:
enabled: false
## Disable pushgateway
##
pushgateway:
enabled: falseDeploy the helm chart:
kubectl apply -f amp_ingest_override_values.yamlNow we can deploy the Prometheus server (Pod) after you have set the configuration:
helm repo add prometheus-community https://prometheus-community.github.io/helm-chart
kubectl create ns prometheus
helm install prometheus-for-amp prometheus-community/prometheus -n prometheus -f ./amp_ingest_override_values.yaml \
--set serviceAccounts.server.annotations."eks\.amazonaws\.com/role-arn"="${IAM_PROXY_PROMETHEUS_ROLE_ARN}" \
--set server.remoteWrite[0].url="https://aps-workspaces.${AWS_REGION}.amazonaws.com/workspaces/${WORKSPACE_ID}/api/v1/remote_write" \
--set server.remoteWrite[0].sigv4.region=${AWS_REGION}Now check the Service account in Kubernetes (AWS EKS):
kubectl get ServiceAccount -A
kubectl get pods --all-namespaces
Step4: Deploy Grafana
Enter the following code to deploy Grafana pods in your AWS EKS cluster :
#Add Grafana helm chart rep
helm repo add grafana https://grafana.github.io/helm-charts
#Create a Grafana namespace
kubectl create ns grafana
#Install Grafana for AMP
helm install grafana-for-amp grafana/grafana -n grafana
#Check the Grafana pods
kubectl get pods --all-namespaces- Now Update your Grafana server to use the AWS signing proxy.
- Create a new file and name it ‘amp_query_override_values.yaml‘. (This file will be used to update your Grafana deployment to enable the Sigv4 protocol which the AWS signing proxy uses to authenticate)
#amp_query_override_values.yaml
serviceAccount
name: "iamproxy-service-account"
annotations:
eks.amazonaws.com/role-arn: "${IAM_PROXY_PROMETHEUS_ROLE_ARN}"
grafana.ini:
auth:
sigv4_auth_enabled: true:- Now execute the following command to update yourGrafana environment
helm upgrade --install grafana-for-amp grafana/grafana -n grafana -f ./amp_query_override_values.yamlPort forward Grafana deployment so we can access the UI from our browser:
#Check Grafana pods
kubectl get pods --all-namespaces
#Port forward Grafana so we can access the dashboard UI from our browser port 5001
kubectl port-forward -n grafana pods/<grafana pod name> 5001:3000Get the default Grafana admin password:
kubectl get secrets grafana-for-amp -n grafana -o jsonpath='{.data.admin-password}'|base64 --decode- Now access via your URL browser: http://localhost:5001
- Username: admin
- Password: <What you generated secrets from above>
Step5: Configure Grafana with Amazon Managed Service for Prometheus
- Before we can visualise the metrics in Grafana, its has to be configured with 1 or more data sources. Lets specify the workspace within Amazon Managed Service for Prometheus (AMP).
- Grafana Configurations > Data Sources > Select Prometheus as data source
2. Input the AWS AMP Workspace query URL under HTTP/URL
3. Turn on SigV4 auth
4. Change default region of the AMP and AWS EKS cluster resides in
5. Lastly, Save & Test the data source. (Ensure the tick and Data source is working)
Step6: Deploy dashboard to monitor our workload in AWS EKS Cluster
- On the main page of Grafana login, lets add in some default dashboard by selecting Import:
- Input ‘3119’ and press Load [ Kubernetes cluster monitoring (via Prometheus) ]
- Then select Prometheus > Import
Done! You have completed deploying Amazon Managed Service for Prometheus (AMP) together with Grafana into your AWS EKS cluster
Credits/References:
- Cloudonaut ( https://cloudonaut.io/ )
- https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-create-workspace.html
Disclaimer:
The views expressed and the content shared in all published articles on this website are solely those of the respective authors, and they do not necessarily reflect the views of the author’s employer or the techbeatly platform. We strive to ensure the accuracy and validity of the content published on our website. However, we cannot guarantee the absolute correctness or completeness of the information provided. It is the responsibility of the readers and users of this website to verify the accuracy and appropriateness of any information or opinions expressed within the articles. If you come across any content that you believe to be incorrect or invalid, please contact us immediately so that we can address the issue promptly.
Tags:
Latest posts
Comments
1 Response
Leave a Reply
[…] on the previous article of setting up Amazon Managed Service for Prometheus (AMP) with open source Grafana , we will now be provisioning Amazon Managed Grafana to integrated with […]