Julio Dutra

Google Kubernetes Engine (GKE) is a managed Kubernetes service that allows you to run containerized applications on Google Cloud. Monitoring your GKE cluster is essential for maintaining optimal performance and identifying potential issues early. Prometheus is a powerful open-source monitoring and alerting toolkit, while Grafana provides beautiful visualizations for metrics. This article will guide you through configuring GKE to push metrics to Prometheus and creating insightful dashboards on Grafana.

1. Setting Up Prometheus on GKE

1.1. Prerequisites

Before setting up Prometheus, ensure you have the following:

• A GKE cluster up and running.
• kubectl configured to interact with your GKE cluster.
• Helm (a Kubernetes package manager) installed on your local machine.

1.2. Install Prometheus using Helm

Helm makes it easy to install Prometheus on a Kubernetes cluster. Follow these steps to deploy Prometheus:

1) Add the Prometheus Community Helm repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

2) Install Prometheus using Helm:

helm install prometheus prometheus-community/prometheus --namespace monitoring --create-namespace

This command installs Prometheus in a new namespace called monitoring. You can verify the installation by

kubectl get pods -n monitoring

You should see Prometheus pods running in the monitoring namespace.

1.3. Configure Prometheus to Scrape Metrics

Prometheus scrapes metrics from endpoints exposed by your applications and Kubernetes components. To configure Prometheus to scrape metrics from your GKE cluster:

1) Create a ServiceMonitor for your applications:

A ServiceMonitor defines how to scrape metrics from a set of services. Create a file named servicemonitor.yaml:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-service-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: example-app
  endpoints:
    - port: metrics

2) Apply the ServiceMonitor to your cluster:

kubectl apply -f servicemonitor.yaml

Ensure that your application’s service exposes a /metrics endpoint on the port specified in the ServiceMonitor file.

2. Setting Up Grafana on GKE

2.1. Install Grafana using Helm

Grafana can also be installed using Helm, making it easy to set up a robust monitoring solution.

1) Add the Grafana Helm repository:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

2) Install Grafana using Helm:

helm install grafana grafana/grafana --namespace monitoring

This command installs Grafana in the same monitoring namespace as Prometheus.

3) Access the Grafana Dashboard:

After installation, you can access the Grafana dashboard by forwarding the service port to your local machine:

kubectl port-forward service/grafana 3000:80 -n monitoring

Open your browser and navigate to http://localhost:3000.

The default login credentials are admin for both username and password. You will be prompted to change the password upon first login.

2.2. Configure Grafana to Use Prometheus as a Data Source

Once you have access to the Grafana dashboard, follow these steps to configure it to use Prometheus as a data source:

1) Add a New Data Source:

• Click on the “Gear” icon (Configuration) in the side menu and select “Data Sources.”
• Click “Add data source” and select “Prometheus.”

2) Configure the Prometheus Data Source:

• In the “HTTP” section, enter the URL for Prometheus (e.g., http://prometheus-server.monitoring.svc.cluster.local).
• Click “Save & Test” to verify the connection.

If the setup is correct, Grafana will display a message confirming that the data source is working.

3. Creating Dashboards on Grafana

With Prometheus configured as a data source in Grafana, you can now create dashboards to visualize your GKE metrics.

3.1. Create a New Dashboard

1) Add a New Dashboard:

• Click on the “+” icon in the side menu and select “Dashboard.”
• Click “Add new panel” to create a new visualization.

2) Configure a Visualization Panel:

• In the “Query” section, select “Prometheus” as the data source.
• Enter a PromQL query to retrieve the desired metrics (e.g., kube_node_status_condition{condition="Ready"} to display the status of all nodes in the cluster).

3) Customize the Panel:

• Use the “Visualization” options to customize the appearance of the panel. You can choose from different visualization types like Graph, Bar Gauge, or Table.
• Add titles, labels, and descriptions to make the panel more informative.

4) Save the Dashboard:

• Once you have configured the panels, click “Save” in the top-right corner.
• Give your dashboard a name and click “Save” again.

3.2. Import Pre-Built Dashboards

Grafana provides a variety of pre-built dashboards for Kubernetes and Prometheus. You can easily import these dashboards to save time and effort:

1) Navigate to the Dashboard Import Page:

• • Click on the “+” icon in the side menu and select “Import.”

2) Import a Dashboard by ID:

• • Enter the ID of a pre-built dashboard from the Grafana website (e.g., 3119 for Kubernetes cluster monitoring).

• • Click “Load” and then select the Prometheus data source configured earlier.

3) Save the Imported Dashboard:

• • Review the imported panels and click “Save” to add the dashboard to your Grafana instance.

Conclusion

By following this guide, you have set up Prometheus to scrape metrics from your GKE cluster and configured Grafana to visualize these metrics through informative dashboards. This setup provides a powerful solution for monitoring and managing Kubernetes clusters, helping you keep an eye on the health and performance of your applications and infrastructure.

For further customization, explore additional Prometheus exporters for more granular metrics and tailor your Grafana dashboards to meet your specific monitoring needs.