[SOLVED] Grafana only detects the namespace/node where the prometheus server is located


This Content is from Stack Overflow. Question asked by Kent Ho

I have installed a Grafana-loki stack on my Kubernetes cluster in namespace “grafana-loki”. The Kubernetes cluster has several other namespaces which are used for applications. I have exposed kube-state-metrics pod and I viewed the metrics and here is an example of what it looks like.

kube_deployment_status_replicas_updated{namespace="grafana-loki",deployment="loki-stack-prometheus-pushgateway"} 1
kube_deployment_status_replicas_updated{namespace="ns-app-dev",deployment="app-configuration-api"} 1
kube_deployment_status_replicas_updated{namespace="ns-app-dev",deployment="app-monitoring-api"} 1
kube_deployment_status_replicas_updated{namespace="grafana-loki",deployment="loki-stack-grafana"} 1
kube_deployment_status_replicas_updated{namespace="default",deployment="aws-load-balancer-controller"} 2
kube_deployment_status_replicas_updated{namespace="kube-system",deployment="ebs-csi-controller"} 2

Then, I created a stat panel in Grafana with below query:


The stat panel shows 8, which is the correct number. However, when I change the namespace to “grafana-loki”, the stat panel still shows 8, and when I change the namespace to “ns-app-dev”, “default”, or “kube-system”, the stat panel shows “no data”.

The problem is the same for every metrics on kube-state-metrics, which Grafana cannot view the correct number for other namespaces/nodes.


The prometheus server will relabel the metrics with new labels, the original label "namespace" was relabeled as "exported_namespace", and the original label "node" was relabeled as "exported_node".

This Question was asked in StackOverflow by Kent Ho and Answered by Kent Ho It is licensed under the terms of CC BY-SA 2.5. - CC BY-SA 3.0. - CC BY-SA 4.0.

people found this article helpful. What about you?