ServiceMonitor resource not picked up by Prometheus kube-prometheus-stack Installation

Prometheus-kube-servicemonitor-not-picked-up.md

As per prometheus-operator/kube-prometheus#1392 Thank you to KaiGrassnick!

The underlying issue is a configuration setting in the kube-prometheus-stack helm chart, that does by default expect you to label your ServiceMonitor in a certain way. For me with no further configuration, the expected label was the helm-deployments "release" label. Since I deployed my blackbox_exporter with a separate helm-cart in ArgoCD, that label wasn't applied and the ServiceMonitor wasn't picked up.

I am assuming you already installed kube-prometheus-stack and it's working. I am also assuming you're using the blackbox_exporter chart from here: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-blackbox-exporter

For the blackbox_exporter chart values.yaml I use the following overrides for testing:

automountServiceAccountToken: true
servicemonitor:
  enabled: true
  selfmonitor:
    enabled: true
releaseLabel: true

Now the issue arises: While the ServiceMonitor Resource is created in the same namespace as kube-prometheus-stack, it's not picked up by Prometheus as a target.

We can verify if kube-prometheus-stack's Prometheus is actually using a Selector for the ServiceMonitor:

kubectl get prometheuses.monitoring.coreos.com --all-namespaces -o jsonpath="{.items[*].spec.serviceMonitorSelector}"

Result:

{"matchLabels":{"release":"kube-prometheus-stack"}}

If you want to pick up all ServiceMonitor Resources in your cluster override the following on your helm values.yml for kube-prometheus-stack

prometheus:
  prometheusSpec:
    ruleSelectorNilUsesHelmValues: false
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    probeSelectorNilUsesHelmValues: false