pki – Kubernetes Metric Server cannot call Kubelet API

In trying to securely install metrics-server on Kubernetes, I’m having problems.

It seems like the metric-server pod is unable to successfully make requests to the Kubelet API on it’s 10250 port.

NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   0/1     1            0           16h

The Metrics Server deployment never becomes ready and it repeats the same sequence of error logs:

I0522 01:27:41.472946       1 serving.go:342] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0522 01:27:41.798068       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0522 01:27:41.798092       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0522 01:27:41.798068       1 dynamic_cafile_content.go:156] "Starting controller" name="request-header::/front-ca/front-proxy-ca.crt"
I0522 01:27:41.798107       1 dynamic_serving_content.go:131] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
I0522 01:27:41.798240       1 secure_serving.go:266] Serving securely on [::]:4443
I0522 01:27:41.798265       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
W0522 01:27:41.798284       1 shared_informer.go:372] The sharedIndexInformer has started, run more than once is not allowed
I0522 01:27:41.898439       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
E0522 01:27:55.297497       1 scraper.go:140] "Failed to scrape node" err="Get "https://192.168.1.106:10250/metrics/resource": context deadline exceeded" node="system76-pc"
E0522 01:28:10.297872       1 scraper.go:140] "Failed to scrape node" err="Get "https://192.168.1.106:10250/metrics/resource": context deadline exceeded" node="system76-pc"
I0522 01:28:10.325613       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0522 01:28:20.325231       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
E0522 01:28:25.297750       1 scraper.go:140] "Failed to scrape node" err="Get "https://192.168.1.106:10250/metrics/resource": context deadline exceeded" node="system76-pc"

I’m running Kubernetes with kubeadm version 1.23.4 and I’m trying to securely use metrics-server.

I’m looking for advice that could help with:

  1. How I can accurately diagnose the problem?
  2. Or alternatively, what configuration seems most fruitful to check first?
  3. Anything that will help with my mental model of which certificates and keys I need to configure explicitly and what is being handled automatically.

So far, I have tried to validate that the I can retrieve API metrics:

kubectl get --raw /api/v1/nodes/system76-pc/proxy/stats/summary

{
  "node": {
    "nodeName": "system76-pc",
    "systemContainers": [
      {
        "name": "kubelet",
        "startTime": "2022-05-20T01:51:28Z",
        "cpu": {
          "time": "2022-05-22T00:48:40Z",
          "usageNanoCores": 59453039,
          "usageCoreNanoSeconds": 9768130002000
        },
        "memory": {
          "time": "2022-05-22T00:48:40Z",
          "usageBytes": 84910080,
          "workingSetBytes": 84434944,
          "rssBytes": 67149824,
          "pageFaults": 893055,
          "majorPageFaults": 290
        }
      },
      {
        "name": "runtime",
        "startTime": "2022-05-20T00:33:24Z",
        "cpu": {
          "time": "2022-05-22T00:48:37Z",
          "usageNanoCores": 24731571,
          "usageCoreNanoSeconds": 3955659226000
        },
        "memory": {
          "time": "2022-05-22T00:48:37Z",
          "usageBytes": 484306944,
          "workingSetBytes": 242638848,
          "rssBytes": 84647936,
          "pageFaults": 56994074,
          "majorPageFaults": 428
        }
      },
      {
        "name": "pods",
        "startTime": "2022-05-20T01:51:28Z",
        "cpu": {
          "time": "2022-05-22T00:48:37Z",
          "usageNanoCores": 292818104,
          "usageCoreNanoSeconds": 45976001446000
        },
        "memory": {
          "time": "2022-05-22T00:48:37Z",
          "availableBytes": 29648396288,
          "usageBytes": 6108573696,

kubectl get --raw /api/v1/nodes/system76-pc/proxy/metrics/resource

# HELP container_cpu_usage_seconds_total [ALPHA] Cumulative cpu time consumed by the container in core-seconds
# TYPE container_cpu_usage_seconds_total counter
container_cpu_usage_seconds_total{container="alertmanager",namespace="flux-system",pod="alertmanager-prometheus-stack-kube-prom-alertmanager-0"} 108.399948 1653182143362
container_cpu_usage_seconds_total{container="calico-kube-controllers",namespace="kube-system",pod="calico-kube-controllers-56fcbf9d6b-n87ts"} 206.442768 1653182144294
container_cpu_usage_seconds_total{container="calico-node",namespace="kube-system",pod="calico-node-p6pxk"} 6147.643669 1653182155672
container_cpu_usage_seconds_total{container="cert-manager",namespace="cert-manager",pod="cert-manager-795d7f859d-8jp4f"} 134.583294 1653182142601
container_cpu_usage_seconds_total{container="cert-manager",namespace="cert-manager",pod="cert-manager-cainjector-5fcddc948c-vw4zz"} 394.286782 1653182151252
container_cpu_usage_seconds_total{container="cert-manager",namespace="cert-manager",pod="cert-manager-webhook-5b64f87794-pl7fb"} 404.53758 1653182140528
container_cpu_usage_seconds_total{container="config-reloader",namespace="flux-system",pod="alertmanager-prometheus-stack-kube-prom-alertmanager-0"} 6.01391 1653182139771
container_cpu_usage_seconds_total{container="config-reloader",namespace="flux-system",pod="prometheus-prometheus-stack-kube-prom-prometheus-0"} 42.706567 1653182130750
container_cpu_usage_seconds_total{container="controller",namespace="flux-system",pod="sealed-secrets-controller-5884bbf4d6-mql9x"} 43.814816 1653182144648
container_cpu_usage_seconds_total{container="controller",namespace="ingress-nginx",pod="ingress-nginx-controller-f9d6fc8d8-sgwst"} 645.109711 1653182141169
container_cpu_usage_seconds_total{container="coredns",namespace="kube-system",pod="coredns-64897985d-crtd9"} 380.682251 1653182141861
container_cpu_usage_seconds_total{container="coredns",namespace="kube-system",pod="coredns-64897985d-rpmxk"} 365.519839 1653182140533
container_cpu_usage_seconds_total{container="dashboard-metrics-scraper",namespace="kubernetes-dashboard",pod="dashboard-metrics-scraper-577dc49767-cbq8r"} 25.733362 1653182141877
container_cpu_usage_seconds_total{container="etcd",namespace="kube-system",pod="etcd-system76-pc"} 4237.357682 1653182140459
container_cpu_usage_seconds_total{container="grafana",namespace="flux-system",pod="prometheus-stack-grafana-757f9b9fcc-9f58g"} 345.034245 1653182154951
container_cpu_usage_seconds_total{container="grafana-sc-dashboard",namespace="flux-system",pod="prometheus-stack-grafana-757f9b9fcc-9f58g"} 123.480584 1653182146757
container_cpu_usage_seconds_total{container="grafana-sc-datasources",namespace="flux-system",pod="prometheus-stack-grafana-757f9b9fcc-9f58g"} 35.851112 1653182145702
container_cpu_usage_seconds_total{container="kube-apiserver",namespace="kube-system",pod="kube-apiserver-system76-pc"} 14166.156638 1653182150749
container_cpu_usage_seconds_total{container="kube-controller-manager",namespace="kube-system",pod="kube-controller-manager-system76-pc"} 4168.427981 1653182148868
container_cpu_usage_seconds_total{container="kube-prometheus-stack",namespace="flux-system",pod="prometheus-stack-kube-prom-operator-54d9f985c8-ml2qj"} 28.79018 1653182155583
container_cpu_usage_seconds_total{container="kube-proxy",namespace="kube-system",pod="kube-proxy-gg2wd"} 67.215459 1653182155156
container_cpu_usage_seconds_total{container="kube-scheduler",namespace="kube-system",pod="kube-scheduler-system76-pc"} 579.321492 1653182147910
container_cpu_usage_seconds_total{container="kube-state-metrics",namespace="flux-system",pod="prometheus-stack-kube-state-metrics-56d4759d67-h6lfv"} 158.343644 1653182153691
container_cpu_usage_seconds_total{container="kubernetes-dashboard",namespace="kubernetes-dashboard",pod="kubernetes-dashboard-69dc48777b-8cckh"} 78.231809 1653182139263
container_cpu_usage_seconds_total{container="manager",namespace="flux-system",pod="helm-controller-dfb4b5478-7zgt6"} 338.974637 1653182143679
container_cpu_usage_seconds_total{container="manager",namespace="flux-system",pod="image-automation-controller-77fd9657c6-lg44h"} 280.841645 1653182154912
container_cpu_usage_seconds_total{container="manager",namespace="flux-system",pod="image-reflector-controller-86db8b6f78-5rz58"} 2909.277578 1653182144081
container_cpu_usage_seconds_total{container="manager",namespace="flux-system",pod="kustomize-controller-cd544c8f8-hxvk6"} 596.392781 1653182152714
container_cpu_usage_seconds_total{container="manager",namespace="flux-system",pod="notification-controller-d9cc9bf46-2jhbq"} 244.387967 1653182142902
container_cpu_usage_seconds_total{container="manager",namespace="flux-system",pod="source-controller-84bfd77bf8-r827h"} 541.650877 1653182148963
container_cpu_usage_seconds_total{container="metrics-server",namespace="flux-system",pod="metrics-server-55bc5f774-zznpb"} 174.229886 1653182146946
container_cpu_usage_seconds_total{container="nfs-subdir-external-provisioner",namespace="flux-system",pod="nfs-subdir-external-provisioner-858745f657-zcr66"} 244.061329 1653182139840
container_cpu_usage_seconds_total{container="node-exporter",namespace="flux-system",pod="prometheus-stack-prometheus-node-exporter-wj2fx"} 29.852036 1653182148779
container_cpu_usage_seconds_total{container="prometheus",namespace="flux-system",pod="prometheus-prometheus-stack-kube-prom-prometheus-0"} 7141.611234 1653182154042
# HELP container_memory_working_set_bytes [ALPHA] Current working set of the container in bytes
# TYPE container_memory_working_set_bytes gauge
container_memory_working_set_bytes{container="alertmanager",namespace="flux-system",pod="alertmanager-prometheus-stack-kube-prom-alertmanager-0"} 2.152448e+07 1653182143362

metric-server config:

 spec:
      containers:
      - args:
        - --secure-port=4443
        - --cert-dir=/tmp
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-preferred-address-types=Hostname
        - --requestheader-client-ca-file=/front-ca/front-proxy-ca.crt
        - --kubelet-certificate-authority=/ca/ca.crt
        image: k8s.gcr.io/metrics-server/metrics-server:v0.6.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          initialDelaySeconds: 20
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: tmp
        - mountPath: /front-ca
          name: front-proxy-ca-dir
        - mountPath: /ca
          name: ca-dir
      dnsPolicy: ClusterFirst
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: metrics-server
      serviceAccountName: metrics-server
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: tmp
      - configMap:
          defaultMode: 420
          name: front-proxy-ca
        name: front-proxy-ca-dir
      - configMap:
          defaultMode: 420
          name: kubelet-ca
        name: ca-dir

kube-apiserver config:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.1.106:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.1.106
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: k8s.gcr.io/kube-apiserver:v1.23.4

Leave a Comment