Kubernetes – After server reboot – Error getting node err=node … not found

Issue

This Content is from Stack Overflow. Question asked by GrasWindowWind

I had restarted the server (master node) and I get since then the following message when I want to use kubelet:
The connection to the server YYY.YYY.YYY.YY:6443 was refused – did you specify the right host or port?

In the log of kubelet it says “Error getting node” err=“node “jupyterhub-test” not found”. jupyterhub-test is the master node.

The only things I always found while researching, I already tested.

  • restart kubelet
  • restart docker
  • swapoff -a

I can’t figure out from the log of kubelet where exactly the problem is and I hope you can help me.

Cluster information:
Kubernetes version: Client Version: version.Info{Major:“1”, Minor:“24”, GitVersion:“v1.24.1”, GitCommit:“3ddd0f45aa91e2f30c70734b175631bec5b5825a”, GitTreeState:“clean”, BuildDate:“2022-05-24T12:26:19Z”, GoVersion:“go1.18.2”, Compiler:“gc”, Platform:“linux/amd64”}
Installation method: kubeadm init
Host OS: Ubuntu Ubuntu 20.04.4 LTS

kubelet log:

Sep 14 08:20:37 jupyterhub-test systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: A start job for unit kubelet.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit kubelet.service has finished successfully.
--
-- The job identifier is 7916.
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: Flag --container-runtime has been deprecated, will be removed in 1.27 as the only valid value is 'remote'
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.562908  178160 server.go:193] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: Flag --container-runtime has been deprecated, will be removed in 1.27 as the only valid value is 'remote'
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.571728  178160 server.go:399] "Kubelet version" kubeletVersion="v1.24.1"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.571751  178160 server.go:401] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.571927  178160 server.go:813] "Client rotation is on, will bootstrap in background"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.572804  178160 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.573378  178160 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.575898  178160 server.go:648] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576200  178160 container_manager_linux.go:262] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576261  178160 container_manager_linux.go:267] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: KubeletOOMScoreAdj:-999 ContainerRuntime: CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerPolicyOptions:map[] ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576319  178160 topology_manager.go:133] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576328  178160 container_manager_linux.go:302] "Creating device plugin manager" devicePluginEnabled=true
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576359  178160 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.579169  178160 kubelet.go:376] "Attempting to sync node with API server"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.579187  178160 kubelet.go:267] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.579203  178160 kubelet.go:278] "Adding apiserver pod source"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.579211  178160 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: WARN 08:20:37.580088  178160 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Node: Get "https://YYY.YYY.YYY.YY:6443/api/v1/nodes?fieldSelector=metadata.name%3Djupyterhub-test&limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.580150  178160 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://YYY.YYY.YYY.YY:6443/api/v1/nodes?fieldSelector=metadata.name%3Djupyterhub-test&limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.580230  178160 kuberuntime_manager.go:239] "Container runtime initialized" containerRuntime="containerd" version="1.6.6" apiVersion="v1"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.581022  178160 server.go:1181] "Started kubelet"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.581694  178160 server.go:150] "Starting to listen" address="0.0.0.0" port=10250
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.582261  178160 cri_stats_provider.go:455] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.582351  178160 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.582738  178160 event.go:276] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"jupyterhub-test.1714ac53a07755b3", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ZZZ_DeprecatedClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"jupyterhub-test", UID:"jupyterhub-test", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"Starting", Message:"Starting kubelet.", Source:v1.EventSource{Component:"kubelet", Host:"jupyterhub-test"}, FirstTimestamp:time.Date(2022, time.September, 14, 8, 20, 37, 580993971, time.Local), LastTimestamp:time.Date(2022, time.September, 14, 8, 20, 37, 580993971, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Post "https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/default/events": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused'(may retry after sleeping)
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: WARN 08:20:37.582943  178160 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Service: Get "https://YYY.YYY.YYY.YY:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.583036  178160 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://YYY.YYY.YYY.YY:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.583351  178160 server.go:410] "Adding debug handlers to kubelet server"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.582359  178160 kubelet.go:1298] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.585879  178160 volume_manager.go:289] "Starting Kubelet Volume Manager"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.585928  178160 desired_state_of_world_populator.go:145] "Desired state populator starts to run"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.586167  178160 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: Get "https://YYY.YYY.YYY.YY:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/jupyterhub-test?timeout=10s": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: WARN 08:20:37.586759  178160 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.CSIDriver: Get "https://YYY.YYY.YYY.YY:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.586877  178160 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://YYY.YYY.YYY.YY:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.598425  178160 kubelet_network_linux.go:76] "Initialized protocol iptables rules." protocol=IPv4
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.614860  178160 kubelet_network_linux.go:76] "Initialized protocol iptables rules." protocol=IPv6
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.614881  178160 status_manager.go:161] "Starting to sync pod status with apiserver"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.614893  178160 kubelet.go:1974] "Starting kubelet main sync loop"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.615122  178160 kubelet.go:1998] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: WARN 08:20:37.616041  178160 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.RuntimeClass: Get "https://YYY.YYY.YYY.YY:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.616074  178160 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get "https://YYY.YYY.YYY.YY:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665155  178160 cpu_manager.go:213] "Starting CPU manager" policy="none"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665168  178160 cpu_manager.go:214] "Reconciling" reconcilePeriod="10s"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665179  178160 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665277  178160 state_mem.go:88] "Updated default CPUSet" cpuSet=""
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665286  178160 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665291  178160 policy_none.go:49] "None policy: Start"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.666175  178160 memory_manager.go:168] "Starting memorymanager" policy="None"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.666199  178160 state_mem.go:35] "Initializing new in-memory state store"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.666468  178160 state_mem.go:75] "Updated machine memory state"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.670363  178160 manager.go:610] "Failed to read data from checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.670567  178160 plugin_manager.go:114] "Starting Kubelet Plugin Manager"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.671373  178160 eviction_manager.go:254] "Eviction manager: failed to get summary stats" err="failed to get node info: node "jupyterhub-test" not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.686080  178160 kubelet.go:2419] "Error getting node" err="node "jupyterhub-test" not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.686550  178160 kubelet_node_status.go:70] "Attempting to register node" node="jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.686984  178160 kubelet_node_status.go:92] "Unable to register node with API server" err="Post "https://YYY.YYY.YYY.YY:6443/api/v1/nodes": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused" node="jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.716206  178160 topology_manager.go:200] "Topology Admit Handler"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.716954  178160 topology_manager.go:200] "Topology Admit Handler"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.717589  178160 topology_manager.go:200] "Topology Admit Handler"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.717932  178160 status_manager.go:664] "Failed to get status for pod" podUID=6caff56f64d1ee735407a5a7ba6a787a pod="kube-system/etcd-jupyterhub-test" err="Get "https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/kube-system/pods/etcd-jupyterhub-test": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718167  178160 topology_manager.go:200] "Topology Admit Handler"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718644  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="daf5debb6988d121c82972b4d0f6c1935c005063b7f42da0c10773efbf0525e1"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718776  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="435139d2c5996b2f24600f7cd4911fc77a92bf06e7361347971e70a68e0a54e5"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718849  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="3554bcaebcaf46d227f480181f505d48af9a6f9d389e93f1c6d3fab703db8df4"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718927  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="257b423a7d1730e88f277239d535923ffac18d5430b723aee160d50af5417553"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718995  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="cae22d913fbc0ce98a6e7b0d8b13cef758179416dea8f7a9dd875ab5172d8a4f"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719055  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="c86abd3d68a6a6f596481f8784db328cc2bf9951b0a514bff979e4243b63e9ce"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719113  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="0db53f26fd3fc078c02941576d2187b8525fe2da77a2e6dffff797bba8b12213"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719177  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="19d5c5e57cd43c00235306c69378fd6a9d224a337a79ab0d19417bb7ae8c91b4"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719239  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="84298f6839000667cb1d02bc812a84c94920e088d5649821fb6ebe3dabc13698"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719301  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="4d624b46c1721a68bf6b8b1a57d0aa23ee8d664108114337234826a1e053c991"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719370  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="03454711bc28b88c9a5b4eeb8ab8c962e758bb8ae34cbee798ecd5651ca37bc8"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719429  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="1a29e83515332ebd4cf4eabecdaa4c305e280de78a90d90210fd4d467a28bffd"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719490  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="1b8edc5b07c4a5eab18f9e7edb73b089ea303a5f393b650deb550935456c5df8"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718813  178160 status_manager.go:664] "Failed to get status for pod" podUID=8cdd26801ec71603976b3d4c3c72beae pod="kube-system/kube-apiserver-jupyterhub-test" err="Get "https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-jupyterhub-test": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719777  178160 status_manager.go:664] "Failed to get status for pod" podUID=c9c46b502c40273f3cde89de382eb13b pod="kube-system/kube-controller-manager-jupyterhub-test" err="Get "https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-jupyterhub-test": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.720022  178160 status_manager.go:664] "Failed to get status for pod" podUID=a7527c69e782fb5d6404c82767da6341 pod="kube-system/kube-scheduler-jupyterhub-test" err="Get "https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-jupyterhub-test": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.786540  178160 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: Get "https://YYY.YYY.YYY.YY:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/jupyterhub-test?timeout=10s": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.786583  178160 kubelet.go:2419] "Error getting node" err="node "jupyterhub-test" not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.786993  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-k8s-certs") pod "kube-apiserver-jupyterhub-test" (UID: "8cdd26801ec71603976b3d4c3c72beae") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787171  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "usr-local-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-usr-local-share-ca-certificates") pod "kube-apiserver-jupyterhub-test" (UID: "8cdd26801ec71603976b3d4c3c72beae") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787275  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "usr-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-usr-share-ca-certificates") pod "kube-apiserver-jupyterhub-test" (UID: "8cdd26801ec71603976b3d4c3c72beae") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787372  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/6caff56f64d1ee735407a5a7ba6a787a-etcd-certs") pod "etcd-jupyterhub-test" (UID: "6caff56f64d1ee735407a5a7ba6a787a") " pod="kube-system/etcd-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787406  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-data" (UniqueName: "kubernetes.io/host-path/6caff56f64d1ee735407a5a7ba6a787a-etcd-data") pod "etcd-jupyterhub-test" (UID: "6caff56f64d1ee735407a5a7ba6a787a") " pod="kube-system/etcd-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787431  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-ca-certs") pod "kube-apiserver-jupyterhub-test" (UID: "8cdd26801ec71603976b3d4c3c72beae") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787452  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "etc-ca-certificates" (UniqueName: "kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-etc-ca-certificates") pod "kube-apiserver-jupyterhub-test" (UID: "8cdd26801ec71603976b3d4c3c72beae") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787479  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "etc-pki" (UniqueName: "kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-etc-pki") pod "kube-apiserver-jupyterhub-test" (UID: "8cdd26801ec71603976b3d4c3c72beae") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.887309  178160 kubelet.go:2419] "Error getting node" err="node "jupyterhub-test" not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.887665  178160 kubelet_node_status.go:70] "Attempting to register node" node="jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889786  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-ca-certs") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889824  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "etc-pki" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-etc-pki") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889845  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-k8s-certs") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889865  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "usr-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-usr-share-ca-certificates") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889885  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/a7527c69e782fb5d6404c82767da6341-kubeconfig") pod "kube-scheduler-jupyterhub-test" (UID: "a7527c69e782fb5d6404c82767da6341") " pod="kube-system/kube-scheduler-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889930  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "flexvolume-dir" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-flexvolume-dir") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889950  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-kubeconfig") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889970  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "usr-local-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-usr-local-share-ca-certificates") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.890011  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "etc-ca-certificates" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-etc-ca-certificates") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.890469  178160 kubelet_node_status.go:92] "Unable to register node with API server" err="Post "https://YYY.YYY.YYY.YY:6443/api/v1/nodes": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused" node="jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.988241  178160 kubelet.go:2419] "Error getting node" err="node "jupyterhub-test" not found"
Sep 14 08:20:38 jupyterhub-test kubelet[178160]: INFO 08:20:38.019896  178160 scope.go:110] "RemoveContainer" containerID="cc4b38a68d8e34264d83b0859a9b9c84b6d71b07f886ba76a9d18e6e2fa63d81"
Sep 14 08:20:38 jupyterhub-test kubelet[178160]: INFO 08:20:38.019999  178160 scope.go:110] "RemoveContainer" containerID="8ceae598793f115e91fdd87e77b087d5ce5ccede2c719ae78c3c7a3c477bf714"
Sep 14 08:20:38 jupyterhub-test kubelet[178160]: ERROR 08:20:38.088723  178160 kubelet.go:2419] "Error getting node" err="node "jupyterhub-test" not found"
Sep 14 08:20:38 jupyterhub-test kubelet[178160]: ERROR 08:20:38.188858  178160 kubelet.go:2419] "Error getting node" err="node "jupyterhub-test" not found"



Solution

This question is not yet answered, be the first one who answer using the comment. Later the confirmed answer will be published as the solution.

This Question and Answer are collected from stackoverflow and tested by JTuto community, is licensed under the terms of CC BY-SA 2.5. - CC BY-SA 3.0. - CC BY-SA 4.0.

people found this article helpful. What about you?