I had restarted the server (master node) and I get since then the following message when I want to use kubelet:
The connection to the server YYY.YYY.YYY.YY:6443 was refused – did you specify the right host or port?

In the log of kubelet it says “Error getting node” err=“node “jupyterhub-test” not found”. jupyterhub-test is the master node.

The only things I always found while researching, I already tested.

• restart kubelet
• restart docker
• swapoff -a

I can’t figure out from the log of kubelet where exactly the problem is and I hope you can help me.

Cluster information:
Kubernetes version: Client Version: version.Info{Major:“1”, Minor:“24”, GitVersion:“v1.24.1”, GitCommit:“3ddd0f45aa91e2f30c70734b175631bec5b5825a”, GitTreeState:“clean”, BuildDate:“2022-05-24T12:26:19Z”, GoVersion:“go1.18.2”, Compiler:“gc”, Platform:“linux/amd64”}
Installation method: kubeadm init
Host OS: Ubuntu Ubuntu 20.04.4 LTS

kubelet log:

Sep 14 08:20:37 jupyterhub-test systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: A start job for unit kubelet.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit kubelet.service has finished successfully.
--
-- The job identifier is 7916.
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: Flag --container-runtime has been deprecated, will be removed in 1.27 as the only valid value is 'remote'
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.562908  178160 server.go:193] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: Flag --container-runtime has been deprecated, will be removed in 1.27 as the only valid value is 'remote'
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.571728  178160 server.go:399] "Kubelet version" kubeletVersion="v1.24.1"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.571751  178160 server.go:401] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.571927  178160 server.go:813] "Client rotation is on, will bootstrap in background"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.572804  178160 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.573378  178160 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.575898  178160 server.go:648] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576200  178160 container_manager_linux.go:262] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576261  178160 container_manager_linux.go:267] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: KubeletOOMScoreAdj:-999 ContainerRuntime: CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerPolicyOptions:map[] ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576319  178160 topology_manager.go:133] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576328  178160 container_manager_linux.go:302] "Creating device plugin manager" devicePluginEnabled=true
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576359  178160 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.579169  178160 kubelet.go:376] "Attempting to sync node with API server"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.579187  178160 kubelet.go:267] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.579203  178160 kubelet.go:278] "Adding apiserver pod source"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.579211  178160 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: WARN 08:20:37.580088  178160 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Node: Get "https://YYY.YYY.YYY.YY:6443/api/v1/nodes?fieldSelector=metadata.name%3Djupyterhub-test&limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.580150  178160 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://YYY.YYY.YYY.YY:6443/api/v1/nodes?fieldSelector=metadata.name%3Djupyterhub-test&limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.580230  178160 kuberuntime_manager.go:239] "Container runtime initialized" containerRuntime="containerd" version="1.6.6" apiVersion="v1"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.581022  178160 server.go:1181] "Started kubelet"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.581694  178160 server.go:150] "Starting to listen" address="0.0.0.0" port=10250
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.582261  178160 cri_stats_provider.go:455] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.582351  178160 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.582738  178160 event.go:276] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"jupyterhub-test.1714ac53a07755b3", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ZZZ_DeprecatedClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"jupyterhub-test", UID:"jupyterhub-test", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"Starting", Message:"Starting kubelet.", Source:v1.EventSource{Component:"kubelet", Host:"jupyterhub-test"}, FirstTimestamp:time.Date(2022, time.September, 14, 8, 20, 37, 580993971, time.Local), LastTimestamp:time.Date(2022, time.September, 14, 8, 20, 37, 580993971, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Post "https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/default/events": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused'(may retry after sleeping)
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: WARN 08:20:37.582943  178160 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Service: Get "https://YYY.YYY.YYY.YY:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.583036  178160 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://YYY.YYY.YYY.YY:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.583351  178160 server.go:410] "Adding debug handlers to kubelet server"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.582359  178160 kubelet.go:1298] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.585879  178160 volume_manager.go:289] "Starting Kubelet Volume Manager"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.585928  178160 desired_state_of_world_populator.go:145] "Desired state populator starts to run"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.586167  178160 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: Get "https://YYY.YYY.YYY.YY:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/jupyterhub-test?timeout=10s": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: WARN 08:20:37.586759  178160 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.CSIDriver: Get "https://YYY.YYY.YYY.YY:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.586877  178160 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://YYY.YYY.YYY.YY:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.598425  178160 kubelet_network_linux.go:76] "Initialized protocol iptables rules." protocol=IPv4
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.614860  178160 kubelet_network_linux.go:76] "Initialized protocol iptables rules." protocol=IPv6
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.614881  178160 status_manager.go:161] "Starting to sync pod status with apiserver"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.614893  178160 kubelet.go:1974] "Starting kubelet main sync loop"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.615122  178160 kubelet.go:1998] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: WARN 08:20:37.616041  178160 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.RuntimeClass: Get "https://YYY.YYY.YYY.YY:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.616074  178160 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get "https://YYY.YYY.YYY.YY:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665155  178160 cpu_manager.go:213] "Starting CPU manager" policy="none"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665168  178160 cpu_manager.go:214] "Reconciling" reconcilePeriod="10s"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665179  178160 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665277  178160 state_mem.go:88] "Updated default CPUSet" cpuSet=""
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665286  178160 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665291  178160 policy_none.go:49] "None policy: Start"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.666175  178160 memory_manager.go:168] "Starting memorymanager" policy="None"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.666199  178160 state_mem.go:35] "Initializing new in-memory state store"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.666468  178160 state_mem.go:75] "Updated machine memory state"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.670363  178160 manager.go:610] "Failed to read data from checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.670567  178160 plugin_manager.go:114] "Starting Kubelet Plugin Manager"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.671373  178160 eviction_manager.go:254] "Eviction manager: failed to get summary stats" err="failed to get node info: node "jupyterhub-test" not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.686080  178160 kubelet.go:2419] "Error getting node" err="node "jupyterhub-test" not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.686550  178160 kubelet_node_status.go:70] "Attempting to register node" node="jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.686984  178160 kubelet_node_status.go:92] "Unable to register node with API server" err="Post "https://YYY.YYY.YYY.YY:6443/api/v1/nodes": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused" node="jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.716206  178160 topology_manager.go:200] "Topology Admit Handler"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.716954  178160 topology_manager.go:200] "Topology Admit Handler"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.717589  178160 topology_manager.go:200] "Topology Admit Handler"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.717932  178160 status_manager.go:664] "Failed to get status for pod" podUID=6caff56f64d1ee735407a5a7ba6a787a pod="kube-system/etcd-jupyterhub-test" err="Get "https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/kube-system/pods/etcd-jupyterhub-test": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718167  178160 topology_manager.go:200] "Topology Admit Handler"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718644  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="daf5debb6988d121c82972b4d0f6c1935c005063b7f42da0c10773efbf0525e1"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718776  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="435139d2c5996b2f24600f7cd4911fc77a92bf06e7361347971e70a68e0a54e5"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718849  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="3554bcaebcaf46d227f480181f505d48af9a6f9d389e93f1c6d3fab703db8df4"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718927  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="257b423a7d1730e88f277239d535923ffac18d5430b723aee160d50af5417553"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718995  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="cae22d913fbc0ce98a6e7b0d8b13cef758179416dea8f7a9dd875ab5172d8a4f"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719055  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="c86abd3d68a6a6f596481f8784db328cc2bf9951b0a514bff979e4243b63e9ce"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719113  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="0db53f26fd3fc078c02941576d2187b8525fe2da77a2e6dffff797bba8b12213"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719177  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="19d5c5e57cd43c00235306c69378fd6a9d224a337a79ab0d19417bb7ae8c91b4"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719239  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="84298f6839000667cb1d02bc812a84c94920e088d5649821fb6ebe3dabc13698"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719301  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="4d624b46c1721a68bf6b8b1a57d0aa23ee8d664108114337234826a1e053c991"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719370  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="03454711bc28b88c9a5b4eeb8ab8c962e758bb8ae34cbee798ecd5651ca37bc8"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719429  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="1a29e83515332ebd4cf4eabecdaa4c305e280de78a90d90210fd4d467a28bffd"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719490  178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="1b8edc5b07c4a5eab18f9e7edb73b089ea303a5f393b650deb550935456c5df8"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718813  178160 status_manager.go:664] "Failed to get status for pod" podUID=8cdd26801ec71603976b3d4c3c72beae pod="kube-system/kube-apiserver-jupyterhub-test" err="Get "https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-jupyterhub-test": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719777  178160 status_manager.go:664] "Failed to get status for pod" podUID=c9c46b502c40273f3cde89de382eb13b pod="kube-system/kube-controller-manager-jupyterhub-test" err="Get "https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-jupyterhub-test": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.720022  178160 status_manager.go:664] "Failed to get status for pod" podUID=a7527c69e782fb5d6404c82767da6341 pod="kube-system/kube-scheduler-jupyterhub-test" err="Get "https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-jupyterhub-test": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.786540  178160 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: Get "https://YYY.YYY.YYY.YY:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/jupyterhub-test?timeout=10s": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.786583  178160 kubelet.go:2419] "Error getting node" err="node "jupyterhub-test" not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.786993  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-k8s-certs") pod "kube-apiserver-jupyterhub-test" (UID: "8cdd26801ec71603976b3d4c3c72beae") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787171  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "usr-local-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-usr-local-share-ca-certificates") pod "kube-apiserver-jupyterhub-test" (UID: "8cdd26801ec71603976b3d4c3c72beae") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787275  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "usr-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-usr-share-ca-certificates") pod "kube-apiserver-jupyterhub-test" (UID: "8cdd26801ec71603976b3d4c3c72beae") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787372  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/6caff56f64d1ee735407a5a7ba6a787a-etcd-certs") pod "etcd-jupyterhub-test" (UID: "6caff56f64d1ee735407a5a7ba6a787a") " pod="kube-system/etcd-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787406  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-data" (UniqueName: "kubernetes.io/host-path/6caff56f64d1ee735407a5a7ba6a787a-etcd-data") pod "etcd-jupyterhub-test" (UID: "6caff56f64d1ee735407a5a7ba6a787a") " pod="kube-system/etcd-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787431  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-ca-certs") pod "kube-apiserver-jupyterhub-test" (UID: "8cdd26801ec71603976b3d4c3c72beae") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787452  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "etc-ca-certificates" (UniqueName: "kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-etc-ca-certificates") pod "kube-apiserver-jupyterhub-test" (UID: "8cdd26801ec71603976b3d4c3c72beae") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787479  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "etc-pki" (UniqueName: "kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-etc-pki") pod "kube-apiserver-jupyterhub-test" (UID: "8cdd26801ec71603976b3d4c3c72beae") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.887309  178160 kubelet.go:2419] "Error getting node" err="node "jupyterhub-test" not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.887665  178160 kubelet_node_status.go:70] "Attempting to register node" node="jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889786  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-ca-certs") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889824  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "etc-pki" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-etc-pki") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889845  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-k8s-certs") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889865  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "usr-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-usr-share-ca-certificates") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889885  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/a7527c69e782fb5d6404c82767da6341-kubeconfig") pod "kube-scheduler-jupyterhub-test" (UID: "a7527c69e782fb5d6404c82767da6341") " pod="kube-system/kube-scheduler-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889930  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "flexvolume-dir" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-flexvolume-dir") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889950  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-kubeconfig") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889970  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "usr-local-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-usr-local-share-ca-certificates") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.890011  178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume "etc-ca-certificates" (UniqueName: "kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-etc-ca-certificates") pod "kube-controller-manager-jupyterhub-test" (UID: "c9c46b502c40273f3cde89de382eb13b") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.890469  178160 kubelet_node_status.go:92] "Unable to register node with API server" err="Post "https://YYY.YYY.YYY.YY:6443/api/v1/nodes": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused" node="jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.988241  178160 kubelet.go:2419] "Error getting node" err="node "jupyterhub-test" not found"
Sep 14 08:20:38 jupyterhub-test kubelet[178160]: INFO 08:20:38.019896  178160 scope.go:110] "RemoveContainer" containerID="cc4b38a68d8e34264d83b0859a9b9c84b6d71b07f886ba76a9d18e6e2fa63d81"
Sep 14 08:20:38 jupyterhub-test kubelet[178160]: INFO 08:20:38.019999  178160 scope.go:110] "RemoveContainer" containerID="8ceae598793f115e91fdd87e77b087d5ce5ccede2c719ae78c3c7a3c477bf714"
Sep 14 08:20:38 jupyterhub-test kubelet[178160]: ERROR 08:20:38.088723  178160 kubelet.go:2419] "Error getting node" err="node "jupyterhub-test" not found"
Sep 14 08:20:38 jupyterhub-test kubelet[178160]: ERROR 08:20:38.188858  178160 kubelet.go:2419] "Error getting node" err="node "jupyterhub-test" not found"