Feb 06 09:08:31 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 3958 (containerd-shim) remains running after unit stopped.
Feb 06 09:08:31 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 4082 (containerd-shim) remains running after unit stopped.
Feb 06 09:08:31 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 4120 (containerd-shim) remains running after unit stopped.
Feb 06 09:08:31 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 4273 (containerd-shim) remains running after unit stopped.
Feb 06 09:08:31 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 4338 (containerd-shim) remains running after unit stopped.
Feb 06 09:08:31 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 4385 (containerd-shim) remains running after unit stopped.
Feb 06 09:08:31 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 4482 (containerd-shim) remains running after unit stopped.
Feb 06 09:08:31 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 4502 (containerd-shim) remains running after unit stopped.
Feb 06 09:08:31 k8s-us-ash-1-prod-1 systemd[1]: Stopped Lightweight Kubernetes.
░░ Subject: A stop job for unit k3s.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A stop job for unit k3s.service has finished.
░░ 
░░ The job identifier is 2060 and the job result is done.
Feb 06 09:08:31 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Consumed 27.530s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit k3s.service completed and consumed the indicated resources.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 1131 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 1393 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 1606 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 1698 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 1828 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 1847 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 2223 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 2259 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 2636 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 2641 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 2831 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 2880 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 2911 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 3139 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 3199 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 3745 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 3958 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 4082 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 4120 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 4273 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 4338 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 4385 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 4482 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 4502 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: Starting Lightweight Kubernetes...
░░ Subject: A start job for unit k3s.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit k3s.service has begun execution.
░░ 
░░ The job identifier is 2148.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 sh[14721]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 1131 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 1393 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 1606 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 1698 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 1828 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 1847 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 2223 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 2259 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 2636 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 2641 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 2831 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 2880 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 2911 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 3139 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 3199 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 3745 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 3958 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 4082 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 4120 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 4273 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 4338 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 4385 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 4482 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Found left-over process 4502 (containerd-shim) in control group while starting unit. Ignoring.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 06 09:11:16 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:16Z" level=info msg="Starting k3s v1.28.5+k3s1 (5b2d1271)"
Feb 06 09:11:16 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:16Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Feb 06 09:11:16 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:16Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Feb 06 09:11:16 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:16Z" level=info msg="Database tables and indexes are up to date"
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:17Z" level=info msg="Kine available at unix://kine.sock"
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:17Z" level=info msg="Reconciling bootstrap data between datastore and disk"
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:17Z" level=info msg="Running kube-apiserver --advertise-address=172.20.0.2 --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --encryption-provider-config-automatic-reload=true --etcd-servers=unix://kine.sock --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:17Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259"
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:17Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true"
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:17Z" level=info msg="Waiting for API server to become available"
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:17.826229   14725 feature_gate.go:241] Setting GA feature gate JobTrackingWithFinalizers=true. It will be removed in a future release.
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:17Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --controllers=*,-route --feature-gates=CloudDualStackNodeIPs=true --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false"
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:17.832012   14725 options.go:220] external host was not specified, using 172.20.0.2
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:17Z" level=info msg="Server node token is available at /var/lib/rancher/k3s/server/token"
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:17Z" level=info msg="To join server node to cluster: k3s server -s https://5.161.184.14:6443 -t ${SERVER_NODE_TOKEN}"
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:17Z" level=info msg="Agent node token is available at /var/lib/rancher/k3s/server/agent-token"
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:17Z" level=info msg="To join agent node to cluster: k3s agent -s https://5.161.184.14:6443 -t ${AGENT_NODE_TOKEN}"
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:17.923539   14725 server.go:156] Version: v1.28.5+k3s1
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:17.924564   14725 server.go:158] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:17Z" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
Feb 06 09:11:17 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:17Z" level=info msg="Run: k3s kubectl"
Feb 06 09:11:18 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:18.267183   14725 shared_informer.go:311] Waiting for caches to sync for node_authorizer
Feb 06 09:11:18 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:18.476170   14725 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
Feb 06 09:11:18 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:18.514953   14725 plugins.go:161] Loaded 13 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,ClusterTrustBundleAttest,CertificateSubjectRestriction,ValidatingAdmissionPolicy,ValidatingAdmissionWebhook,ResourceQuota.
Feb 06 09:11:18 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:18.547106   14725 instance.go:298] Using reconciler: lease
Feb 06 09:11:18 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:18.890473   14725 handler.go:275] Adding GroupVersion apiextensions.k8s.io v1 to ResourceManager
Feb 06 09:11:18 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:18.891363   14725 genericapiserver.go:744] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:21 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:21.481604   14725 trace.go:236] Trace[686191642]: "List(recursive=true) etcd3" audit-id:,key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (06-Feb-2024 09:11:18.764) (total time: 2714ms):
Feb 06 09:11:21 k8s-us-ash-1-prod-1 k3s[14725]: Trace[686191642]: [2.714950702s] [2.714950702s] END
Feb 06 09:11:21 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:21.708066   14725 trace.go:236] Trace[145827880]: "List(recursive=true) etcd3" audit-id:,key:/configmaps,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (06-Feb-2024 09:11:20.137) (total time: 1566ms):
Feb 06 09:11:21 k8s-us-ash-1-prod-1 k3s[14725]: Trace[145827880]: [1.566835444s] [1.566835444s] END
Feb 06 09:11:21 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:21Z" level=info msg="Handling backend connection request [k8s-us-ash-1-prod-2]"
Feb 06 09:11:23 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:23.062874   14725 trace.go:236] Trace[1923690303]: "List(recursive=true) etcd3" audit-id:,key:/pods,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (06-Feb-2024 09:11:21.758) (total time: 1301ms):
Feb 06 09:11:23 k8s-us-ash-1-prod-1 k3s[14725]: Trace[1923690303]: [1.301308848s] [1.301308848s] END
Feb 06 09:11:23 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:23.080909   14725 trace.go:236] Trace[1294174083]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (06-Feb-2024 09:11:20.001) (total time: 3078ms):
Feb 06 09:11:23 k8s-us-ash-1-prod-1 k3s[14725]: Trace[1294174083]: [3.078708437s] [3.078708437s] END
Feb 06 09:11:23 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:23Z" level=info msg="Password verified locally for node k8s-us-ash-1-prod-1"
Feb 06 09:11:23 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:23Z" level=info msg="certificate CN=k8s-us-ash-1-prod-1 signed by CN=k3s-server-ca@1707198628: notBefore=2024-02-06 05:50:28 +0000 UTC notAfter=2025-02-05 09:11:23 +0000 UTC"
Feb 06 09:11:23 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:23.752349   14725 handler.go:275] Adding GroupVersion  v1 to ResourceManager
Feb 06 09:11:23 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:23.761432   14725 instance.go:709] API group "internal.apiserver.k8s.io" is not enabled, skipping.
Feb 06 09:11:24 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:24.797165   14725 trace.go:236] Trace[1466924419]: "List(recursive=true) etcd3" audit-id:,key:/clusterroles,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (06-Feb-2024 09:11:24.188) (total time: 604ms):
Feb 06 09:11:24 k8s-us-ash-1-prod-1 k3s[14725]: Trace[1466924419]: [604.71308ms] [604.71308ms] END
Feb 06 09:11:25 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:25Z" level=info msg="certificate CN=system:node:k8s-us-ash-1-prod-1,O=system:nodes signed by CN=k3s-client-ca@1707198628: notBefore=2024-02-06 05:50:28 +0000 UTC notAfter=2025-02-05 09:11:25 +0000 UTC"
Feb 06 09:11:25 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:25.847248   14725 instance.go:709] API group "resource.k8s.io" is not enabled, skipping.
Feb 06 09:11:25 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:25.927216   14725 trace.go:236] Trace[1207781778]: "List(recursive=true) etcd3" audit-id:,key:/controllerrevisions,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (06-Feb-2024 09:11:25.370) (total time: 551ms):
Feb 06 09:11:25 k8s-us-ash-1-prod-1 k3s[14725]: Trace[1207781778]: [551.819862ms] [551.819862ms] END
Feb 06 09:11:25 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:25.930074   14725 trace.go:236] Trace[935046789]: "List(recursive=true) etcd3" audit-id:,key:/validatingwebhookconfigurations,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (06-Feb-2024 09:11:25.414) (total time: 515ms):
Feb 06 09:11:25 k8s-us-ash-1-prod-1 k3s[14725]: Trace[935046789]: [515.348625ms] [515.348625ms] END
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.105791   14725 trace.go:236] Trace[1405636587]: "List(recursive=true) etcd3" audit-id:,key:/mutatingwebhookconfigurations,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (06-Feb-2024 09:11:25.446) (total time: 656ms):
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: Trace[1405636587]: [656.117893ms] [656.117893ms] END
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.306851   14725 trace.go:236] Trace[1401932427]: "List(recursive=true) etcd3" audit-id:,key:/replicasets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (06-Feb-2024 09:11:25.312) (total time: 990ms):
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: Trace[1401932427]: [990.226283ms] [990.226283ms] END
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.665180   14725 handler.go:275] Adding GroupVersion authentication.k8s.io v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.666681   14725 genericapiserver.go:744] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.666696   14725 genericapiserver.go:744] Skipping API authentication.k8s.io/v1alpha1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.670629   14725 handler.go:275] Adding GroupVersion authorization.k8s.io v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.670640   14725 genericapiserver.go:744] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.689095   14725 handler.go:275] Adding GroupVersion autoscaling v2 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.693217   14725 handler.go:275] Adding GroupVersion autoscaling v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.693235   14725 genericapiserver.go:744] Skipping API autoscaling/v2beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.693240   14725 genericapiserver.go:744] Skipping API autoscaling/v2beta2 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.699875   14725 handler.go:275] Adding GroupVersion batch v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.699888   14725 genericapiserver.go:744] Skipping API batch/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.708020   14725 handler.go:275] Adding GroupVersion certificates.k8s.io v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.708192   14725 genericapiserver.go:744] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.708199   14725 genericapiserver.go:744] Skipping API certificates.k8s.io/v1alpha1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.712689   14725 handler.go:275] Adding GroupVersion coordination.k8s.io v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.712712   14725 genericapiserver.go:744] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.712760   14725 genericapiserver.go:744] Skipping API discovery.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.716317   14725 handler.go:275] Adding GroupVersion discovery.k8s.io v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.732623   14725 handler.go:275] Adding GroupVersion networking.k8s.io v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.732644   14725 genericapiserver.go:744] Skipping API networking.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.732653   14725 genericapiserver.go:744] Skipping API networking.k8s.io/v1alpha1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.734985   14725 handler.go:275] Adding GroupVersion node.k8s.io v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.735038   14725 genericapiserver.go:744] Skipping API node.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.735043   14725 genericapiserver.go:744] Skipping API node.k8s.io/v1alpha1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.745641   14725 handler.go:275] Adding GroupVersion policy v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.745656   14725 genericapiserver.go:744] Skipping API policy/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.754225   14725 handler.go:275] Adding GroupVersion rbac.authorization.k8s.io v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.754339   14725 genericapiserver.go:744] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.754508   14725 genericapiserver.go:744] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.758174   14725 handler.go:275] Adding GroupVersion scheduling.k8s.io v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.758333   14725 genericapiserver.go:744] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.758343   14725 genericapiserver.go:744] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.769127   14725 handler.go:275] Adding GroupVersion storage.k8s.io v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.769330   14725 genericapiserver.go:744] Skipping API storage.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.769485   14725 genericapiserver.go:744] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.780329   14725 handler.go:275] Adding GroupVersion flowcontrol.apiserver.k8s.io v1beta3 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.782596   14725 handler.go:275] Adding GroupVersion flowcontrol.apiserver.k8s.io v1beta2 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.782730   14725 genericapiserver.go:744] Skipping API flowcontrol.apiserver.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.782841   14725 genericapiserver.go:744] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.816088   14725 handler.go:275] Adding GroupVersion apps v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.816285   14725 genericapiserver.go:744] Skipping API apps/v1beta2 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.816298   14725 genericapiserver.go:744] Skipping API apps/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.820719   14725 handler.go:275] Adding GroupVersion admissionregistration.k8s.io v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.820906   14725 genericapiserver.go:744] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.821022   14725 genericapiserver.go:744] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:26.824982   14725 handler.go:275] Adding GroupVersion events.k8s.io v1 to ResourceManager
Feb 06 09:11:26 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:26.825091   14725 genericapiserver.go:744] Skipping API events.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:27 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:27.032985   14725 handler.go:275] Adding GroupVersion apiregistration.k8s.io v1 to ResourceManager
Feb 06 09:11:27 k8s-us-ash-1-prod-1 k3s[14725]: W0206 09:11:27.033766   14725 genericapiserver.go:744] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
Feb 06 09:11:27 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:27Z" level=info msg="Module overlay was already loaded"
Feb 06 09:11:27 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:27Z" level=info msg="Module nf_conntrack was already loaded"
Feb 06 09:11:27 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:27Z" level=info msg="Module br_netfilter was already loaded"
Feb 06 09:11:27 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:27Z" level=info msg="Module iptable_nat was already loaded"
Feb 06 09:11:27 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:27Z" level=info msg="Module iptable_filter was already loaded"
Feb 06 09:11:27 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:27Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Feb 06 09:11:27 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:27Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Feb 06 09:11:28 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:28Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: connection refused\""
Feb 06 09:11:29 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:29Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: connection refused\""
Feb 06 09:11:30 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:30Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: connection refused\""
Feb 06 09:11:31 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:31Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: connection refused\""
Feb 06 09:11:33 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:33Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
Feb 06 09:11:34 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:34Z" level=info msg="containerd is now running"
Feb 06 09:11:35 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:35Z" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
Feb 06 09:11:35 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:35Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --feature-gates=CloudDualStackNodeIPs=true --healthz-bind-address=127.0.0.1 --hostname-override=k8s-us-ash-1-prod-1 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-ip=172.20.0.2 --node-labels= --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
Feb 06 09:11:35 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:35.548302   14725 trace.go:236] Trace[508497059]: "List" accept:application/json, */*,audit-id:6b0f32f4-7a3d-4234-a7d3-f7a9c31f8969,client:172.20.0.3,protocol:HTTP/2.0,resource:nodes,scope:cluster,url:/api/v1/nodes,user-agent:k3s/v1.28.5+k3s1 (linux/amd64) kubernetes/5b2d127,verb:LIST (06-Feb-2024 09:11:34.952) (total time: 592ms):
Feb 06 09:11:35 k8s-us-ash-1-prod-1 k3s[14725]: Trace[508497059]: ---"Writing http response done" count:2 589ms (09:11:35.545)
Feb 06 09:11:35 k8s-us-ash-1-prod-1 k3s[14725]: Trace[508497059]: [592.752624ms] [592.752624ms] END
Feb 06 09:11:35 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:35Z" level=info msg="Handling backend connection request [k8s-us-ash-1-prod-1]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.387485   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.398864   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.405350   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.409079   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.414577   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.420950   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.425052   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.428082   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.434216   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.442173   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.446534   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.493076   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.498067   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.502253   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.504688   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.508116   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.513945   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.517206   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.522082   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.543813   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.546400   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.547633   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.548956   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.550534   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.551527   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.553292   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.556099   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.556461   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.557512   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.559555   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.561225   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/ingress-nginx/serviceaccounts/ingress-nginx\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.561550   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-proxy-injector\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.569249   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/ingress-nginx/serviceaccounts/ingress-nginx\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.572691   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/ingress-nginx/serviceaccounts/ingress-nginx\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.573087   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/ingress-nginx/serviceaccounts/ingress-nginx\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.576085   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/ingress-nginx/serviceaccounts/ingress-nginx\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.576917   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-system/serviceaccounts/metrics-server\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.582370   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-system/serviceaccounts/metrics-server\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.585359   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-system/serviceaccounts/metrics-server\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.588882   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.596283   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.747732   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/tap\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.763336   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/tap\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.782406   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-proxy-injector\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.813113   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/loki-grafana-agent\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.827971   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.600485   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-proxy-injector\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.865629   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-jaeger/serviceaccounts/jaeger-injector\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.888272   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/tap\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.893637   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-system/serviceaccounts/local-path-provisioner-service-account\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.896859   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/tap\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.901501   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/tap\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.905077   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/tap\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.909616   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/tap\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.913412   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/metrics-api\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.916227   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/metrics-api\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.918253   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/metrics-api\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.921145   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/metrics-api\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.923942   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/metrics-api\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.925291   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/metrics-api\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.928686   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/metrics-api\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.931885   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/metrics-api\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.933940   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/metrics-api\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.947747   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.950231   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.952835   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.956685   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.959338   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.961604   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.962033   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.964460   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.966307   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.966723   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.968941   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.969345   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.971325   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.972375   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.973950   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.974328   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.975577   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-prome-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.601229   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.977851   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.986008   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-system/serviceaccounts/local-path-provisioner-service-account\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.987530   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-system/serviceaccounts/local-path-provisioner-service-account\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.988278   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.993805   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:37 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.998788   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.966477   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:38.004300   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.620064   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:38.008245   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:38.012905   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:38.022252   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:38.027533   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.620259   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:38.031166   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.620929   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:38.037214   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.621485   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-proxy-injector\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:38Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.621703   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.622169   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-proxy-injector\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.622050   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.622933   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.623217   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-proxy-injector\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.623330   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.623357   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.623584   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.624034   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.624723   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.626210   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.627352   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.628096   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.628398   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-kube-state-metrics\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.743601   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/tap\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:37.746030   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/tap\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:38.003665   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd/serviceaccounts/linkerd-destination\": net/http: TLS handshake timeout]"
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:38.875221   14725 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:38.878105   14725 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:38.878339   14725 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:38.882773   14725 trace.go:236] Trace[49062091]: "Patch" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:4c361bc9-3760-469a-b631-955b8812ae5b,client:172.20.0.3,protocol:HTTP/2.0,resource:nodes,scope:resource,url:/api/v1/nodes/k8s-us-ash-1-prod-2/status,user-agent:k3s/v1.28.5+k3s1 (linux/amd64) kubernetes/5b2d127,verb:PATCH (06-Feb-2024 09:11:28.885) (total time: 9994ms):
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: Trace[49062091]: ---"About to check admission control" 143ms (09:11:29.035)
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: Trace[49062091]: [9.994345035s] [9.994345035s] END
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:38.897745   14725 timeout.go:142] post-timeout activity - time-elapsed: 12.418323ms, PATCH "/api/v1/nodes/k8s-us-ash-1-prod-2/status" result: <nil>
Feb 06 09:11:38 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:38.912719   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/tap-injector\": net/http: TLS handshake timeout]"
Feb 06 09:11:39 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:39.070686   14725 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 164.118574ms, panicked: false, err: context canceled, panic-reason: <nil>
Feb 06 09:11:39 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:39.125174   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/weave-gitops/serviceaccounts/weave-gitops\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.091012   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/loki-grafana-agent-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.099474   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/loki-grafana-agent-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.111594   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/loki-grafana-agent-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.118612   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/loki-grafana-agent-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.149139   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/loki-grafana-agent-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.150705   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-grafana\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.151448   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/loki-grafana-agent-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.153118   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/loki-grafana-agent-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.154378   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/loki-grafana-agent-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.155524   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/loki-grafana-agent-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.156767   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/loki-grafana-agent-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.160228   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/loki-grafana-agent-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.091016   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/loki-grafana-agent-operator\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.185333   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-grafana\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.192956   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-grafana\": net/http: TLS handshake timeout]"
Feb 06 09:11:40 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:40.197690   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/kube-prometheus/serviceaccounts/kube-prometheus-grafana\": net/http: TLS handshake timeout]"
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:42.350129   14725 trace.go:236] Trace[759379325]: "Create" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:5ab9eb7e-e8f6-47ea-b6f3-5a440ddd6275,client:172.20.0.3,protocol:HTTP/2.0,resource:tokenreviews,scope:resource,url:/apis/authentication.k8s.io/v1/tokenreviews,user-agent:k3s/v1.28.5+k3s1 (linux/amd64) kubernetes/5b2d127,verb:POST (06-Feb-2024 09:11:31.310) (total time: 11037ms):
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: Trace[759379325]: ---"Write to database call failed" len:1249,err:tokenreviews.authentication.k8s.io "Unknown" is forbidden: not yet ready to handle request 10612ms (09:11:41.938)
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: Trace[759379325]: [11.037454889s] [11.037454889s] END
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:42.480038   14725 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:42.481287   14725 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:42.495493   14725 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:42.497875   14725 trace.go:236] Trace[1028615439]: "Update" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:4ad66fdb-4ec9-4283-b7d7-dc8acd919697,client:172.20.0.3,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/k8s-us-ash-1-prod-2,user-agent:k3s/v1.28.5+k3s1 (linux/amd64) kubernetes/5b2d127,verb:PUT (06-Feb-2024 09:11:32.013) (total time: 10482ms):
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: Trace[1028615439]: ---"Write to database call failed" len:506,err:Timeout: request did not complete within requested timeout - context deadline exceeded 9983ms (09:11:42.004)
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: Trace[1028615439]: [10.48244952s] [10.48244952s] END
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:42.620075   14725 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 606.933864ms, panicked: false, err: context deadline exceeded, panic-reason: <nil>
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:42.654717   14725 trace.go:236] Trace[1339851581]: "Patch" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:55993062-572d-4960-8c97-2ba8c607b285,client:172.20.0.3,protocol:HTTP/2.0,resource:pods,scope:resource,url:/api/v1/namespaces/cert-manager/pods/trust-manager-775bfcf747-wzzhb/status,user-agent:k3s/v1.28.5+k3s1 (linux/amd64) kubernetes/5b2d127,verb:PATCH (06-Feb-2024 09:11:29.143) (total time: 13508ms):
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: Trace[1339851581]: ["GuaranteedUpdate etcd3" audit-id:55993062-572d-4960-8c97-2ba8c607b285,key:/pods/cert-manager/trust-manager-775bfcf747-wzzhb,type:*core.Pod,resource:pods 13467ms (09:11:29.184)]
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: Trace[1339851581]: ---"About to check admission control" 1298ms (09:11:30.524)
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: Trace[1339851581]: [13.508944798s] [13.508944798s] END
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:42.726277   14725 timeout.go:142] post-timeout activity - time-elapsed: 533.141109ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/k8s-us-ash-1-prod-2" result: <nil>
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:42.842891   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, context canceled]"
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:42.977470   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:42.984133   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:42.994913   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:42 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:42.998924   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:43 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:43.001618   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:43 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:43.002644   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/linkerd-viz/serviceaccounts/prometheus\": net/http: TLS handshake timeout]"
Feb 06 09:11:43 k8s-us-ash-1-prod-1 k3s[14725]: I0206 09:11:43.613710   14725 trace.go:236] Trace[200158035]: "Get" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:21edbdf7-2f5e-4f39-acbe-31a626237117,client:172.20.0.3,protocol:HTTP/2.0,resource:pods,scope:resource,url:/api/v1/namespaces/cert-manager/pods/cert-manager-9d89467db-mpw2v,user-agent:k3s/v1.28.5+k3s1 (linux/amd64) kubernetes/5b2d127,verb:GET (06-Feb-2024 09:11:42.907) (total time: 702ms):
Feb 06 09:11:43 k8s-us-ash-1-prod-1 k3s[14725]: Trace[200158035]: ---"About to write a response" 667ms (09:11:43.575)
Feb 06 09:11:43 k8s-us-ash-1-prod-1 k3s[14725]: Trace[200158035]: [702.43951ms] [702.43951ms] END
Feb 06 09:11:43 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:43.718127   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/promtail\": net/http: TLS handshake timeout]"
Feb 06 09:11:44 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:44Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error"
Feb 06 09:11:45 k8s-us-ash-1-prod-1 k3s[14725]: E0206 09:11:45.696702   14725 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Get \"https://127.0.0.1:6444/api/v1/namespaces/loki/serviceaccounts/promtail\": net/http: TLS handshake timeout]"
Feb 06 09:11:46 k8s-us-ash-1-prod-1 k3s[14725]: time="2024-02-06T09:11:46Z" level=fatal msg="Kubelet failed to wait for apiserver ready: timed out waiting for the condition, failed to get apiserver /readyz status: forbidden: User \"system:node:k8s-us-ash-1-prod-1\" cannot get path \"/readyz\""
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ An ExecStart= process belonging to unit k3s.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit k3s.service has entered the 'failed' state with result 'exit-code'.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 1131 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 1393 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 1606 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 1698 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 1828 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 1847 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 2223 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 2259 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 2636 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 2641 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 2831 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 2880 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 2911 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 3139 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 3199 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 3745 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 3958 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 4082 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 4120 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 4273 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 4338 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 4385 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 4482 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Unit process 4502 (containerd-shim) remains running after unit stopped.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: Stopped Lightweight Kubernetes.
░░ Subject: A stop job for unit k3s.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A stop job for unit k3s.service has finished.
░░ 
░░ The job identifier is 2231 and the job result is done.
Feb 06 09:11:46 k8s-us-ash-1-prod-1 systemd[1]: k3s.service: Consumed 14.794s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit k3s.service completed and consumed the indicated resources.