Sep 09 03:13:15 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:15Z" level=info msg="Starting k3s v1.30.14+k3s2 (071b1ead)" Sep 09 03:13:15 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:15Z" level=info msg="Configuring pgx database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s" Sep 09 03:13:15 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:15Z" level=info msg="Configuring database table schema and indexes, this may take a moment..." Sep 09 03:13:15 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:15Z" level=info msg="Database tables and indexes are up to date" Sep 09 03:13:15 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:15Z" level=info msg="Kine available at unix://kine.sock" Sep 09 03:13:15 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:15Z" level=info msg="Reconciling bootstrap data between datastore and disk" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=error msg="Sending HTTP/1.1 503 response to 10.6.61.172:57412: starting" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-app-blue-2-9c964185]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-ops-blue-1-89a427cc]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-ingress-blue-1-4323cca2]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-ingress-blue-0-ac9168ff]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-app-blue-4-79b5072b]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-ingress-blue-2-39068466]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-app-blue-9-63cdf3ff]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-app-blue-6-cd93959b]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-ops-blue-0-4b80a6a8]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-app-blue-8-49a23f58]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-app-blue-0-f0380b7a]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-app-blue-10-1ef6ca3c]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-app-blue-7-c3860444]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-app-blue-5-bb69767d]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-app-blue-1-a234b216]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-app-blue-3-aef87d06]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-data-green-3-47319b7a]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-data-green-1-efd5b590]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-data-green-5-f52e9ef0]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-data-green-4-96deb91b]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-data-green-6-420706e2]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-data-green-7-93605edd]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-agent-data-green-0-9b98de05]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Handling backend connection request [prod-server-blue-k3s-00]" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="Password verified locally for node prod-server-blue-k3s-01" Sep 09 03:13:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:16Z" level=info msg="certificate CN=prod-server-blue-k3s-01 signed by CN=k3s-server-ca@1681478841: notBefore=2023-04-14 13:27:21 +0000 UTC notAfter=2026-09-09 03:13:16 +0000 UTC" Sep 09 03:13:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:17Z" level=info msg="Handling backend connection request [prod-agent-app-blue-11-2dccff04]" Sep 09 03:13:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:17Z" level=info msg="certificate CN=system:node:prod-server-blue-k3s-01,O=system:nodes signed by CN=k3s-client-ca@1681478841: notBefore=2023-04-14 13:27:21 +0000 UTC notAfter=2026-09-09 03:13:17 +0000 UTC" Sep 09 03:13:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:17Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1681478841: notBefore=2023-04-14 13:27:21 +0000 UTC notAfter=2026-09-09 03:13:17 +0000 UTC" Sep 09 03:13:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:17Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1681478841: notBefore=2023-04-14 13:27:21 +0000 UTC notAfter=2026-09-09 03:13:17 +0000 UTC" Sep 09 03:13:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:17Z" level=info msg="Using private registry config file at /etc/rancher/k3s/registries.yaml" Sep 09 03:13:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:17Z" level=info msg="Module overlay was already loaded" Sep 09 03:13:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:17Z" level=info msg="Module nf_conntrack was already loaded" Sep 09 03:13:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:17Z" level=info msg="Module br_netfilter was already loaded" Sep 09 03:13:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:17Z" level=info msg="Module iptable_nat was already loaded" Sep 09 03:13:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:17Z" level=info msg="Module iptable_filter was already loaded" Sep 09 03:13:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:17Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log" Sep 09 03:13:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:17Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd" Sep 09 03:13:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:17Z" level=info msg="Polling for API server readiness: GET /readyz failed: the server is currently unable to handle the request" Sep 09 03:13:18 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:18Z" level=info msg="containerd is now running" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Creating k3s-cert-monitor event broadcaster" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=172.30.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=prod-server-blue-k3s-01 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-ip=10.6.61.142 --node-labels=node.kubernetes.io/controller=true,cluster-name=prod-blue --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --register-with-taints=node-role.kubernetes.io/control-plane=:NoSchedule --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Connection to etcd is ready" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="ETCD server is now running" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Running kube-apiserver --advertise-address=10.6.61.142 --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=sts.amazonaws.com --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --etcd-servers=unix://kine.sock --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://s3.ap-southeast-3.amazonaws.com/dc-k3s-aws-irsa --service-account-issuer=k3s --service-account-key-file=/etc/rancher/k3s/tls/sa-signer-pkcs8.pub --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/etc/rancher/k3s/tls/sa-signer.key --service-cluster-ip-range=172.30.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 --tls-cert-file=/var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.key" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.627688 983458 options.go:221] external host was not specified, using 10.6.61.142 Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.128.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect-lease-duration=60s --leader-elect-renew-deadline=30s --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=172.30.0.0/16 --tls-cert-file=/var/lib/rancher/k3s/server/tls/kube-controller-manager/kube-controller-manager.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/kube-controller-manager/kube-controller-manager.key --use-service-account-credentials=true" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.628949 983458 server.go:156] Version: v1.30.14+k3s2 Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.629004 983458 server.go:158] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=10.128.0.0/16 --configure-cloud-routes=false --controllers=*,-route,-service --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Handling backend connection request [prod-server-blue-k3s-01]" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Remotedialer connected to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Server node token is available at /var/lib/rancher/k3s/server/token" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="To join server node to cluster: k3s server -s https://10.6.61.142:6443 -t ${SERVER_NODE_TOKEN}" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Agent node token is available at /var/lib/rancher/k3s/server/agent-token" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="To join agent node to cluster: k3s agent -s https://10.6.61.142:6443 -t ${AGENT_NODE_TOKEN}" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Run: k3s kubectl" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.650849 983458 shared_informer.go:313] Waiting for caches to sync for node_authorizer Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.658740 983458 shared_informer.go:313] Waiting for caches to sync for *generic.policySource[*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicy,*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicyBinding,k8s.io/apiserver/pkg/admission/plugin/policy/validating.Validator] Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.662925 983458 plugins.go:157] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.663075 983458 plugins.go:160] Loaded 13 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,ClusterTrustBundleAttest,CertificateSubjectRestriction,ValidatingAdmissionPolicy,ValidatingAdmissionWebhook,ResourceQuota. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.663420 983458 instance.go:299] Using reconciler: lease Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.669951 983458 handler.go:286] Adding GroupVersion apiextensions.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.669981 983458 genericapiserver.go:733] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.707104 983458 handler.go:286] Adding GroupVersion v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.707297 983458 instance.go:696] API group "internal.apiserver.k8s.io" is not enabled, skipping. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.722868 983458 instance.go:696] API group "storagemigration.k8s.io" is not enabled, skipping. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.752106 983458 instance.go:696] API group "resource.k8s.io" is not enabled, skipping. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=error msg="Sending HTTP/1.1 503 response to 127.0.0.1:59972: runtime core not ready" Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.772590 983458 handler.go:286] Adding GroupVersion authentication.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.772613 983458 genericapiserver.go:733] Skipping API authentication.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.772621 983458 genericapiserver.go:733] Skipping API authentication.k8s.io/v1alpha1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.773218 983458 handler.go:286] Adding GroupVersion authorization.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.773235 983458 genericapiserver.go:733] Skipping API authorization.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.774040 983458 handler.go:286] Adding GroupVersion autoscaling v2 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.774793 983458 handler.go:286] Adding GroupVersion autoscaling v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.774810 983458 genericapiserver.go:733] Skipping API autoscaling/v2beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.774816 983458 genericapiserver.go:733] Skipping API autoscaling/v2beta2 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.776238 983458 handler.go:286] Adding GroupVersion batch v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.776257 983458 genericapiserver.go:733] Skipping API batch/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.777241 983458 handler.go:286] Adding GroupVersion certificates.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.777263 983458 genericapiserver.go:733] Skipping API certificates.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.777269 983458 genericapiserver.go:733] Skipping API certificates.k8s.io/v1alpha1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.777897 983458 handler.go:286] Adding GroupVersion coordination.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.777918 983458 genericapiserver.go:733] Skipping API coordination.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.777996 983458 genericapiserver.go:733] Skipping API discovery.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.778553 983458 handler.go:286] Adding GroupVersion discovery.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.780157 983458 handler.go:286] Adding GroupVersion networking.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.780177 983458 genericapiserver.go:733] Skipping API networking.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.780183 983458 genericapiserver.go:733] Skipping API networking.k8s.io/v1alpha1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.780676 983458 handler.go:286] Adding GroupVersion node.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.780694 983458 genericapiserver.go:733] Skipping API node.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.780700 983458 genericapiserver.go:733] Skipping API node.k8s.io/v1alpha1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.781585 983458 handler.go:286] Adding GroupVersion policy v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.781605 983458 genericapiserver.go:733] Skipping API policy/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.783411 983458 handler.go:286] Adding GroupVersion rbac.authorization.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.783427 983458 genericapiserver.go:733] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.783435 983458 genericapiserver.go:733] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.784009 983458 handler.go:286] Adding GroupVersion scheduling.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.784031 983458 genericapiserver.go:733] Skipping API scheduling.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.784037 983458 genericapiserver.go:733] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.786225 983458 handler.go:286] Adding GroupVersion storage.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.786274 983458 genericapiserver.go:733] Skipping API storage.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.786280 983458 genericapiserver.go:733] Skipping API storage.k8s.io/v1alpha1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.787495 983458 handler.go:286] Adding GroupVersion flowcontrol.apiserver.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.788701 983458 handler.go:286] Adding GroupVersion flowcontrol.apiserver.k8s.io v1beta3 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.788722 983458 genericapiserver.go:733] Skipping API flowcontrol.apiserver.k8s.io/v1beta2 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.788728 983458 genericapiserver.go:733] Skipping API flowcontrol.apiserver.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.792503 983458 handler.go:286] Adding GroupVersion apps v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.792529 983458 genericapiserver.go:733] Skipping API apps/v1beta2 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.792538 983458 genericapiserver.go:733] Skipping API apps/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.794819 983458 handler.go:286] Adding GroupVersion admissionregistration.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.794839 983458 genericapiserver.go:733] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.794844 983458 genericapiserver.go:733] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.795509 983458 handler.go:286] Adding GroupVersion events.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.795528 983458 genericapiserver.go:733] Skipping API events.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:20.798547 983458 handler.go:286] Adding GroupVersion apiregistration.k8s.io v1 to ResourceManager Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:20.798564 983458 genericapiserver.go:733] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources. Sep 09 03:13:20 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:20Z" level=info msg="Running kube-proxy --cluster-cidr=10.128.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=prod-server-blue-k3s-01 --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables" Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.151974 983458 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt" Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.151971 983458 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt" Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.152496 983458 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.153003 983458 secure_serving.go:213] Serving securely on 127.0.0.1:6444 Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.153089 983458 tlsconfig.go:240] "Starting DynamicServingCertificateController" Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.153485 983458 dynamic_serving_content.go:132] "Starting controller" name="aggregator-proxy-cert::/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt::/var/lib/rancher/k3s/server/tls/client-auth-proxy.key" Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.153742 983458 apf_controller.go:374] Starting API Priority and Fairness config controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.153874 983458 aggregator.go:163] waiting for initial CRD sync... Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.153894 983458 controller.go:78] Starting OpenAPI AggregationController Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.153961 983458 system_namespaces_controller.go:67] Starting system namespaces controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.153993 983458 controller.go:116] Starting legacy_token_tracking_controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154030 983458 customresource_discovery_controller.go:289] Starting DiscoveryController Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154050 983458 crdregistration_controller.go:111] Starting crd-autoregister controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154057 983458 shared_informer.go:313] Waiting for caches to sync for crd-autoregister Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154053 983458 apiservice_controller.go:97] Starting APIServiceRegistrationController Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154080 983458 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154085 983458 controller.go:139] Starting OpenAPI controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154113 983458 controller.go:87] Starting OpenAPI V3 controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154133 983458 naming_controller.go:291] Starting NamingConditionController Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154141 983458 gc_controller.go:78] Starting apiserver lease garbage collector Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154148 983458 establishing_controller.go:76] Starting EstablishingController Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154163 983458 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154177 983458 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154191 983458 crd_finalizer.go:266] Starting CRDFinalizer Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154012 983458 shared_informer.go:313] Waiting for caches to sync for configmaps Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.153786 983458 available_controller.go:423] Starting AvailableConditionController Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154216 983458 cache.go:32] Waiting for caches to sync for AvailableConditionController controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154535 983458 controller.go:80] Starting OpenAPI V3 AggregationController Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154564 983458 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt" Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.154632 983458 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt" Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.153762 983458 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.156128 983458 shared_informer.go:313] Waiting for caches to sync for cluster_authentication_trust_controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.254145 983458 cache.go:39] Caches are synced for APIServiceRegistrationController controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.254243 983458 cache.go:39] Caches are synced for AvailableConditionController controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.254362 983458 handler.go:286] Adding GroupVersion cilium.io v2alpha1 to ResourceManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.254427 983458 handler.go:286] Adding GroupVersion generators.external-secrets.io v1alpha1 to ResourceManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.254447 983458 handler.go:286] Adding GroupVersion encryptor.dev v1beta1 to ResourceManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.254519 983458 handler.go:286] Adding GroupVersion storage.hpe.com v1 to ResourceManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.254558 983458 handler.go:286] Adding GroupVersion cert-manager.io v1 to ResourceManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.254614 983458 handler.go:286] Adding GroupVersion cilium.io v2 to ResourceManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.254692 983458 handler.go:286] Adding GroupVersion external-secrets.io v1alpha1 to ResourceManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.254721 983458 handler.go:286] Adding GroupVersion external-secrets.io v1beta1 to ResourceManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.255387 983458 handler.go:286] Adding GroupVersion acme.cert-manager.io v1 to ResourceManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.255421 983458 shared_informer.go:320] Caches are synced for crd-autoregister Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.255546 983458 handler.go:286] Adding GroupVersion kubernetes-client.io v1 to ResourceManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.255706 983458 handler.go:286] Adding GroupVersion storage.hpe.com v2 to ResourceManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.255901 983458 handler.go:286] Adding GroupVersion k3s.cattle.io v1 to ResourceManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.255920 983458 handler.go:286] Adding GroupVersion helm.cattle.io v1 to ResourceManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.256072 983458 apf_controller.go:379] Running API Priority and Fairness config worker Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.256087 983458 apf_controller.go:382] Running API Priority and Fairness periodic rebalancing process Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.256279 983458 shared_informer.go:320] Caches are synced for cluster_authentication_trust_controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.256301 983458 aggregator.go:165] initial CRD sync complete... Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.256308 983458 autoregister_controller.go:141] Starting autoregister controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.256352 983458 cache.go:32] Waiting for caches to sync for autoregister controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.256359 983458 cache.go:39] Caches are synced for autoregister controller Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.258208 983458 shared_informer.go:320] Caches are synced for configmaps Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.269595 983458 handler_discovery.go:447] Starting ResourceDiscoveryManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.297507 983458 handler.go:286] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:21.356872 983458 lease.go:265] Resetting endpoints for master service "kubernetes" to [10.6.61.141] Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.358802 983458 shared_informer.go:320] Caches are synced for *generic.policySource[*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicy,*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicyBinding,k8s.io/apiserver/pkg/admission/plugin/policy/validating.Validator] Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.358828 983458 policy_source.go:224] refreshing policies Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.359334 983458 controller.go:615] quota admission added evaluator for: endpoints Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.376392 983458 controller.go:615] quota admission added evaluator for: endpointslices.discovery.k8s.io Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: E0909 03:13:21.385382 983458 controller.go:100] Found stale data, removed previous endpoints on kubernetes service, apiserver didn't exit successfully previously Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.393021 983458 controller.go:615] quota admission added evaluator for: leases.coordination.k8s.io Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:21.551104 983458 shared_informer.go:320] Caches are synced for node_authorizer Sep 09 03:13:21 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:21Z" level=info msg="Polling for API server readiness: GET /readyz failed: an error on the server (\"[+]ping ok\\n[+]log ok\\n[+]etcd ok\\n[+]etcd-readiness ok\\n[-]informer-sync failed: reason withheld\\n[+]poststarthook/start-apiserver-admission-initializer ok\\n[+]poststarthook/generic-apiserver-start-informers ok\\n[+]poststarthook/priority-and-fairness-config-consumer ok\\n[+]poststarthook/priority-and-fairness-filter ok\\n[+]poststarthook/storage-object-count-tracker-hook ok\\n[+]poststarthook/start-apiextensions-informers ok\\n[+]poststarthook/start-apiextensions-controllers ok\\n[+]poststarthook/crd-informer-synced ok\\n[+]poststarthook/start-service-ip-repair-controllers ok\\n[-]poststarthook/rbac/bootstrap-roles failed: reason withheld\\n[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: reason withheld\\n[+]poststarthook/priority-and-fairness-config-producer ok\\n[+]poststarthook/start-system-namespaces-controller ok\\n[+]poststarthook/bootstrap-controller ok\\n[+]poststarthook/start-cluster-authentication-info-controller ok\\n[+]poststarthook/start-kube-apiserver-identity-lease-controller ok\\n[+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok\\n[+]poststarthook/start-legacy-token-tracking-controller ok\\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\\n[+]poststarthook/start-kube-aggregator-informers ok\\n[+]poststarthook/apiservice-registration-controller ok\\n[+]poststarthook/apiservice-status-available-controller ok\\n[+]poststarthook/apiservice-discovery-controller ok\\n[+]poststarthook/kube-apiserver-autoregistration ok\\n[+]autoregister-completion ok\\n[+]poststarthook/apiservice-openapi-controller ok\\n[+]poststarthook/apiservice-openapiv3-controller ok\\n[+]shutdown ok\\nreadyz check failed\") has prevented the request from succeeding" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:22.203174 983458 storage_scheduling.go:111] all system priority classes are created successfully or already exist. Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:22Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:22 prod-server-blue-k3s-01 k3s[983458]: W0909 03:13:22.822113 983458 lease.go:265] Resetting endpoints for master service "kubernetes" to [10.6.61.141 10.6.61.142] Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Kube API server is now running" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="k3s is up and running" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Waiting for untainted node" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Waiting for cloud-controller-manager privileges to become available" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed. Sep 09 03:13:23 prod-server-blue-k3s-01 systemd[1]: Started Lightweight Kubernetes. Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.623151 983458 server.go:484] "Kubelet version" kubeletVersion="v1.30.14+k3s2" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.623178 983458 server.go:486] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Creating k3s-supervisor event broadcaster" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.624868 983458 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Annotations and labels have already set on node: prod-server-blue-k3s-01" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.636094 983458 server.go:1062] "Successfully retrieved node IP(s)" IPs=["10.6.61.142"] Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.640758 983458 server.go:742] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.640808 983458 server.go:810] "NoSwap is set due to memorySwapBehavior not specified" memorySwapBehavior="" FailSwapOn=false Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.640987 983458 container_manager_linux.go:265] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.641006 983458 container_manager_linux.go:270] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"prod-server-blue-k3s-01","RuntimeCgroupsName":"","SystemCgroupsName":"","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":false,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":null,"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"ExperimentalMemoryManagerPolicy":"None","ExperimentalMemoryManagerReservedMemory":null,"PodPidsLimit":-1,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null} Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.641345 983458 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.641356 983458 container_manager_linux.go:301] "Creating device plugin manager" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.641405 983458 state_mem.go:36] "Initialized new in-memory state store" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.641484 983458 kubelet.go:401] "Attempting to sync node with API server" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.641499 983458 kubelet.go:302] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.641525 983458 kubelet.go:313] "Adding apiserver pod source" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.641548 983458 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.643125 983458 kuberuntime_manager.go:261] "Container runtime initialized" containerRuntime="containerd" version="v1.7.27-k3s1" apiVersion="v1" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.643239 983458 kubelet.go:816] "Not starting ClusterTrustBundle informer because we are in static kubelet mode" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.643698 983458 server.go:1264] "Started kubelet" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.646355 983458 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.648012 983458 server.go:659] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.648064 983458 server_linux.go:165] "Using iptables Proxier" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.651061 983458 server_linux.go:511] "Detect-local-mode set to ClusterCIDR, but no cluster CIDR for family" ipFamily="IPv6" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.651085 983458 server_linux.go:528] "Defaulting to no-op detect-local" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.651168 983458 proxier.go:243] "Setting route_localnet=1 to allow node-ports on localhost; to change this either disable iptables.localhostNodePorts (--iptables-localhost-nodeports) or set nodePortAddresses (--nodeport-addresses) to filter loopback addresses" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.651429 983458 server.go:872] "Version info" version="v1.30.14+k3s2" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.651447 983458 server.go:874] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.652082 983458 server.go:163] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.652210 983458 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.652504 983458 server.go:227] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.657059 983458 server.go:444] "Adding debug handlers to kubelet server" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.658183 983458 config.go:192] "Starting service config controller" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.658301 983458 shared_informer.go:313] Waiting for caches to sync for service config Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.658403 983458 config.go:101] "Starting endpoint slice config controller" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.658485 983458 shared_informer.go:313] Waiting for caches to sync for endpoint slice config Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.658975 983458 config.go:319] "Starting node config controller" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.659093 983458 shared_informer.go:313] Waiting for caches to sync for node config Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.663029 983458 volume_manager.go:291] "Starting Kubelet Volume Manager" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.664869 983458 factory.go:221] Registration of the systemd container factory successfully Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.665113 983458 factory.go:219] Registration of the crio container factory failed: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.665600 983458 desired_state_of_world_populator.go:150] "Desired state populator starts to run" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.666610 983458 reconciler.go:26] "Reconciler: start to sync state" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: E0909 03:13:23.668800 983458 kubelet.go:1468] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.669091 983458 kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv4" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.670104 983458 kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv6" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.670129 983458 status_manager.go:217] "Starting to sync pod status with apiserver" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.670151 983458 kubelet.go:2356] "Starting kubelet main sync loop" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: E0909 03:13:23.670197 983458 kubelet.go:2380] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.672599 983458 factory.go:221] Registration of the containerd container factory successfully Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.758657 983458 shared_informer.go:320] Caches are synced for endpoint slice config Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.758706 983458 shared_informer.go:320] Caches are synced for service config Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.759276 983458 shared_informer.go:320] Caches are synced for node config Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.765106 983458 kubelet_node_status.go:73] "Attempting to register node" node="prod-server-blue-k3s-01" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.768226 983458 cpu_manager.go:214] "Starting CPU manager" policy="none" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.768399 983458 cpu_manager.go:215] "Reconciling" reconcilePeriod="10s" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.768517 983458 state_mem.go:36] "Initialized new in-memory state store" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.768823 983458 state_mem.go:88] "Updated default CPUSet" cpuSet="" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.768999 983458 state_mem.go:96] "Updated CPUSet assignments" assignments={} Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.769134 983458 policy_none.go:49] "None policy: Start" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.770180 983458 memory_manager.go:170] "Starting memorymanager" policy="None" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.770269 983458 state_mem.go:35] "Initializing new in-memory state store" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.770623 983458 state_mem.go:75] "Updated machine memory state" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: E0909 03:13:23.770820 983458 kubelet.go:2380] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.776793 983458 manager.go:479] "Failed to read data from checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.776925 983458 container_log_manager.go:186] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.777040 983458 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=warning msg="Failed to get apiserver addresses from supervisor: " Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.782008 983458 kubelet_node_status.go:112] "Node was previously registered" node="prod-server-blue-k3s-01" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.782108 983458 kubelet_node_status.go:76] "Successfully registered node" node="prod-server-blue-k3s-01" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.783379 983458 kuberuntime_manager.go:1525] "Updating runtime config through cri with podcidr" CIDR="10.128.38.0/24" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:23.783787 983458 kubelet_network.go:61] "Updating Pod CIDR" originalPodCIDR="" newPodCIDR="10.128.38.0/24" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Updating embedded CRD addons.k3s.cattle.io" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-ops-blue-0-4b80a6a8]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-app-blue-6-cd93959b]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-app-blue-11-2dccff04]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-ingress-blue-1-4323cca2]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-app-blue-2-9c964185]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-app-blue-0-f0380b7a]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-ingress-blue-2-39068466]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-ops-blue-1-89a427cc]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-app-blue-5-bb69767d]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-app-blue-8-49a23f58]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-app-blue-4-79b5072b]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-app-blue-10-1ef6ca3c]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-app-blue-9-63cdf3ff]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-app-blue-7-c3860444]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-ingress-blue-0-ac9168ff]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-app-blue-1-a234b216]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Updating embedded CRD etcdsnapshotfiles.k3s.cattle.io" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Handling backend connection request [prod-agent-app-blue-3-aef87d06]" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Updating embedded CRD helmchartconfigs.helm.cattle.io" Sep 09 03:13:23 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:23Z" level=info msg="Updating embedded CRD helmcharts.helm.cattle.io" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-27.0.201+up27.0.2.tgz" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-crd-27.0.201+up27.0.2.tgz" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/ccm.yaml" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/local-storage.yaml" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/runtimes.yaml" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Tunnel server egress proxy mode: agent" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Starting dynamiclistener CN filter node controller with SANs: [kube-api-lb-prod.dc.id.vida.vpc 10.6.61.133 127.0.0.1 ::1 localhost prod-server-blue-k3s-01 10.6.61.142 10.6.61.142 172.30.0.1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local]" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.190044 983458 controllermanager.go:189] "Starting" version="v1.30.14+k3s2" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.190077 983458 controllermanager.go:191] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.193062 983458 server.go:154] "Starting Kubernetes Scheduler" version="v1.30.14+k3s2" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.193090 983458 server.go:156] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.196566 983458 secure_serving.go:213] Serving securely on 127.0.0.1:10257 Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.196705 983458 tlsconfig.go:240] "Starting DynamicServingCertificateController" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.196735 983458 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/var/lib/rancher/k3s/server/tls/kube-controller-manager/kube-controller-manager.crt::/var/lib/rancher/k3s/server/tls/kube-controller-manager/kube-controller-manager.key" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.196740 983458 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.196756 983458 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.196777 983458 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.196788 983458 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.196800 983458 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.196808 983458 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.196954 983458 leaderelection.go:250] attempting to acquire leader lease kube-system/kube-controller-manager... Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.198099 983458 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.198137 983458 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.198211 983458 secure_serving.go:213] Serving securely on 127.0.0.1:10259 Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.198204 983458 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.198231 983458 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.198245 983458 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.198250 983458 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.198277 983458 tlsconfig.go:240] "Starting DynamicServingCertificateController" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.198218 983458 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt::/var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.key" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Creating deploy event broadcaster" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.266669 983458 event.go:389] "Event occurred" object="kube-system/ccm" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/ccm.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.297046 983458 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.297065 983458 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.297085 983458 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.298227 983458 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.298336 983458 leaderelection.go:250] attempting to acquire leader lease kube-system/kube-scheduler... Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.298370 983458 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.298392 983458 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.338376 983458 controller.go:615] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.341610 983458 event.go:389] "Event occurred" object="kube-system/ccm" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/ccm.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.347362 983458 controller.go:615] quota admission added evaluator for: addons.k3s.cattle.io Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.361943 983458 event.go:389] "Event occurred" object="kube-system/coredns" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/coredns.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.372037 983458 leaderelection.go:250] attempting to acquire leader lease kube-system/k3s... Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Starting /v1, Kind=Node controller" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Cluster dns configmap already exists" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.401210 983458 leaderelection.go:260] successfully acquired lease kube-system/k3s Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Creating helm-controller event broadcaster" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Labels and annotations have been set successfully on node: prod-server-blue-k3s-01" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.421467 983458 serving.go:387] Generated self-signed cert in-memory Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.617790 983458 controller.go:615] quota admission added evaluator for: deployments.apps Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:24Z" level=info msg="Tunnel authorizer set Kubelet Port 0.0.0.0:10250" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.642561 983458 apiserver.go:52] "Watching apiserver" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.643534 983458 event.go:389] "Event occurred" object="kube-system/coredns" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/coredns.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.646278 983458 topology_manager.go:215] "Topology Admit Handler" podUID="fb00cac0-1e30-4e56-b0fe-18af4f27b70a" podNamespace="ops" podName="prometheus-node-exporter-65q7p" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.647381 983458 topology_manager.go:215] "Topology Admit Handler" podUID="138d70ce-552c-4855-b317-6be6f8883751" podNamespace="ops" podName="wazuh-agent-wrmqb" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.647731 983458 topology_manager.go:215] "Topology Admit Handler" podUID="dc720361-519e-4f58-8d06-0aa872643004" podNamespace="ops" podName="datadog-xvhf6" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.648056 983458 topology_manager.go:215] "Topology Admit Handler" podUID="2ee20d47-0618-452a-a258-5e3909a79874" podNamespace="kube-system" podName="cilium-kndsv" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.649136 983458 topology_manager.go:215] "Topology Admit Handler" podUID="aea38dfa-368e-4100-b652-af7a7232a6ed" podNamespace="kube-system" podName="coredns-657d7c6574-d8zhf" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.649429 983458 topology_manager.go:215] "Topology Admit Handler" podUID="c7ac1e0a-6dc0-4475-a600-cee0b38c3b28" podNamespace="kube-system" podName="metrics-server-7cf87b4dbd-qbqz5" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.649545 983458 topology_manager.go:215] "Topology Admit Handler" podUID="4a092ece-bd27-4684-a21a-49081a2868ac" podNamespace="kube-system" podName="cilium-operator-6b658bf6bc-qlpsk" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.657984 983458 event.go:389] "Event occurred" object="kube-system/local-storage" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/local-storage.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.666824 983458 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.696595 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fb00cac0-1e30-4e56-b0fe-18af4f27b70a-root\") pod \"prometheus-node-exporter-65q7p\" (UID: \"fb00cac0-1e30-4e56-b0fe-18af4f27b70a\") " pod="ops/prometheus-node-exporter-65q7p" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.696627 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dev\" (UniqueName: \"kubernetes.io/host-path/138d70ce-552c-4855-b317-6be6f8883751-dev\") pod \"wazuh-agent-wrmqb\" (UID: \"138d70ce-552c-4855-b317-6be6f8883751\") " pod="ops/wazuh-agent-wrmqb" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.696645 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc\" (UniqueName: \"kubernetes.io/host-path/138d70ce-552c-4855-b317-6be6f8883751-etc\") pod \"wazuh-agent-wrmqb\" (UID: \"138d70ce-552c-4855-b317-6be6f8883751\") " pod="ops/wazuh-agent-wrmqb" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.696741 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"passwd\" (UniqueName: \"kubernetes.io/host-path/dc720361-519e-4f58-8d06-0aa872643004-passwd\") pod \"datadog-xvhf6\" (UID: \"dc720361-519e-4f58-8d06-0aa872643004\") " pod="ops/datadog-xvhf6" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.696782 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb00cac0-1e30-4e56-b0fe-18af4f27b70a-sys\") pod \"prometheus-node-exporter-65q7p\" (UID: \"fb00cac0-1e30-4e56-b0fe-18af4f27b70a\") " pod="ops/prometheus-node-exporter-65q7p" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.696800 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr\" (UniqueName: \"kubernetes.io/host-path/138d70ce-552c-4855-b317-6be6f8883751-usr\") pod \"wazuh-agent-wrmqb\" (UID: \"138d70ce-552c-4855-b317-6be6f8883751\") " pod="ops/wazuh-agent-wrmqb" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.696839 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logscontainerspath\" (UniqueName: \"kubernetes.io/host-path/dc720361-519e-4f58-8d06-0aa872643004-logscontainerspath\") pod \"datadog-xvhf6\" (UID: \"dc720361-519e-4f58-8d06-0aa872643004\") " pod="ops/datadog-xvhf6" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.696867 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ee20d47-0618-452a-a258-5e3909a79874-etc-cni-netd\") pod \"cilium-kndsv\" (UID: \"2ee20d47-0618-452a-a258-5e3909a79874\") " pod="kube-system/cilium-kndsv" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.696894 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/138d70ce-552c-4855-b317-6be6f8883751-proc\") pod \"wazuh-agent-wrmqb\" (UID: \"138d70ce-552c-4855-b317-6be6f8883751\") " pod="ops/wazuh-agent-wrmqb" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.696912 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"modules\" (UniqueName: \"kubernetes.io/host-path/138d70ce-552c-4855-b317-6be6f8883751-modules\") pod \"wazuh-agent-wrmqb\" (UID: \"138d70ce-552c-4855-b317-6be6f8883751\") " pod="ops/wazuh-agent-wrmqb" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.696927 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"procdir\" (UniqueName: \"kubernetes.io/host-path/dc720361-519e-4f58-8d06-0aa872643004-procdir\") pod \"datadog-xvhf6\" (UID: \"dc720361-519e-4f58-8d06-0aa872643004\") " pod="ops/datadog-xvhf6" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.696942 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cgroups\" (UniqueName: \"kubernetes.io/host-path/dc720361-519e-4f58-8d06-0aa872643004-cgroups\") pod \"datadog-xvhf6\" (UID: \"dc720361-519e-4f58-8d06-0aa872643004\") " pod="ops/datadog-xvhf6" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.696956 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bpf-maps\" (UniqueName: \"kubernetes.io/host-path/2ee20d47-0618-452a-a258-5e3909a79874-bpf-maps\") pod \"cilium-kndsv\" (UID: \"2ee20d47-0618-452a-a258-5e3909a79874\") " pod="kube-system/cilium-kndsv" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.696974 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-path\" (UniqueName: \"kubernetes.io/host-path/2ee20d47-0618-452a-a258-5e3909a79874-cni-path\") pod \"cilium-kndsv\" (UID: \"2ee20d47-0618-452a-a258-5e3909a79874\") " pod="kube-system/cilium-kndsv" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.697047 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fb00cac0-1e30-4e56-b0fe-18af4f27b70a-proc\") pod \"prometheus-node-exporter-65q7p\" (UID: \"fb00cac0-1e30-4e56-b0fe-18af4f27b70a\") " pod="ops/prometheus-node-exporter-65q7p" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.697076 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pointerdir\" (UniqueName: \"kubernetes.io/host-path/dc720361-519e-4f58-8d06-0aa872643004-pointerdir\") pod \"datadog-xvhf6\" (UID: \"dc720361-519e-4f58-8d06-0aa872643004\") " pod="ops/datadog-xvhf6" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.697091 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logdockercontainerpath\" (UniqueName: \"kubernetes.io/host-path/dc720361-519e-4f58-8d06-0aa872643004-logdockercontainerpath\") pod \"datadog-xvhf6\" (UID: \"dc720361-519e-4f58-8d06-0aa872643004\") " pod="ops/datadog-xvhf6" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.697106 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostproc\" (UniqueName: \"kubernetes.io/host-path/2ee20d47-0618-452a-a258-5e3909a79874-hostproc\") pod \"cilium-kndsv\" (UID: \"2ee20d47-0618-452a-a258-5e3909a79874\") " pod="kube-system/cilium-kndsv" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.697127 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-proc-sys-net\" (UniqueName: \"kubernetes.io/host-path/2ee20d47-0618-452a-a258-5e3909a79874-host-proc-sys-net\") pod \"cilium-kndsv\" (UID: \"2ee20d47-0618-452a-a258-5e3909a79874\") " pod="kube-system/cilium-kndsv" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.697157 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/138d70ce-552c-4855-b317-6be6f8883751-var-run\") pod \"wazuh-agent-wrmqb\" (UID: \"138d70ce-552c-4855-b317-6be6f8883751\") " pod="ops/wazuh-agent-wrmqb" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.697273 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cilium-run\" (UniqueName: \"kubernetes.io/host-path/2ee20d47-0618-452a-a258-5e3909a79874-cilium-run\") pod \"cilium-kndsv\" (UID: \"2ee20d47-0618-452a-a258-5e3909a79874\") " pod="kube-system/cilium-kndsv" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.697362 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-proc-sys-kernel\" (UniqueName: \"kubernetes.io/host-path/2ee20d47-0618-452a-a258-5e3909a79874-host-proc-sys-kernel\") pod \"cilium-kndsv\" (UID: \"2ee20d47-0618-452a-a258-5e3909a79874\") " pod="kube-system/cilium-kndsv" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.698810 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dsdsocket\" (UniqueName: \"kubernetes.io/host-path/dc720361-519e-4f58-8d06-0aa872643004-dsdsocket\") pod \"datadog-xvhf6\" (UID: \"dc720361-519e-4f58-8d06-0aa872643004\") " pod="ops/datadog-xvhf6" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.698994 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logpodpath\" (UniqueName: \"kubernetes.io/host-path/dc720361-519e-4f58-8d06-0aa872643004-logpodpath\") pod \"datadog-xvhf6\" (UID: \"dc720361-519e-4f58-8d06-0aa872643004\") " pod="ops/datadog-xvhf6" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.699095 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cilium-cgroup\" (UniqueName: \"kubernetes.io/host-path/2ee20d47-0618-452a-a258-5e3909a79874-cilium-cgroup\") pod \"cilium-kndsv\" (UID: \"2ee20d47-0618-452a-a258-5e3909a79874\") " pod="kube-system/cilium-kndsv" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.699120 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ee20d47-0618-452a-a258-5e3909a79874-lib-modules\") pod \"cilium-kndsv\" (UID: \"2ee20d47-0618-452a-a258-5e3909a79874\") " pod="kube-system/cilium-kndsv" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.699144 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xtables-lock\" (UniqueName: \"kubernetes.io/host-path/2ee20d47-0618-452a-a258-5e3909a79874-xtables-lock\") pod \"cilium-kndsv\" (UID: \"2ee20d47-0618-452a-a258-5e3909a79874\") " pod="kube-system/cilium-kndsv" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.699267 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/138d70ce-552c-4855-b317-6be6f8883751-sys\") pod \"wazuh-agent-wrmqb\" (UID: \"138d70ce-552c-4855-b317-6be6f8883751\") " pod="ops/wazuh-agent-wrmqb" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.699309 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"boot\" (UniqueName: \"kubernetes.io/host-path/138d70ce-552c-4855-b317-6be6f8883751-boot\") pod \"wazuh-agent-wrmqb\" (UID: \"138d70ce-552c-4855-b317-6be6f8883751\") " pod="ops/wazuh-agent-wrmqb" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.699337 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log\" (UniqueName: \"kubernetes.io/host-path/138d70ce-552c-4855-b317-6be6f8883751-log\") pod \"wazuh-agent-wrmqb\" (UID: \"138d70ce-552c-4855-b317-6be6f8883751\") " pod="ops/wazuh-agent-wrmqb" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.699369 983458 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release-file\" (UniqueName: \"kubernetes.io/host-path/dc720361-519e-4f58-8d06-0aa872643004-os-release-file\") pod \"datadog-xvhf6\" (UID: \"dc720361-519e-4f58-8d06-0aa872643004\") " pod="ops/datadog-xvhf6" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.715392 983458 serving.go:387] Generated self-signed cert in-memory Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.810429 983458 event.go:389] "Event occurred" object="kube-system/local-storage" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/local-storage.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.815881 983458 event.go:389] "Event occurred" object="kube-system/aggregated-metrics-reader" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.828001 983458 event.go:389] "Event occurred" object="kube-system/aggregated-metrics-reader" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.832195 983458 event.go:389] "Event occurred" object="kube-system/auth-delegator" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.840636 983458 event.go:389] "Event occurred" object="kube-system/auth-delegator" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.846212 983458 event.go:389] "Event occurred" object="kube-system/auth-reader" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.853457 983458 event.go:389] "Event occurred" object="kube-system/auth-reader" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.859814 983458 event.go:389] "Event occurred" object="kube-system/metrics-apiservice" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.872084 983458 event.go:389] "Event occurred" object="kube-system/metrics-apiservice" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.893471 983458 event.go:389] "Event occurred" object="kube-system/metrics-server-deployment" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.950000 983458 scope.go:117] "RemoveContainer" containerID="2d4eedd50b9891e31796098c523a0175fa62d9d45f8c49b1aa6514002909043d" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.957612 983458 event.go:389] "Event occurred" object="kube-system/metrics-server-deployment" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.967292 983458 event.go:389] "Event occurred" object="kube-system/metrics-server-service" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.984868 983458 event.go:389] "Event occurred" object="kube-system/metrics-server-service" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml\"" Sep 09 03:13:24 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:24.993580 983458 event.go:389] "Event occurred" object="kube-system/resource-reader" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml\"" Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.019231 983458 event.go:389] "Event occurred" object="kube-system/resource-reader" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml\"" Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.026476 983458 event.go:389] "Event occurred" object="kube-system/rolebindings" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/rolebindings.yaml\"" Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.065162 983458 event.go:389] "Event occurred" object="kube-system/rolebindings" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/rolebindings.yaml\"" Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.074761 983458 event.go:389] "Event occurred" object="kube-system/runtimes" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/runtimes.yaml\"" Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.105609 983458 event.go:389] "Event occurred" object="kube-system/runtimes" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/runtimes.yaml\"" Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.271324 983458 controllermanager.go:169] Version: v1.30.14+k3s2 Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.274609 983458 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.274630 983458 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.274694 983458 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.274708 983458 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.274722 983458 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.274728 983458 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.274798 983458 secure_serving.go:213] Serving securely on 127.0.0.1:10258 Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.275002 983458 leaderelection.go:250] attempting to acquire leader lease kube-system/k3s-cloud-controller-manager... Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.275118 983458 tlsconfig.go:240] "Starting DynamicServingCertificateController" Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.374699 983458 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.374783 983458 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file Sep 09 03:13:25 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:25.374789 983458 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file Sep 09 03:13:26 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:26Z" level=info msg="Started tunnel to 10.6.61.141:6443" Sep 09 03:13:26 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:26Z" level=info msg="Started tunnel to 10.6.61.142:6443" Sep 09 03:13:26 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:26Z" level=info msg="Stopped tunnel to 127.0.0.1:6443" Sep 09 03:13:26 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:26Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:13:26 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:26Z" level=info msg="Connecting to proxy" url="wss://10.6.61.142:6443/v1-k3s/connect" Sep 09 03:13:26 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:26Z" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:6443/v1-k3s/connect" Sep 09 03:13:26 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:26Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:13:26 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:26Z" level=info msg="Handling backend connection request [prod-server-blue-k3s-01]" Sep 09 03:13:26 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:26Z" level=info msg="Remotedialer connected to proxy" url="wss://10.6.61.142:6443/v1-k3s/connect" Sep 09 03:13:26 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:26Z" level=info msg="Remotedialer connected to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:13:26 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:26.976167 983458 event.go:389] "Event occurred" object="prod-server-blue-k3s-01" fieldPath="" kind="Node" apiVersion="" type="Warning" reason="NodePasswordValidationFailed" message="Deferred node password secret validation failed: secrets \"prod-server-blue-k3s-01.node-password.k3s\" already exists" Sep 09 03:13:27 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:27Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller" Sep 09 03:13:27 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:27Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller" Sep 09 03:13:27 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:27Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller" Sep 09 03:13:27 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:27Z" level=info msg="Starting batch/v1, Kind=Job controller" Sep 09 03:13:27 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:27Z" level=info msg="Starting /v1, Kind=ServiceAccount controller" Sep 09 03:13:27 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:27Z" level=info msg="Starting /v1, Kind=Secret controller" Sep 09 03:13:27 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:27Z" level=info msg="Starting /v1, Kind=ConfigMap controller" Sep 09 03:13:27 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:13:27Z" level=info msg="Updating TLS secret for kube-system/k3s-serving (count: 21): map[listener.cattle.io/cn-10.6.61.133:10.6.61.133 listener.cattle.io/cn-10.6.61.141:10.6.61.141 listener.cattle.io/cn-10.6.61.142:10.6.61.142 listener.cattle.io/cn-10.6.61.143:10.6.61.143 listener.cattle.io/cn-10.6.61.144:10.6.61.144 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.30.0.1:172.30.0.1 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-kube-api-lb-prod-blue.dc.id.vida.vpc:kube-api-lb-prod-blue.dc.id.vida.vpc listener.cattle.io/cn-kube-api-lb-prod.dc.id.vida.vpc:kube-api-lb-prod.dc.id.vida.vpc listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-prod-server-blue-k3s-00:prod-server-blue-k3s-00 listener.cattle.io/cn-prod-server-blue-k3s-01:prod-server-blue-k3s-01 listener.cattle.io/cn-prod-server-green-k3s-00:prod-server-green-k3s-00 listener.cattle.io/cn-prod-server-green-k3s-01:prod-server-green-k3s-01 listener.cattle.io/cn-prod-server-k3s-00:prod-server-k3s-00 listener.cattle.io/cn-prod-server-k3s-01:prod-server-k3s-01 listener.cattle.io/fingerprint:SHA1=99FEDC097158139980B7154701C1DF6F4CAEDB8D]" Sep 09 03:13:27 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:27.206090 983458 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 09 03:13:31 prod-server-blue-k3s-01 k3s[983458]: I0909 03:13:31.963450 983458 event.go:389] "Event occurred" object="prod-server-blue-k3s-01" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="NodePasswordValidationComplete" message="Deferred node password secret validation complete" Sep 09 03:18:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:18:16Z" level=info msg="COMPACT compactRev=653334825 targetCompactRev=653335825 currentRev=653341816" Sep 09 03:18:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:18:16Z" level=info msg="COMPACT deleted 1000 rows from 1000 revisions in 49.543165ms - compacted to 653335825/653341816" Sep 09 03:18:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:18:16Z" level=info msg="COMPACT compactRev=653335825 targetCompactRev=653336825 currentRev=653341816" Sep 09 03:18:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:18:16Z" level=info msg="COMPACT deleted 955 rows from 1000 revisions in 37.793142ms - compacted to 653336825/653341816" Sep 09 03:18:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:18:16Z" level=info msg="COMPACT compactRev=653336825 targetCompactRev=653337825 currentRev=653341816" Sep 09 03:18:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:18:16Z" level=info msg="COMPACT deleted 972 rows from 1000 revisions in 37.255664ms - compacted to 653337825/653341816" Sep 09 03:18:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:18:16Z" level=info msg="COMPACT compactRev=653337825 targetCompactRev=653338825 currentRev=653341817" Sep 09 03:18:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:18:16Z" level=info msg="COMPACT deleted 995 rows from 1000 revisions in 29.47063ms - compacted to 653338825/653341817" Sep 09 03:18:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:18:16Z" level=info msg="COMPACT compactRev=653338825 targetCompactRev=653338933 currentRev=653341817" Sep 09 03:18:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:18:16Z" level=info msg="COMPACT deleted 78 rows from 108 revisions in 20.138954ms - compacted to 653338933/653341817" Sep 09 03:18:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:18:16Z" level=info msg="COMPACT compacted from 653334825 to 653338933 in 5 transactions over 185ms" Sep 09 03:20:49 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:20:49Z" level=warning msg="Slow SQL (started: 2025-09-09 03:20:44.417580706 +0000 UTC m=+448.932353418) (total time: 5.282518582s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=5.282518582s Sep 09 03:20:49 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:20:49Z" level=warning msg="Slow SQL (started: 2025-09-09 03:20:42.832822718 +0000 UTC m=+447.347595434) (total time: 6.867496414s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=6.867496414s Sep 09 03:20:49 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:20:49Z" level=warning msg="Slow SQL (started: 2025-09-09 03:20:42.730466174 +0000 UTC m=+447.245238889) (total time: 6.974272581s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=6.974272581s Sep 09 03:20:49 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:20:49Z" level=info msg="Slow SQL (started: 2025-09-09 03:20:47.840930816 +0000 UTC m=+452.355703548) (total time: 1.86653642s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=1.86653642s Sep 09 03:20:49 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:20:49Z" level=error msg="insert error for key /registry/masterleases/10.6.61.142: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=653338986 key=/registry/masterleases/10.6.61.142 previousRevision=653343158 Sep 09 03:23:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:16Z" level=info msg="COMPACT compactRev=653338933 targetCompactRev=653339933 currentRev=653344697" Sep 09 03:23:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:16Z" level=info msg="COMPACT deleted 925 rows from 1000 revisions in 39.80224ms - compacted to 653339933/653344697" Sep 09 03:23:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:16Z" level=info msg="COMPACT compactRev=653339933 targetCompactRev=653340933 currentRev=653344697" Sep 09 03:23:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:16Z" level=info msg="COMPACT deleted 1000 rows from 1000 revisions in 31.133379ms - compacted to 653340933/653344697" Sep 09 03:23:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:16Z" level=info msg="COMPACT compactRev=653340933 targetCompactRev=653341817 currentRev=653344697" Sep 09 03:23:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:16Z" level=info msg="COMPACT deleted 884 rows from 884 revisions in 23.101892ms - compacted to 653341817/653344697" Sep 09 03:23:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:16Z" level=info msg="COMPACT compacted from 653338933 to 653341817 in 3 transactions over 100ms" Sep 09 03:23:31 prod-server-blue-k3s-01 k3s[983458]: W0909 03:23:31.814717 983458 transport.go:356] Unable to cancel request for *otelhttp.Transport Sep 09 03:23:31 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:31.814864 983458 controller.go:195] "Failed to update lease" err="Put \"https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-server-blue-k3s-01?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 03:23:31 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:31.815143 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:31 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:31.815180 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:31 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:31.815143 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 15.299µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:31 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:31Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:21.821900072 +0000 UTC m=+606.336672792) (total time: 9.993189677s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.993189677s Sep 09 03:23:31 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:31Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-server-blue-k3s-01: context canceled" createRevision=320940478 key=/registry/leases/kube-node-lease/prod-server-blue-k3s-01 previousRevision=653344634 Sep 09 03:23:31 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:31.817142 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:31 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:31.817287 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.270828ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-server-blue-k3s-01" result: Sep 09 03:23:32 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:32Z" level=error msg="Remotedialer proxy error; reconnecting..." error="websocket: close 1006 (abnormal closure): unexpected EOF" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:32 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:32Z" level=warning msg="[2] encountered error \"use of closed network connection\" while writing error \"writeto tcp 127.0.0.1:35904->127.0.0.1:10250: read tcp 127.0.0.1:35904->127.0.0.1:10250: use of closed network connection\" to close remotedialer" Sep 09 03:23:32 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:32Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF" Sep 09 03:23:35 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:35Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:35 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:35Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.6.61.141:6443: connect: connection refused" Sep 09 03:23:35 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:35Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.6.61.141:6443: connect: connection refused" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:38 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:38Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:38 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:38Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.6.61.141:6443: connect: connection refused" Sep 09 03:23:38 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:38Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.6.61.141:6443: connect: connection refused" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:41 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:41Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:41 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:41Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.6.61.141:6443: connect: connection refused" Sep 09 03:23:41 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:41Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.6.61.141:6443: connect: connection refused" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:41 prod-server-blue-k3s-01 k3s[983458]: W0909 03:23:41.815358 983458 transport.go:356] Unable to cancel request for *otelhttp.Transport Sep 09 03:23:41 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:41.815489 983458 controller.go:195] "Failed to update lease" err="Put \"https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-server-blue-k3s-01?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 03:23:41 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:41.815837 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:41 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:41.815876 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:41 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:41.815961 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 62.717µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:41 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:41Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:31.823693178 +0000 UTC m=+616.338465895) (total time: 9.992359389s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.992359389s Sep 09 03:23:41 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:41Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-server-blue-k3s-01: context canceled" createRevision=320940478 key=/registry/leases/kube-node-lease/prod-server-blue-k3s-01 previousRevision=653344634 Sep 09 03:23:41 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:41.816937 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:41 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:41.817123 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.132411ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-server-blue-k3s-01" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.059534 983458 leaderelection.go:383] Failed to update lock: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": context deadline exceeded Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=info msg="Slow SQL (started: 2025-09-09 03:23:37.066456934 +0000 UTC m=+621.581229664) (total time: 4.993282414s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.993282414s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-system/kube-scheduler: timeout: context deadline exceeded" createRevision=387 key=/registry/leases/kube-system/kube-scheduler previousRevision=653344736 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.060183 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 14.831µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.060835 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.061477 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.062320 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.063172 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.446077ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.413656 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.413718 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.413770 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 12.179µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:32.921743946 +0000 UTC m=+617.436516649) (total time: 9.49248274s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.49248274s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/ingress-nginx/ingress-nginx-leader: context canceled" createRevision=98641045 key=/registry/leases/ingress-nginx/ingress-nginx-leader previousRevision=653344729 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.415090 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.415265 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.723286ms, PUT "/apis/coordination.k8s.io/v1/namespaces/ingress-nginx/leases/ingress-nginx-leader" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.419506 983458 leaderelection.go:383] Failed to update lock: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/k3s-cloud-controller-manager?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers) Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=info msg="Slow SQL (started: 2025-09-09 03:23:37.425050248 +0000 UTC m=+621.939822967) (total time: 4.994648465s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.994648465s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.419809 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 16.324µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-system/k3s-cloud-controller-manager: context canceled" createRevision=368 key=/registry/leases/kube-system/k3s-cloud-controller-manager previousRevision=653344738 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.419857 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.420986 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.421060 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.422820 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.039421ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/k3s-cloud-controller-manager" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.472202 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 6.669µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.472226 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:32.950819439 +0000 UTC m=+617.465592117) (total time: 9.521405592s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.521405592s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/ingress-nginx-secondary/ingress-nginx-secondary-leader: context canceled" createRevision=474540760 key=/registry/leases/ingress-nginx-secondary/ingress-nginx-secondary-leader previousRevision=653344730 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.473392 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.473474 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.474861 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.715183ms, PUT "/apis/coordination.k8s.io/v1/namespaces/ingress-nginx-secondary/leases/ingress-nginx-secondary-leader" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.783478 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 18.714µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:32.918859978 +0000 UTC m=+617.433632676) (total time: 9.864632443s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.864632443s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ops-blue-1-89a427cc: context canceled" createRevision=321006889 key=/registry/leases/kube-node-lease/prod-agent-ops-blue-1-89a427cc previousRevision=653344659 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.784108 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.784157 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 10.011µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:32.880563283 +0000 UTC m=+617.395335962) (total time: 9.903631198s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.903631198s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ingress-blue-0-ac9168ff: context canceled" createRevision=320947169 key=/registry/leases/kube-node-lease/prod-agent-ingress-blue-0-ac9168ff previousRevision=653344686 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.784348 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 58.905µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.784475 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.784557 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:32.915618056 +0000 UTC m=+617.430390749) (total time: 9.868857858s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.868857858s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-3-aef87d06: context canceled" createRevision=320960233 key=/registry/leases/kube-node-lease/prod-agent-app-blue-3-aef87d06 previousRevision=653344685 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.784878 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.784967 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 52.159µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.785175 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.785229 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 33.128µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.016219496 +0000 UTC m=+617.530992174) (total time: 9.769107943s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.769107943s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ingress-blue-2-39068466: context canceled" createRevision=320947107 key=/registry/leases/kube-node-lease/prod-agent-ingress-blue-2-39068466 previousRevision=653344680 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.785454 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.785537 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 50.102µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.785630 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.785892 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:32.947915582 +0000 UTC m=+617.462688274) (total time: 9.837959371s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.837959371s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.785970 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 46.08µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-4-79b5072b: context canceled" createRevision=320960218 key=/registry/leases/kube-node-lease/prod-agent-app-blue-4-79b5072b previousRevision=653344692 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.786024 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.786311 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.786725 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.787880 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.787921 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.787893 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.787950 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.881182ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ingress-blue-0-ac9168ff" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.788210 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.788274 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 36.372µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:32.909965687 +0000 UTC m=+617.424738378) (total time: 9.878407233s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.878407233s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-10-1ef6ca3c: context canceled" createRevision=613356118 key=/registry/leases/kube-node-lease/prod-agent-app-blue-10-1ef6ca3c previousRevision=653344688 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:32.896681858 +0000 UTC m=+617.411454536) (total time: 9.891856296s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.891856296s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ingress-blue-1-4323cca2: context canceled" createRevision=320946796 key=/registry/leases/kube-node-lease/prod-agent-ingress-blue-1-4323cca2 previousRevision=653344650 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:32.911414251 +0000 UTC m=+617.426186929) (total time: 9.877291602s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.877291602s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-0-f0380b7a: context canceled" createRevision=320960246 key=/registry/leases/kube-node-lease/prod-agent-app-blue-0-f0380b7a previousRevision=653344668 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.788913 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.789202 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.789238 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.789285 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.051µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.789285 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 52.372µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.789415 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.789466 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.789562 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.789739 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.427113ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ingress-blue-2-39068466" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.789772 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.789855 983458 timeout.go:142] post-timeout activity - time-elapsed: 5.461177ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-3-aef87d06" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:32.884179587 +0000 UTC m=+617.398952264) (total time: 9.906489506s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.906489506s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.790802 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.791026 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.791036 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.581382ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-9-63cdf3ff" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-1-a234b216: context canceled" createRevision=320959216 key=/registry/leases/kube-node-lease/prod-agent-app-blue-1-a234b216 previousRevision=653344667 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.791239 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.791243 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.755642ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ingress-blue-1-4323cca2" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:32.891328439 +0000 UTC m=+617.406101131) (total time: 9.899305084s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.899305084s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-9-63cdf3ff: context canceled" createRevision=613356165 key=/registry/leases/kube-node-lease/prod-agent-app-blue-9-63cdf3ff previousRevision=653344696 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.791399 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.791425 983458 timeout.go:142] post-timeout activity - time-elapsed: 6.173255ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-4-79b5072b" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.791941 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.784098 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.792276 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.544168ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-0-f0380b7a" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.792349 983458 timeout.go:142] post-timeout activity - time-elapsed: 5.16019ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-10-1ef6ca3c" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.792801 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.294948ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-1-a234b216" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.793008 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.793993 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.794137 983458 timeout.go:142] post-timeout activity - time-elapsed: 10.74534ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ops-blue-1-89a427cc" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.820566 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.820592 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 7.918µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.820625 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:32.904853515 +0000 UTC m=+617.419626193) (total time: 9.915761016s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.915761016s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-11-2dccff04: context canceled" createRevision=613356009 key=/registry/leases/kube-node-lease/prod-agent-app-blue-11-2dccff04 previousRevision=653344693 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.821804 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.821893 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.286842ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-11-2dccff04" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.994298 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.994369 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.994459 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 63.771µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.024712552 +0000 UTC m=+617.539485245) (total time: 9.969873892s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.969873892s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-2-9c964185: context canceled" createRevision=320960197 key=/registry/leases/kube-node-lease/prod-agent-app-blue-2-9c964185 previousRevision=653344675 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.995026 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 11.679µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.075883319 +0000 UTC m=+617.590656006) (total time: 9.919128062s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.919128062s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.995094 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ops-blue-0-4b80a6a8: context canceled" createRevision=321006849 key=/registry/leases/kube-node-lease/prod-agent-ops-blue-0-4b80a6a8 previousRevision=653344656 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.995463 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.996145 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.996186 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.996216 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 9.761µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.089443348 +0000 UTC m=+617.604216041) (total time: 9.906713982s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.906713982s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-7-c3860444: context canceled" createRevision=320959253 key=/registry/leases/kube-node-lease/prod-agent-app-blue-7-c3860444 previousRevision=653344660 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.996306 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.807187ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-2-9c964185" result: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.996545 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.996685 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 5.172µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.009767741 +0000 UTC m=+617.524540430) (total time: 9.986874143s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.986874143s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-8-49a23f58: context canceled" createRevision=448789453 key=/registry/leases/kube-node-lease/prod-agent-app-blue-8-49a23f58 previousRevision=653344734 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.996898 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.996954 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.997002 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 19.645µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.071796251 +0000 UTC m=+617.586568943) (total time: 9.925222799s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.925222799s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-3-47319b7a: context canceled" createRevision=624036915 key=/registry/leases/kube-node-lease/prod-agent-data-green-3-47319b7a previousRevision=653344684 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.065977766 +0000 UTC m=+617.580750458) (total time: 9.931356113s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.931356113s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-5-f52e9ef0: context canceled" createRevision=624037756 key=/registry/leases/kube-node-lease/prod-agent-data-green-5-f52e9ef0 previousRevision=653344682 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.997771 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.997941 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.997964 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.998065 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.998235 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.998242 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.998310 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 58.601µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.998355 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.998483 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 9.633µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.998495 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.079194903 +0000 UTC m=+617.593967594) (total time: 9.919361307s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.919361307s Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-4-96deb91b: context canceled" createRevision=624037787 key=/registry/leases/kube-node-lease/prod-agent-data-green-4-96deb91b previousRevision=653344689 Sep 09 03:23:42 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.998682 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 169.303µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.017737746 +0000 UTC m=+617.532510438) (total time: 9.981002261s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.981002261s Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-5-bb69767d: context canceled" createRevision=320959392 key=/registry/leases/kube-node-lease/prod-agent-app-blue-5-bb69767d previousRevision=653344674 Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.999001 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.947648ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-7-c3860444" result: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.999060 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.475543ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-8-49a23f58" result: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.999087 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.148614ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ops-blue-0-4b80a6a8" result: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.999102 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.999310 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.999369 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.999590 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 58.803µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.072242272 +0000 UTC m=+617.587014964) (total time: 9.927407377s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.927407377s Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.999718 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:42Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-7-93605edd: context canceled" createRevision=624037614 key=/registry/leases/kube-node-lease/prod-agent-data-green-7-93605edd previousRevision=653344691 Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:42.999781 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.000515 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.000520 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.540886ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-3-47319b7a" result: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.000529 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.000712 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:43Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.022154413 +0000 UTC m=+617.536927105) (total time: 9.978720763s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.978720763s Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:43Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-6-cd93959b: context canceled" createRevision=320959048 key=/registry/leases/kube-node-lease/prod-agent-app-blue-6-cd93959b previousRevision=653344672 Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.000956 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.000972 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.000998 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 28.123µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.001020 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 9.05µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:43Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.100098245 +0000 UTC m=+617.614870957) (total time: 9.900955468s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.900955468s Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:43Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-0-9b98de05: context canceled" createRevision=623966046 key=/registry/leases/kube-node-lease/prod-agent-data-green-0-9b98de05 previousRevision=653344666 Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.001124 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.793852ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-4-96deb91b" result: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.001164 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.001223 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.001243 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 3.47µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:43Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.075001919 +0000 UTC m=+617.589774612) (total time: 9.926303292s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.926303292s Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:43Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-1-efd5b590: context canceled" createRevision=623966116 key=/registry/leases/kube-node-lease/prod-agent-data-green-1-efd5b590 previousRevision=653344657 Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.001584 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.001810 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.001934 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.711071ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-5-f52e9ef0" result: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.002180 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.002214 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.002262 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 50.398µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.002553 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.786198ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-7-93605edd" result: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:43Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.088429427 +0000 UTC m=+617.603202119) (total time: 9.914107391s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.914107391s Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:43Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-6-420706e2: context canceled" createRevision=624036980 key=/registry/leases/kube-node-lease/prod-agent-data-green-6-420706e2 previousRevision=653344661 Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.002731 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.003042 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.003133 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.003242 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.003288 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.771848ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-5-bb69767d" result: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.003765 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.739627ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-0-9b98de05" result: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.003840 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.004301 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.004306 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.116619ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-1-efd5b590" result: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.005273 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.44738ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-6-cd93959b" result: Sep 09 03:23:43 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:43.005425 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.138569ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-6-420706e2" result: Sep 09 03:23:44 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:44Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:44 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:44Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.6.61.141:6443: connect: connection refused" Sep 09 03:23:44 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:44Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.6.61.141:6443: connect: connection refused" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.289395 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.548µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:45Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:35.29497468 +0000 UTC m=+619.809747423) (total time: 9.994368759s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.994368759s Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:45Z" level=error msg="insert error for key /registry/configmaps/ops/datadog-leader-election: timeout: context deadline exceeded" createRevision=4206015 key=/registry/configmaps/ops/datadog-leader-election previousRevision=653344733 Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.289405 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:45Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.290516 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.290879 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.291851 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.531443ms, PUT "/api/v1/namespaces/ops/configmaps/datadog-leader-election" result: Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.447326 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.447784 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.447415 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 13.355µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:45Z" level=info msg="Slow SQL (started: 2025-09-09 03:23:40.45338742 +0000 UTC m=+624.968160154) (total time: 4.993974121s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.993974121s Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:45Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-operator-resource-lock: timeout: context deadline exceeded" createRevision=706 key=/registry/leases/kube-system/cilium-operator-resource-lock previousRevision=653344739 Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:45Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.449285 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.449737 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.445883ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result: Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.975638 983458 kubelet_node_status.go:544] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"NetworkUnavailable\\\"},{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:23:35Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:23:35Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:23:35Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:23:35Z\\\",\\\"type\\\":\\\"Ready\\\"}]}}\" for node \"prod-server-blue-k3s-01\": Patch \"https://127.0.0.1:6443/api/v1/nodes/prod-server-blue-k3s-01/status?timeout=10s\": context deadline exceeded" Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:45Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:36.006134933 +0000 UTC m=+620.520907676) (total time: 9.969777061s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.969777061s Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:45Z" level=error msg="insert error for key /registry/minions/prod-server-blue-k3s-01: timeout: context deadline exceeded" createRevision=320940137 key=/registry/minions/prod-server-blue-k3s-01 previousRevision=653341971 Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.976161 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.976208 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.976256 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 11.597µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:45Z" level=error msg="error in txn compare: success:\\n6volumes.kubernetes.io/controller-managed-attach-detach\\022\\004truer\\027wrangler.cattle.io/node\\212\\001\\232\\001\\n'k3s-supervisor@prod-server-green-k3s-01\\022\\006Update\\032\\002v1\\\"\\010\\010\\302\\234\\302\\266\\006\\020\\0002\\010FieldsV1:M\\nK{\\\"f:metadata\\\":{\\\"f:finalizers\\\":{\\\".\\\":{},\\\"v:\\\\\\\"wrangler.cattle.io/node\\\\\\\"\\\":{}}}}B\\000\\212\\001\\276\\001\\n&k3s-supervisor@prod-server-blue-k3s-01\\022\\006Update\\032\\002v1\\\"\\010\\010\\303\\234\\302\\266\\006\\020\\0002\\010FieldsV1:r\\np{\\\"f:metadata\\\":{\\\"f:labels\\\":{\\\"f:node-role.kubernetes.io/control-plane\\\":{},\\\"f:node-role.kubernetes.io/master\\\":{}}}}B\\000\\212\\001\\366\\001\\n\\016Go-http-client\\022\\006Update\\032\\002v1\\\"\\010\\010\\276\\235\\302\\266\\006\\020\\0002\\010FieldsV1:\\273\\001\\n\\270\\001{\\\"f:status\\\":{\\\"f:conditions\\\":{\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"NetworkUnavailable\\\\\\\"}\\\":{\\\".\\\":{},\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{},\\\"f:type\\\":{}}}}}B\\006status\\212\\001\\350\\005\\n\\003k3s\\022\\006Update\\032\\002v1\\\"\\010\\010\\352\\357\\373\\305\\006\\020\\0002\\010FieldsV1:\\276\\005\\n\\273\\005{\\\"f:metadata\\\":{\\\"f:annotations\\\":{\\\".\\\":{},\\\"f:alpha.kubernetes.io/provided-node-ip\\\":{},\\\"f:k3s.io/hostname\\\":{},\\\"f:k3s.io/internal-ip\\\":{},\\\"f:k3s.io/node-args\\\":{},\\\"f:k3s.io/node-config-hash\\\":{},\\\"f:k3s.io/node-env\\\":{},\\\"f:node.alpha.kubernetes.io/ttl\\\":{},\\\"f:volumes.kubernetes.io/controller-managed-attach-detach\\\":{}},\\\"f:labels\\\":{\\\".\\\":{},\\\"f:beta.kubernetes.io/arch\\\":{},\\\"f:beta.kubernetes.io/instance-type\\\":{},\\\"f:beta.kubernetes.io/os\\\":{},\\\"f:cluster-name\\\":{},\\\"f:kubernetes.io/arch\\\":{},\\\"f:kubernetes.io/hostname\\\":{},\\\"f:kubernetes.io/os\\\":{},\\\"f:node.kubernetes.io/controller\\\":{},\\\"f:node.kubernetes.io/instance-type\\\":{}}},\\\"f:spec\\\":{\\\"f:podCIDR\\\":{},\\\"f:podCIDRs\\\":{\\\".\\\":{},\\\"v:\\\\\\\"10.128.38.0/24\\\\\\\"\\\":{}},\\\"f:providerID\\\":{}}}B\\000\\212\\001Y\\n\\027cilium-operator-generic\\022\\006Update\\032\\002v1\\\"\\010\\010\\204\\360\\373\\305\\006\\020\\0002\\010FieldsV1:\\034\\n\\032{\\\"f:spec\\\":{\\\"f:taints\\\":{}}}B\\000\\212\\001\\211\\006\\n\\003k3s\\022\\006Update\\032\\002v1\\\"\\010\\010\\267\\273\\376\\305\\006\\020\\0002\\010FieldsV1:\\331\\005\\n\\326\\005{\\\"f:status\\\":{\\\"f:allocatable\\\":{\\\"f:ephemeral-storage\\\":{}},\\\"f:capacity\\\":{\\\"f:ephemeral-storage\\\":{}},\\\"f:conditions\\\":{\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"DiskPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"MemoryPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"PIDPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"Ready\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{}}},\\\"f:images\\\":{},\\\"f:nodeInfo\\\":{\\\"f:containerRuntimeVersion\\\":{},\\\"f:kubeProxyVersion\\\":{},\\\"f:kubeletVersion\\\":{}}}}B\\006status\\022z\\n\\01610.128.38.0/24\\022\\000\\032\\035k3s://prod-server-blue-k3s-01 \\000*5\\n%node-role.kubernetes.io/control-plane\\022\\000\\032\\nNoSchedule:\\01610.128.38.0/24\\032\\253\\030\\n\\n\\n\\003cpu\\022\\003\\n\\0018\\n\\\"\\n\\021ephemeral-storage\\022\\r\\n\\013100951636Ki\\n\\024\\n\\rhugepages-1Gi\\022\\003\\n\\0010\\n\\024\\n\\rhugepages-2Mi\\022\\003\\n\\0010\\n\\026\\n\\006memory\\022\\014\\n\\n12243396Ki\\n\\r\\n\\004pods\\022\\005\\n\\003110\\022\\n\\n\\003cpu\\022\\003\\n\\0018\\022\\\"\\n\\021ephemeral-storage\\022\\r\\n\\01398205751424\\022\\024\\n\\rhugepages-1Gi\\022\\003\\n\\0010\\022\\024\\n\\rhugepages-2Mi\\022\\003\\n\\0010\\022\\026\\n\\006memory\\022\\014\\n\\n12243396Ki\\022\\r\\n\\004pods\\022\\005\\n\\003110\\032\\000\\\"[\\n\\022NetworkUnavailable\\022\\005False\\032\\010\\010\\276\\235\\302\\266\\006\\020\\000\\\"\\010\\010\\276\\235\\302\\266\\006\\020\\000*\\nCiliumIsUp2\\036Cilium is running on this node\\\"p\\n\\016MemoryPressure\\022\\005False\\032\\010\\010\\267\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\331\\370\\372\\305\\006\\020\\000*\\032KubeletHasSufficientMemory2'kubelet has sufficient memory available\\\"a\\n\\014DiskPressure\\022\\005False\\032\\010\\010\\267\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\331\\370\\372\\305\\006\\020\\000*\\030KubeletHasNoDiskPressure2\\034kubelet has no disk pressure\\\"g\\n\\013PIDPressure\\022\\005False\\032\\010\\010\\267\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\331\\370\\372\\305\\006\\020\\000*\\027KubeletHasSufficientPID2$kubelet has sufficient PID available\\\"P\\n\\005Ready\\022\\004True\\032\\010\\010\\267\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\331\\370\\372\\305\\006\\020\\000*\\014KubeletReady2\\037kubelet is posting ready status*\\031\\n\\nInternalIP\\022\\01310.6.61.142*#\\n\\010Hostname\\022\\027prod-server-blue-k3s-012\\005\\n\\003\\010\\212P:\\334\\001\\n 930d76008d2c408e9ba36a050eb87a99\\022$4ed00a42-0756-e0b3-96f9-addb38b101a8\\032$d55d8d1f-c315-4039-a84d-e6ace7b1837c\\\"\\0225.15.0-119-generic*\\022Ubuntu 22.04.4 LTS2\\030containerd://1.7.27-k3s1:\\rv1.30.14+k3s2B\\rv1.30.14+k3s2J\\005linuxR\\005amd64B\\225\\001\\ndpublic.ecr.aws/datadog/agent@sha256:465da828947d6c055eacf399df8d0f96423f6e029fd1218a806f091a0dc0200b\\n'public.ecr.aws/datadog/agent:7.62.0-jmx\\020\\207\\330\\325\\347\\001B\\334\\001\\n}868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops@sha256:307c60e7c0a570a7bbad2a45e1315f5ec6c39f6aea2b3d7dc521b57219fd73f1\\nU868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops:docker-wazuh-agent-f8b1f9f-main\\020\\311\\253\\321\\342\\001B\\334\\001\\n}868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops@sha256:e88bf7e606df57a9e7c3efa1f489132a72ad76f981bf7419e9598ce83ff9005c\\nU868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops:docker-wazuh-agent-a62b8c9-main\\020\\323\\304\\323\\335\\001B\\204\\001\\n]quay.io/cilium/cilium@sha256:9dc74ba5321c999e498b5f05202c7e27015360dd19278f19b15a25bee79d22f1\\n\\036quay.io/cilium/cilium:v1.13.18\\020\\341\\201\\210_B\\203\\001\\n]quay.io/cilium/cilium@sha256:5a94b561f4651fcfd85970a50bc78b201cfbd6e2ab1a03848eab25a82832653a\\n\\035quay.io/cilium/cilium:v1.14.0\\020\\300\\317\\204WBd\\n]quay.io/cilium/cilium@sha256:6544a3441b086a2e09005d3e21d1a4afb216fae19c5a60b35793c8a9438f8f68\\020\\260\\375\\356RB\\227\\001\\ngquay.io/cilium/operator-generic@sha256:3014d4bcb8352f0ddef90fa3b5eb1bbf179b91024813a90a0066eb4517ba93c9\\n'quay.io/cilium/operator-generic:v1.14.0\\020\\270\\377\\240\\013B\\254\\001\\nrdocker.io/rancher/mirrored-coredns-coredns@sha256:9eb4929fd181541e7743959884000f010bcc8b1ebf8715ff0965be04509882e5\\n1docker.io/rancher/mirrored-coredns-coredns:1.12.1\\020\\304\\233\\326\\nBn\\ngquay.io/cilium/operator-generic@sha256:4b58d5b33e53378355f6e8ceb525ccf938b7b6f5384b35373f1f46787467ebf5\\020\\312\\372\\306\\nB\\252\\001\\nqdocker.io/rancher/mirrored-metrics-server@sha256:dccf8474fb910fef261d31d9483d7e4c1df7b86cf4d638fb6a7d7c88bd51600a\\n0docker.io/rancher/mirrored-metrics-server:v0.7.2\\020\\252\\352\\245\\tB\\254\\001\\nrdocker.io/rancher/mirrored-coredns-coredns@sha256:a11fafae1f8037cbbd66c5afa40ba2423936b72b4fd50a7034a7e8b955163594\\n1docker.io/rancher/mirrored-coredns-coredns:1.10.1\\020\\271\\225\\334\\007B\\230\\001\\nhquay.io/prometheus/node-exporter@sha256:4032c6d5bfd752342c3e631c2f1de93ba6b86c41db6b167b9a35372c139e7706\\n'quay.io/prometheus/node-exporter:v1.8.2\\020\\370\\371\\336\\005B\\224\\001\\nhdocker.io/rancher/mirrored-pause@sha256:74c4244427b7312c5b901fe0f67cbc53683d06f4f24c6faee65d4182bf0fa893\\n$docker.io/rancher/mirrored-pause:3.6\\020\\227\\263\\022\\032\\000\\\"\\000\" > > failure: > : context deadline exceeded" Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.977365 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:45 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:45.977598 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.513668ms, PATCH "/api/v1/nodes/prod-server-blue-k3s-01/status" result: Sep 09 03:23:47 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:47.603687 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:47 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:47.603730 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:47 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:47.603756 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.418µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:47 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:47Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:37.617641714 +0000 UTC m=+622.132414400) (total time: 9.986142475s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.986142475s Sep 09 03:23:47 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:47Z" level=error msg="insert error for key /registry/minions/prod-agent-app-blue-4-79b5072b: context canceled" createRevision=320959816 key=/registry/minions/prod-agent-app-blue-4-79b5072b previousRevision=653341980 Sep 09 03:23:47 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:47.605207 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:47 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:47.605350 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.74809ms, PATCH "/api/v1/nodes/prod-agent-app-blue-4-79b5072b/status" result: Sep 09 03:23:47 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:47Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:47 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:47Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.6.61.141:6443: connect: connection refused" Sep 09 03:23:47 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:47Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.6.61.141:6443: connect: connection refused" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:50 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:50Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:50 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:50Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.6.61.141:6443: connect: connection refused" Sep 09 03:23:50 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:50Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.6.61.141:6443: connect: connection refused" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.163624 983458 leaderelection.go:383] Failed to update lock: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/k3s-cloud-controller-manager?timeout=5s": context deadline exceeded Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:51Z" level=info msg="Slow SQL (started: 2025-09-09 03:23:46.16894565 +0000 UTC m=+630.683718410) (total time: 4.994932962s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.994932962s Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.163968 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.164004 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 7.653µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:51Z" level=error msg="insert error for key /registry/leases/kube-system/k3s-cloud-controller-manager: context canceled" createRevision=368 key=/registry/leases/kube-system/k3s-cloud-controller-manager previousRevision=653344738 Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.164014 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.165878 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.166018 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.196118ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/k3s-cloud-controller-manager" result: Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:51Z" level=info msg="Slow SQL (started: 2025-09-09 03:23:46.231935859 +0000 UTC m=+630.746708618) (total time: 4.994963257s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.994963257s Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:51Z" level=error msg="insert error for key /registry/leases/kube-system/kube-scheduler: timeout: context deadline exceeded" createRevision=387 key=/registry/leases/kube-system/kube-scheduler previousRevision=653344736 Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.227413 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.227455 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.227491 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.958µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.227530 983458 leaderelection.go:383] Failed to update lock: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers) Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:51Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.228546 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.228695 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.445681ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result: Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: W0909 03:23:51.816748 983458 transport.go:356] Unable to cancel request for *otelhttp.Transport Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:51Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:41.844218263 +0000 UTC m=+626.358990992) (total time: 9.97244401s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.97244401s Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:51Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-server-blue-k3s-01: timeout: context deadline exceeded" createRevision=320940478 key=/registry/leases/kube-node-lease/prod-server-blue-k3s-01 previousRevision=653344634 Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.817106 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.817150 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.817184 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 7.717µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.817335 983458 controller.go:195] "Failed to update lease" err="Put \"https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-server-blue-k3s-01?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:51Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.818318 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:51 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:51.818619 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.10196ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-server-blue-k3s-01" result: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.622848 983458 leaderelection.go:340] Failed to update lock optimitically: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/k3s?timeout=15m0s": context deadline exceeded, falling back to slow path Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.623054 983458 leaderelection.go:347] error retrieving resource lock kube-system/k3s: context deadline exceeded Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: I0909 03:23:52.623079 983458 leaderelection.go:285] failed to renew lease kube-system/k3s: timed out waiting for the condition Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.623148 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.623193 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.623229 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 12.065µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:22.632151692 +0000 UTC m=+607.146924450) (total time: 29.991221287s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=29.991221287s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-system/k3s: context canceled" createRevision=234 key=/registry/leases/kube-system/k3s previousRevision=653344737 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.624346 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.624522 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.365108ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/k3s" result: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:50888: failed to find Session for client prod-server-blue-k3s-00" Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.784967 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.785030 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:42.827047127 +0000 UTC m=+627.341819804) (total time: 9.957804024s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.957804024s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ops-blue-1-89a427cc: context canceled" createRevision=321006889 key=/registry/leases/kube-node-lease/prod-agent-ops-blue-1-89a427cc previousRevision=653344659 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.785179 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.663µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.785215 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.785299 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.785310 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 11.989µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.785394 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.052µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:42.829472036 +0000 UTC m=+627.344244729) (total time: 9.955980854s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.955980854s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ingress-blue-0-ac9168ff: context canceled" createRevision=320947169 key=/registry/leases/kube-node-lease/prod-agent-ingress-blue-0-ac9168ff previousRevision=653344686 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:42.79584062 +0000 UTC m=+627.310613338) (total time: 9.989601131s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.989601131s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-3-aef87d06: context canceled" createRevision=320960233 key=/registry/leases/kube-node-lease/prod-agent-app-blue-3-aef87d06 previousRevision=653344685 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.785967 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.786017 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.2µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.786056 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 171.118µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:42.840122446 +0000 UTC m=+627.354895128) (total time: 9.946004894s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.946004894s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-4-79b5072b: context canceled" createRevision=320960218 key=/registry/leases/kube-node-lease/prod-agent-app-blue-4-79b5072b previousRevision=653344692 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:42.834569406 +0000 UTC m=+627.349342106) (total time: 9.951581008s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.951581008s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ingress-blue-2-39068466: context canceled" createRevision=320947107 key=/registry/leases/kube-node-lease/prod-agent-ingress-blue-2-39068466 previousRevision=653344680 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.786580 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.787069 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.787162 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 53.92µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.787182 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.787249 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.787661 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:42.834236206 +0000 UTC m=+627.349008924) (total time: 9.953482961s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.953482961s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-0-f0380b7a: context canceled" createRevision=320960246 key=/registry/leases/kube-node-lease/prod-agent-app-blue-0-f0380b7a previousRevision=653344668 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.788538 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.788714 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.788719 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.789090 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.188534ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-4-79b5072b" result: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.789144 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 6.241µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:42.825545546 +0000 UTC m=+627.340318252) (total time: 9.963856724s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.963856724s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-10-1ef6ca3c: context canceled" createRevision=613356118 key=/registry/leases/kube-node-lease/prod-agent-app-blue-10-1ef6ca3c previousRevision=653344688 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.789578 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:42.840233258 +0000 UTC m=+627.355006017) (total time: 9.949368864s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.949368864s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-9-63cdf3ff: context canceled" createRevision=613356165 key=/registry/leases/kube-node-lease/prod-agent-app-blue-9-63cdf3ff previousRevision=653344696 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:42.834484522 +0000 UTC m=+627.349257205) (total time: 9.955162687s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.955162687s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ingress-blue-1-4323cca2: timeout: context deadline exceeded" createRevision=320946796 key=/registry/leases/kube-node-lease/prod-agent-ingress-blue-1-4323cca2 previousRevision=653344650 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.790374 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.790432 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.790447 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 33.359µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.790485 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.790511 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 45.781µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.790566 983458 timeout.go:142] post-timeout activity - time-elapsed: 5.222279ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ingress-blue-0-ac9168ff" result: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.790657 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.790678 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:42.843684444 +0000 UTC m=+627.358457128) (total time: 9.946268323s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.946268323s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-1-a234b216: timeout: context deadline exceeded" createRevision=320959216 key=/registry/leases/kube-node-lease/prod-agent-app-blue-1-a234b216 previousRevision=653344667 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.791613 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.791825 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.635079ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-0-f0380b7a" result: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.791836 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.791852 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 4.398µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.791876 983458 timeout.go:142] post-timeout activity - time-elapsed: 7.14668ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ops-blue-1-89a427cc" result: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.791893 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.791941 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.791954 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.791956 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.799382ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-10-1ef6ca3c" result: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.791998 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.792880 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.792980 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.505038ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-9-63cdf3ff" result: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.792993 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.793013 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.793026 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.793051 983458 timeout.go:142] post-timeout activity - time-elapsed: 8.131237ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-3-aef87d06" result: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.794480 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.54091ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ingress-blue-1-4323cca2" result: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.794506 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.623652ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-1-a234b216" result: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.794597 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.796051 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.123626ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ingress-blue-2-39068466" result: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:43.002997296 +0000 UTC m=+627.517770006) (total time: 9.992019521s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.992019521s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-2-9c964185: timeout: context deadline exceeded" createRevision=320960197 key=/registry/leases/kube-node-lease/prod-agent-app-blue-2-9c964185 previousRevision=653344675 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.995251 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.995331 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.995374 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 9.639µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.995927 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 12.433µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.995968 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:43.042767813 +0000 UTC m=+627.557540506) (total time: 9.9531123s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.9531123s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ops-blue-0-4b80a6a8: timeout: context deadline exceeded" createRevision=321006849 key=/registry/leases/kube-node-lease/prod-agent-ops-blue-0-4b80a6a8 previousRevision=653344656 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:43.032788875 +0000 UTC m=+627.547561570) (total time: 9.963512411s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.963512411s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-7-c3860444: timeout: context deadline exceeded" createRevision=320959253 key=/registry/leases/kube-node-lease/prod-agent-app-blue-7-c3860444 previousRevision=653344660 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.996630 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.996916 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 10.215µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.996961 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.997055 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:43.051172329 +0000 UTC m=+627.565945033) (total time: 9.946018005s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.946018005s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-8-49a23f58: timeout: context deadline exceeded" createRevision=448789453 key=/registry/leases/kube-node-lease/prod-agent-app-blue-8-49a23f58 previousRevision=653344734 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.997364 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.997451 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 40.849µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.997467 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.401394ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-2-9c964185" result: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:43.048066744 +0000 UTC m=+627.562839506) (total time: 9.949671202s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.949671202s Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-3-47319b7a: context canceled" createRevision=624036915 key=/registry/leases/kube-node-lease/prod-agent-data-green-3-47319b7a previousRevision=653344684 Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.997824 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 6.086µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.997891 983458 wrap.go:54] timeout or abort while handling: method=PUT URI="/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-3-47319b7a?timeout=10s" audit-ID="b7d6b36d-8087-498e-9821-63afd5f5d859" Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.997913 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.441µs, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-3-47319b7a" result: Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.998074 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.998119 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.998134 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:52 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.998860 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.998941 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 10.243µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:43.049952735 +0000 UTC m=+627.564725439) (total time: 9.949021681s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.949021681s Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-5-f52e9ef0: context canceled" createRevision=624037756 key=/registry/leases/kube-node-lease/prod-agent-data-green-5-f52e9ef0 previousRevision=653344682 Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.999211 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.999247 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:43.046061908 +0000 UTC m=+627.560834600) (total time: 9.953322808s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.953322808s Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-4-96deb91b: timeout: context deadline exceeded" createRevision=624037787 key=/registry/leases/kube-node-lease/prod-agent-data-green-4-96deb91b previousRevision=653344689 Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:52Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:52.999825 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.000013 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.000039 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 187.017µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.000298 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.000542 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.039597ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-7-c3860444" result: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.001076 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:43.053818115 +0000 UTC m=+627.568590821) (total time: 9.947317439s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.947317439s Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-7-93605edd: timeout: context deadline exceeded" createRevision=624037614 key=/registry/leases/kube-node-lease/prod-agent-data-green-7-93605edd previousRevision=653344691 Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:43.04928618 +0000 UTC m=+627.564058884) (total time: 9.95209141s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.95209141s Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-5-bb69767d: timeout: context deadline exceeded" createRevision=320959392 key=/registry/leases/kube-node-lease/prod-agent-app-blue-5-bb69767d previousRevision=653344674 Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:43.005610016 +0000 UTC m=+627.520382710) (total time: 9.995881139s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.995881139s Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-6-cd93959b: timeout: context deadline exceeded" createRevision=320959048 key=/registry/leases/kube-node-lease/prod-agent-app-blue-6-cd93959b previousRevision=653344672 Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:43.056900416 +0000 UTC m=+627.571673121) (total time: 9.945857082s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.945857082s Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-1-efd5b590: context canceled" createRevision=623966116 key=/registry/leases/kube-node-lease/prod-agent-data-green-1-efd5b590 previousRevision=653344657 Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:43.055030148 +0000 UTC m=+627.569802890) (total time: 9.947425015s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.947425015s Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-0-9b98de05: timeout: context deadline exceeded" createRevision=623966046 key=/registry/leases/kube-node-lease/prod-agent-data-green-0-9b98de05 previousRevision=653344666 Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.001219 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 4.856µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.003291 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 2.186022ms, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.001084 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.001356 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.001403 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.672041ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-5-f52e9ef0" result: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.001763 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.002162 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.002228 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 6.49µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.002835 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.002934 983458 timeout.go:142] post-timeout activity - time-elapsed: 7.048277ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ops-blue-0-4b80a6a8" result: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.003204 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.003217 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 3.045µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.003523 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:43.054438238 +0000 UTC m=+627.569210979) (total time: 9.949025432s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.949025432s Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.003573 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.927µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-6-420706e2: timeout: context deadline exceeded" createRevision=624036980 key=/registry/leases/kube-node-lease/prod-agent-data-green-6-420706e2 previousRevision=653344661 Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.003781 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.003806 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 4.284µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.004371 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.004387 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.004415 983458 timeout.go:142] post-timeout activity - time-elapsed: 7.109302ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-8-49a23f58" result: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.004438 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.004442 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.004454 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.004502 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.541885ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-4-96deb91b" result: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.004609 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.004867 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.005464 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.006338 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.006373 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.140725ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-6-cd93959b" result: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.006407 983458 timeout.go:142] post-timeout activity - time-elapsed: 5.232722ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-5-bb69767d" result: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.006416 983458 timeout.go:142] post-timeout activity - time-elapsed: 5.236173ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-7-93605edd" result: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.006433 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.006448 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.006573 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.974796ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-6-420706e2" result: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.008284 983458 timeout.go:142] post-timeout activity - time-elapsed: 6.038778ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-0-9b98de05" result: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.008376 983458 timeout.go:142] post-timeout activity - time-elapsed: 5.291868ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-1-efd5b590" result: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=info msg="Slow SQL (started: 2025-09-09 03:23:48.123155235 +0000 UTC m=+632.637927994) (total time: 4.992520937s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.992520937s Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-operator-resource-lock: timeout: context deadline exceeded" createRevision=706 key=/registry/leases/kube-system/cilium-operator-resource-lock previousRevision=653344739 Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.115946 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.115994 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.116027 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 7.93µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.117730 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:53.117900 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.057041ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result: Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.6.61.141:6443: connect: connection refused" Sep 09 03:23:53 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:53Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.6.61.141:6443: connect: connection refused" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:55.298594 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:55Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:45.304022304 +0000 UTC m=+629.818795022) (total time: 9.994441174s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.994441174s Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:55.298725 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:55Z" level=error msg="insert error for key /registry/configmaps/ops/datadog-leader-election: timeout: context deadline exceeded" createRevision=4206015 key=/registry/configmaps/ops/datadog-leader-election previousRevision=653344733 Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:55.298672 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 12.143µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:55Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:55.300781 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:55.300938 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.417189ms, PUT "/api/v1/namespaces/ops/configmaps/datadog-leader-election" result: Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:55.988026 983458 kubelet_node_status.go:544] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"NetworkUnavailable\\\"},{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:23:45Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:23:45Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:23:45Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:23:45Z\\\",\\\"type\\\":\\\"Ready\\\"}]}}\" for node \"prod-server-blue-k3s-01\": Patch \"https://127.0.0.1:6443/api/v1/nodes/prod-server-blue-k3s-01/status?timeout=10s\": context deadline exceeded" Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:55Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:45.997562365 +0000 UTC m=+630.512335057) (total time: 9.990537827s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.990537827s Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:55Z" level=error msg="insert error for key /registry/minions/prod-server-blue-k3s-01: timeout: context deadline exceeded" createRevision=320940137 key=/registry/minions/prod-server-blue-k3s-01 previousRevision=653341971 Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:55.988749 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.551µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:55.988794 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:55Z" level=error msg="error in txn compare: success:\\n6volumes.kubernetes.io/controller-managed-attach-detach\\022\\004truer\\027wrangler.cattle.io/node\\212\\001\\232\\001\\n'k3s-supervisor@prod-server-green-k3s-01\\022\\006Update\\032\\002v1\\\"\\010\\010\\302\\234\\302\\266\\006\\020\\0002\\010FieldsV1:M\\nK{\\\"f:metadata\\\":{\\\"f:finalizers\\\":{\\\".\\\":{},\\\"v:\\\\\\\"wrangler.cattle.io/node\\\\\\\"\\\":{}}}}B\\000\\212\\001\\276\\001\\n&k3s-supervisor@prod-server-blue-k3s-01\\022\\006Update\\032\\002v1\\\"\\010\\010\\303\\234\\302\\266\\006\\020\\0002\\010FieldsV1:r\\np{\\\"f:metadata\\\":{\\\"f:labels\\\":{\\\"f:node-role.kubernetes.io/control-plane\\\":{},\\\"f:node-role.kubernetes.io/master\\\":{}}}}B\\000\\212\\001\\366\\001\\n\\016Go-http-client\\022\\006Update\\032\\002v1\\\"\\010\\010\\276\\235\\302\\266\\006\\020\\0002\\010FieldsV1:\\273\\001\\n\\270\\001{\\\"f:status\\\":{\\\"f:conditions\\\":{\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"NetworkUnavailable\\\\\\\"}\\\":{\\\".\\\":{},\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{},\\\"f:type\\\":{}}}}}B\\006status\\212\\001\\350\\005\\n\\003k3s\\022\\006Update\\032\\002v1\\\"\\010\\010\\352\\357\\373\\305\\006\\020\\0002\\010FieldsV1:\\276\\005\\n\\273\\005{\\\"f:metadata\\\":{\\\"f:annotations\\\":{\\\".\\\":{},\\\"f:alpha.kubernetes.io/provided-node-ip\\\":{},\\\"f:k3s.io/hostname\\\":{},\\\"f:k3s.io/internal-ip\\\":{},\\\"f:k3s.io/node-args\\\":{},\\\"f:k3s.io/node-config-hash\\\":{},\\\"f:k3s.io/node-env\\\":{},\\\"f:node.alpha.kubernetes.io/ttl\\\":{},\\\"f:volumes.kubernetes.io/controller-managed-attach-detach\\\":{}},\\\"f:labels\\\":{\\\".\\\":{},\\\"f:beta.kubernetes.io/arch\\\":{},\\\"f:beta.kubernetes.io/instance-type\\\":{},\\\"f:beta.kubernetes.io/os\\\":{},\\\"f:cluster-name\\\":{},\\\"f:kubernetes.io/arch\\\":{},\\\"f:kubernetes.io/hostname\\\":{},\\\"f:kubernetes.io/os\\\":{},\\\"f:node.kubernetes.io/controller\\\":{},\\\"f:node.kubernetes.io/instance-type\\\":{}}},\\\"f:spec\\\":{\\\"f:podCIDR\\\":{},\\\"f:podCIDRs\\\":{\\\".\\\":{},\\\"v:\\\\\\\"10.128.38.0/24\\\\\\\"\\\":{}},\\\"f:providerID\\\":{}}}B\\000\\212\\001Y\\n\\027cilium-operator-generic\\022\\006Update\\032\\002v1\\\"\\010\\010\\204\\360\\373\\305\\006\\020\\0002\\010FieldsV1:\\034\\n\\032{\\\"f:spec\\\":{\\\"f:taints\\\":{}}}B\\000\\212\\001\\211\\006\\n\\003k3s\\022\\006Update\\032\\002v1\\\"\\010\\010\\301\\273\\376\\305\\006\\020\\0002\\010FieldsV1:\\331\\005\\n\\326\\005{\\\"f:status\\\":{\\\"f:allocatable\\\":{\\\"f:ephemeral-storage\\\":{}},\\\"f:capacity\\\":{\\\"f:ephemeral-storage\\\":{}},\\\"f:conditions\\\":{\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"DiskPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"MemoryPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"PIDPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"Ready\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{}}},\\\"f:images\\\":{},\\\"f:nodeInfo\\\":{\\\"f:containerRuntimeVersion\\\":{},\\\"f:kubeProxyVersion\\\":{},\\\"f:kubeletVersion\\\":{}}}}B\\006status\\022z\\n\\01610.128.38.0/24\\022\\000\\032\\035k3s://prod-server-blue-k3s-01 \\000*5\\n%node-role.kubernetes.io/control-plane\\022\\000\\032\\nNoSchedule:\\01610.128.38.0/24\\032\\253\\030\\n\\n\\n\\003cpu\\022\\003\\n\\0018\\n\\\"\\n\\021ephemeral-storage\\022\\r\\n\\013100951636Ki\\n\\024\\n\\rhugepages-1Gi\\022\\003\\n\\0010\\n\\024\\n\\rhugepages-2Mi\\022\\003\\n\\0010\\n\\026\\n\\006memory\\022\\014\\n\\n12243396Ki\\n\\r\\n\\004pods\\022\\005\\n\\003110\\022\\n\\n\\003cpu\\022\\003\\n\\0018\\022\\\"\\n\\021ephemeral-storage\\022\\r\\n\\01398205751424\\022\\024\\n\\rhugepages-1Gi\\022\\003\\n\\0010\\022\\024\\n\\rhugepages-2Mi\\022\\003\\n\\0010\\022\\026\\n\\006memory\\022\\014\\n\\n12243396Ki\\022\\r\\n\\004pods\\022\\005\\n\\003110\\032\\000\\\"[\\n\\022NetworkUnavailable\\022\\005False\\032\\010\\010\\276\\235\\302\\266\\006\\020\\000\\\"\\010\\010\\276\\235\\302\\266\\006\\020\\000*\\nCiliumIsUp2\\036Cilium is running on this node\\\"p\\n\\016MemoryPressure\\022\\005False\\032\\010\\010\\301\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\331\\370\\372\\305\\006\\020\\000*\\032KubeletHasSufficientMemory2'kubelet has sufficient memory available\\\"a\\n\\014DiskPressure\\022\\005False\\032\\010\\010\\301\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\331\\370\\372\\305\\006\\020\\000*\\030KubeletHasNoDiskPressure2\\034kubelet has no disk pressure\\\"g\\n\\013PIDPressure\\022\\005False\\032\\010\\010\\301\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\331\\370\\372\\305\\006\\020\\000*\\027KubeletHasSufficientPID2$kubelet has sufficient PID available\\\"P\\n\\005Ready\\022\\004True\\032\\010\\010\\301\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\331\\370\\372\\305\\006\\020\\000*\\014KubeletReady2\\037kubelet is posting ready status*\\031\\n\\nInternalIP\\022\\01310.6.61.142*#\\n\\010Hostname\\022\\027prod-server-blue-k3s-012\\005\\n\\003\\010\\212P:\\334\\001\\n 930d76008d2c408e9ba36a050eb87a99\\022$4ed00a42-0756-e0b3-96f9-addb38b101a8\\032$d55d8d1f-c315-4039-a84d-e6ace7b1837c\\\"\\0225.15.0-119-generic*\\022Ubuntu 22.04.4 LTS2\\030containerd://1.7.27-k3s1:\\rv1.30.14+k3s2B\\rv1.30.14+k3s2J\\005linuxR\\005amd64B\\225\\001\\ndpublic.ecr.aws/datadog/agent@sha256:465da828947d6c055eacf399df8d0f96423f6e029fd1218a806f091a0dc0200b\\n'public.ecr.aws/datadog/agent:7.62.0-jmx\\020\\207\\330\\325\\347\\001B\\334\\001\\n}868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops@sha256:307c60e7c0a570a7bbad2a45e1315f5ec6c39f6aea2b3d7dc521b57219fd73f1\\nU868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops:docker-wazuh-agent-f8b1f9f-main\\020\\311\\253\\321\\342\\001B\\334\\001\\n}868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops@sha256:e88bf7e606df57a9e7c3efa1f489132a72ad76f981bf7419e9598ce83ff9005c\\nU868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops:docker-wazuh-agent-a62b8c9-main\\020\\323\\304\\323\\335\\001B\\204\\001\\n]quay.io/cilium/cilium@sha256:9dc74ba5321c999e498b5f05202c7e27015360dd19278f19b15a25bee79d22f1\\n\\036quay.io/cilium/cilium:v1.13.18\\020\\341\\201\\210_B\\203\\001\\n]quay.io/cilium/cilium@sha256:5a94b561f4651fcfd85970a50bc78b201cfbd6e2ab1a03848eab25a82832653a\\n\\035quay.io/cilium/cilium:v1.14.0\\020\\300\\317\\204WBd\\n]quay.io/cilium/cilium@sha256:6544a3441b086a2e09005d3e21d1a4afb216fae19c5a60b35793c8a9438f8f68\\020\\260\\375\\356RB\\227\\001\\ngquay.io/cilium/operator-generic@sha256:3014d4bcb8352f0ddef90fa3b5eb1bbf179b91024813a90a0066eb4517ba93c9\\n'quay.io/cilium/operator-generic:v1.14.0\\020\\270\\377\\240\\013B\\254\\001\\nrdocker.io/rancher/mirrored-coredns-coredns@sha256:9eb4929fd181541e7743959884000f010bcc8b1ebf8715ff0965be04509882e5\\n1docker.io/rancher/mirrored-coredns-coredns:1.12.1\\020\\304\\233\\326\\nBn\\ngquay.io/cilium/operator-generic@sha256:4b58d5b33e53378355f6e8ceb525ccf938b7b6f5384b35373f1f46787467ebf5\\020\\312\\372\\306\\nB\\252\\001\\nqdocker.io/rancher/mirrored-metrics-server@sha256:dccf8474fb910fef261d31d9483d7e4c1df7b86cf4d638fb6a7d7c88bd51600a\\n0docker.io/rancher/mirrored-metrics-server:v0.7.2\\020\\252\\352\\245\\tB\\254\\001\\nrdocker.io/rancher/mirrored-coredns-coredns@sha256:a11fafae1f8037cbbd66c5afa40ba2423936b72b4fd50a7034a7e8b955163594\\n1docker.io/rancher/mirrored-coredns-coredns:1.10.1\\020\\271\\225\\334\\007B\\230\\001\\nhquay.io/prometheus/node-exporter@sha256:4032c6d5bfd752342c3e631c2f1de93ba6b86c41db6b167b9a35372c139e7706\\n'quay.io/prometheus/node-exporter:v1.8.2\\020\\370\\371\\336\\005B\\224\\001\\nhdocker.io/rancher/mirrored-pause@sha256:74c4244427b7312c5b901fe0f67cbc53683d06f4f24c6faee65d4182bf0fa893\\n$docker.io/rancher/mirrored-pause:3.6\\020\\227\\263\\022\\032\\000\\\"\\000\" > > failure: > : context deadline exceeded" Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:55.990203 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:55.990275 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:55 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:55.991571 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.927413ms, PATCH "/api/v1/nodes/prod-server-blue-k3s-01/status" result: Sep 09 03:23:56 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:56Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:56 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:56Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.6.61.141:6443: connect: connection refused" Sep 09 03:23:56 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:56Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.6.61.141:6443: connect: connection refused" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:57Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:47.62761739 +0000 UTC m=+632.142390090) (total time: 9.989954288s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.989954288s Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:57Z" level=error msg="insert error for key /registry/minions/prod-agent-app-blue-4-79b5072b: context canceled" createRevision=320959816 key=/registry/minions/prod-agent-app-blue-4-79b5072b previousRevision=653341980 Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:57.617779 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:57.617826 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 14.017µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:57.619647 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:57.619734 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:57.621255 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.667029ms, PATCH "/api/v1/nodes/prod-agent-app-blue-4-79b5072b/status" result: Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:57Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:47.949461292 +0000 UTC m=+632.464234007) (total time: 9.985783445s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.985783445s Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:57.935407 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:57.935469 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.229µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:57Z" level=error msg="insert error for key /registry/minions/prod-agent-data-green-5-f52e9ef0: timeout: context deadline exceeded" createRevision=624037426 key=/registry/minions/prod-agent-data-green-5-f52e9ef0 previousRevision=653342072 Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:57.935481 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:57Z" level=error msg="error in txn compare: success:\\n6volumes.kubernetes.io/controller-managed-attach-detach\\022\\004truer\\027wrangler.cattle.io/node\\212\\001\\231\\001\\n&k3s-supervisor@prod-server-blue-k3s-01\\022\\006Update\\032\\002v1\\\"\\010\\010\\207\\263\\314\\304\\006\\020\\0002\\010FieldsV1:M\\nK{\\\"f:metadata\\\":{\\\"f:finalizers\\\":{\\\".\\\":{},\\\"v:\\\\\\\"wrangler.cattle.io/node\\\\\\\"\\\":{}}}}B\\000\\212\\001\\267\\006\\n\\003k3s\\022\\006Update\\032\\002v1\\\"\\010\\010\\257\\263\\314\\304\\006\\020\\0002\\010FieldsV1:\\215\\006\\n\\212\\006{\\\"f:metadata\\\":{\\\"f:annotations\\\":{\\\".\\\":{},\\\"f:alpha.kubernetes.io/provided-node-ip\\\":{},\\\"f:k3s.io/hostname\\\":{},\\\"f:k3s.io/internal-ip\\\":{},\\\"f:k3s.io/node-args\\\":{},\\\"f:k3s.io/node-config-hash\\\":{},\\\"f:k3s.io/node-env\\\":{},\\\"f:node.alpha.kubernetes.io/ttl\\\":{},\\\"f:volumes.kubernetes.io/controller-managed-attach-detach\\\":{}},\\\"f:labels\\\":{\\\".\\\":{},\\\"f:beta.kubernetes.io/arch\\\":{},\\\"f:beta.kubernetes.io/instance-type\\\":{},\\\"f:beta.kubernetes.io/os\\\":{},\\\"f:cluster-name\\\":{},\\\"f:failure-domain.beta.kubernetes.io/zone\\\":{},\\\"f:kubernetes.io/arch\\\":{},\\\"f:kubernetes.io/hostname\\\":{},\\\"f:kubernetes.io/os\\\":{},\\\"f:node-group\\\":{},\\\"f:node-role\\\":{},\\\"f:node.kubernetes.io/instance-type\\\":{},\\\"f:topology.kubernetes.io/zone\\\":{}}},\\\"f:spec\\\":{\\\"f:podCIDR\\\":{},\\\"f:podCIDRs\\\":{\\\".\\\":{},\\\"v:\\\\\\\"10.128.36.0/24\\\\\\\"\\\":{}},\\\"f:providerID\\\":{}}}B\\000\\212\\001\\377\\001\\n\\027cilium-operator-generic\\022\\006Update\\032\\002v1\\\"\\010\\010\\263\\263\\314\\304\\006\\020\\0002\\010FieldsV1:\\273\\001\\n\\270\\001{\\\"f:status\\\":{\\\"f:conditions\\\":{\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"NetworkUnavailable\\\\\\\"}\\\":{\\\".\\\":{},\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{},\\\"f:type\\\":{}}}}}B\\006status\\212\\001Y\\n\\027cilium-operator-generic\\022\\006Update\\032\\002v1\\\"\\010\\010\\375\\277\\371\\305\\006\\020\\0002\\010FieldsV1:\\034\\n\\032{\\\"f:spec\\\":{\\\"f:taints\\\":{}}}B\\000\\212\\001\\273\\004\\n\\003k3s\\022\\006Update\\032\\002v1\\\"\\010\\010\\303\\273\\376\\305\\006\\020\\0002\\010FieldsV1:\\213\\004\\n\\210\\004{\\\"f:metadata\\\":{\\\"f:annotations\\\":{\\\"f:csi.volume.kubernetes.io/nodeid\\\":{}}},\\\"f:status\\\":{\\\"f:addresses\\\":{\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"Hostname\\\\\\\"}\\\":{\\\"f:address\\\":{}}},\\\"f:conditions\\\":{\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"DiskPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"MemoryPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"PIDPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"Ready\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{}}},\\\"f:images\\\":{},\\\"f:volumesAttached\\\":{},\\\"f:volumesInUse\\\":{}}}B\\006status\\022k\\n\\01610.128.36.0/24\\022\\000\\032&k3s://prod-agent-data-green-5-f52e9ef0 \\000*\\035\\n\\tnode-role\\022\\004data\\032\\nNoSchedule:\\01610.128.36.0/24\\032\\3046\\n\\013\\n\\003cpu\\022\\004\\n\\00216\\n!\\n\\021ephemeral-storage\\022\\014\\n\\n29751268Ki\\n\\024\\n\\rhugepages-1Gi\\022\\003\\n\\0010\\n\\024\\n\\rhugepages-2Mi\\022\\003\\n\\0010\\n\\026\\n\\006memory\\022\\014\\n\\n32860604Ki\\n\\r\\n\\004pods\\022\\005\\n\\003110\\022\\013\\n\\003cpu\\022\\004\\n\\00216\\022\\\"\\n\\021ephemeral-storage\\022\\r\\n\\01328942033488\\022\\024\\n\\rhugepages-1Gi\\022\\003\\n\\0010\\022\\024\\n\\rhugepages-2Mi\\022\\003\\n\\0010\\022\\026\\n\\006memory\\022\\014\\n\\n32860604Ki\\022\\r\\n\\004pods\\022\\005\\n\\003110\\032\\000\\\"[\\n\\022NetworkUnavailable\\022\\005False\\032\\010\\010\\263\\263\\314\\304\\006\\020\\000\\\"\\010\\010\\263\\263\\314\\304\\006\\020\\000*\\nCiliumIsUp2\\036Cilium is running on this node\\\"p\\n\\016MemoryPressure\\022\\005False\\032\\010\\010\\303\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\206\\263\\314\\304\\006\\020\\000*\\032KubeletHasSufficientMemory2'kubelet has sufficient memory available\\\"a\\n\\014DiskPressure\\022\\005False\\032\\010\\010\\303\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\206\\263\\314\\304\\006\\020\\000*\\030KubeletHasNoDiskPressure2\\034kubelet has no disk pressure\\\"g\\n\\013PIDPressure\\022\\005False\\032\\010\\010\\303\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\206\\263\\314\\304\\006\\020\\000*\\027KubeletHasSufficientPID2$kubelet has sufficient PID available\\\"P\\n\\005Ready\\022\\004True\\032\\010\\010\\303\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\255\\263\\314\\304\\006\\020\\000*\\014KubeletReady2\\037kubelet is posting ready status*\\031\\n\\nInternalIP\\022\\01310.6.61.246*#\\n\\010Hostname\\022\\027prod-agent-data-green-52\\005\\n\\003\\010\\212P:\\334\\001\\n 930d76008d2c408e9ba36a050eb87a99\\022$95b20a42-2616-9e05-2b78-886563194654\\032$0fabacee-f579-4a6f-87a4-b52ec2614dd9\\\"\\0225.15.0-119-generic*\\022Ubuntu 22.04.4 LTS2\\030containerd://1.7.27-k3s1:\\rv1.30.14+k3s2B\\rv1.30.14+k3s2J\\005linuxR\\005amd64B\\220\\001\\nddocker.io/percona/pmm-server@sha256:7e2552a8628e8e83bfb88ae6aa0496d68b5e6d0b50b0457788a1d02ad3c9d736\\n\\\"docker.io/percona/pmm-server:3.0.0\\020\\306\\261\\357\\327\\003B\\225\\001\\ndpublic.ecr.aws/datadog/agent@sha256:465da828947d6c055eacf399df8d0f96423f6e029fd1218a806f091a0dc0200b\\n'public.ecr.aws/datadog/agent:7.62.0-jmx\\020\\207\\330\\325\\347\\001B\\334\\001\\n}868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops@sha256:e88bf7e606df57a9e7c3efa1f489132a72ad76f981bf7419e9598ce83ff9005c\\nU868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops:docker-wazuh-agent-a62b8c9-main\\020\\323\\304\\323\\335\\001B\\252\\002\\n\\254\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/tf-serving-kira-native-cpu/onprem@sha256:71a22b984b8a609cfadf3e1796e991ededa0c1e26985d83d14a161b1a6459e5b\\nsus-central1-docker.pkg.dev/document-verification-public/docver-gcm/tf-serving-kira-native-cpu/onprem:v2.13.0-3.15.1\\020\\311\\375\\253\\207\\001B\\203\\001\\n]quay.io/cilium/cilium@sha256:5a94b561f4651fcfd85970a50bc78b201cfbd6e2ab1a03848eab25a82832653a\\n\\035quay.io/cilium/cilium:v1.14.0\\020\\300\\317\\204WB\\242\\002\\n\\254\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/tf-serving-kira-native-cpu/onprem@sha256:4bfcd319098ec6d0f16346222fa2114bcdc2efcb15c58bf4e5cb6ed86564bb4d\\nlus-central1-docker.pkg.dev/document-verification-public/docver-gcm/tf-serving-kira-native-cpu/onprem:v2.13.0\\020\\304\\233\\302IB\\244\\002\\n\\256\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/tritonserver-cpu-onnxruntime/onprem@sha256:67f677d008d438602881b950dc2309b774d643affc0d2e0aeabdc91d6ff76c12\\nlus-central1-docker.pkg.dev/document-verification-public/docver-gcm/tritonserver-cpu-onnxruntime/onprem:23.06\\020\\212\\202\\305HB\\212\\002\\n\\241\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-visual-anomaly@sha256:1e0a60549563c9742089a7c370dada22d3672611b71b6a833e7429b4f954604a\\n_us-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-visual-anomaly:1.2.1\\020\\305\\325\\227=B\\222\\001\\nequay.io/hpestorage/csi-driver@sha256:72c2e0477e750ea7301fa3543f951dd9f03e29aa8b7eba5e3084b2f1cc06c997\\n$quay.io/hpestorage/csi-driver:v2.5.2\\020\\201\\315\\3575B\\212\\002\\n\\241\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-visual-anomaly@sha256:db8f6ffd8c56a712eb82061eb29bae17a303ba5cc7ff72b1d91e64432827e4ae\\n_us-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-visual-anomaly:1.3.0\\020\\323\\311\\2754B\\236\\001\\nddocker.io/bitnami/postgresql@sha256:aedda040a9d1b70df4b76cfb237dcf8f913afde5c41b998c61f8e6a0a7f1ee3a\\n1docker.io/bitnami/postgresql:15.2.0-debian-11-r30\\020\\331\\373\\247+B\\373\\001\\n\\231\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-runner@sha256:f4b783b60f9396bfd22a6b2249a060cfe59a2afb94d2103d563a87f5da7d5c96\\nXus-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-runner:3.15.0\\020\\366\\212\\352#B\\214\\002\\n\\241\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/embedding-store/onprem@sha256:2ccc03f7d575a928f879cf207ba789ad5a4eedd5a36afc643fba4dc9787e5d78\\naus-central1-docker.pkg.dev/document-verification-public/docver-gcm/embedding-store/onprem:v0.32.5\\020\\215\\266\\372 B\\372\\001\\n\\231\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-runner@sha256:0a172f4a12738999fcaa76df7082f2d1c966736d30f2f202a2a85b9d87f997cc\\nWus-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-runner:3.4.0\\020\\322\\226\\314 B\\214\\002\\n\\241\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/embedding-store/onprem@sha256:7fbbbacc4d025aab8664ddebcbbb32e5c16f85a8d57c43da0a86d30859b8a734\\naus-central1-docker.pkg.dev/document-verification-public/docver-gcm/embedding-store/onprem:v0.31.1\\020\\345\\260\\254\\036B\\251\\001\\njdocker.io/fabiocicerchia/nginx-lua@sha256:5bfbce1515934a8a085355379dd592d38dac3ee5f790971dd3ac2e53436cb685\\n6docker.io/fabiocicerchia/nginx-lua:1.29.1-alpine3.22.1\\020\\342\\320\\341\\026B\\214\\002\\n\\241\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/mlp-model-proxy/onprem@sha256:57298d3755aeced6e97f8277efadc95aac9f2b9a71139d70fdbef3bb8ddeb404\\naus-central1-docker.pkg.dev/document-verification-public/docver-gcm/mlp-model-proxy/onprem:v0.25.2\\020\\264\\222\\245\\026B\\252\\002\\n\\260\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/anomaly-detection-intermediary/onprem@sha256:3e50d482fa1d401206664fd29f7d0c15f254a64ed5cb1fcd150aa43dd7ad5b84\\npus-central1-docker.pkg.dev/document-verification-public/docver-gcm/anomaly-detection-intermediary/onprem:v0.11.6\\020\\362\\205\\353\\024B\\251\\001\\njdocker.io/fabiocicerchia/nginx-lua@sha256:e21edef393d8a114ffd8d33292520aa960638cfe36946b09118290655b8acc61\\n6docker.io/fabiocicerchia/nginx-lua:1.25.2-alpine3.18.3\\020\\204\\271\\310\\024B\\214\\002\\n\\241\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/mlp-model-proxy/onprem@sha256:dfee307564fcd5cf7c856c1da56d9046248633e91c32b96cd818fa8a1f60e0f4\\naus-central1-docker.pkg.dev/document-verification-public/docver-gcm/mlp-model-proxy/onprem:v0.19.2\\020\\320\\351\\330\\023B\\375\\001\\n\\232\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-doc-ver@sha256:cca2da3c9f11afd33acaba8aaf51db636e1d0d4d8cb5f62e5b904d04c4139e23\\nYus-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-doc-ver:3.15.0\\020\\243\\350\\354\\022B\\374\\001\\n\\232\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-doc-ver@sha256:eccf730b6a51a23c71f0d52e10cf2f92e3831a89e0bd89afb0345d8340e7363d\\nXus-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-doc-ver:3.4.0\\020\\231\\336\\313\\022B\\201\\001\\n_docker.io/rclone/rclone@sha256:fd635aecd9667ee3c3bf920d14118090d4f2a83a080c1fa77e0bafbd4587ca87\\n\\031docker.io/rclone/rclone:1\\020\\365\\203\\260\\rB\\303\\001\\n}registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:d7138bcc3aa5f267403d45ad4292c95397e421ea17a0035888850f424c7de25d\\n=registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0\\020\\277\\230\\206\\007B\\230\\001\\nhquay.io/prometheus/node-exporter@sha256:4032c6d5bfd752342c3e631c2f1de93ba6b86c41db6b167b9a35372c139e7706\\n'quay.io/prometheus/node-exporter:v1.8.2\\020\\370\\371\\336\\005B\\224\\001\\nhdocker.io/rancher/mirrored-pause@sha256:74c4244427b7312c5b901fe0f67cbc53683d06f4f24c6faee65d4182bf0fa893\\n$docker.io/rancher/mirrored-pause:3.6\\020\\227\\263\\022JHkubernetes.io/csi/csi.hpe.com^06766d9dd0b83f1e91000000000000000000000042RL\\nHkubernetes.io/csi/csi.hpe.com^06766d9dd0b83f1e91000000000000000000000042\\022\\000\\032\\000\\\"\\000\" > > failure: > : context deadline exceeded" Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:57.937056 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:57 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:57.937233 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.038804ms, PATCH "/api/v1/nodes/prod-agent-data-green-5-f52e9ef0/status" result: Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:59.115264 983458 leaderelection.go:383] Failed to update lock: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": context deadline exceeded Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:59Z" level=info msg="Slow SQL (started: 2025-09-09 03:23:54.122646916 +0000 UTC m=+638.637419631) (total time: 4.992880239s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.992880239s Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:59Z" level=error msg="insert error for key /registry/leases/kube-system/kube-scheduler: context canceled" createRevision=387 key=/registry/leases/kube-system/kube-scheduler previousRevision=653344736 Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:59.115724 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:59.115771 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:59.115805 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.85µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:59.117591 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:59.117862 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.026363ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result: Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:59.706263 983458 leaderelection.go:383] Failed to update lock: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/k3s-cloud-controller-manager?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers) Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:59.706617 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:59.706661 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:59.706720 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 11.551µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:59Z" level=info msg="Slow SQL (started: 2025-09-09 03:23:54.710538728 +0000 UTC m=+639.225311410) (total time: 4.996412092s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.996412092s Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:59Z" level=error msg="insert error for key /registry/leases/kube-system/k3s-cloud-controller-manager: timeout: context deadline exceeded" createRevision=368 key=/registry/leases/kube-system/k3s-cloud-controller-manager previousRevision=653344738 Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:59Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:59.708363 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:59.708517 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.04835ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/k3s-cloud-controller-manager" result: Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:59Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:59Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.6.61.141:6443: connect: connection refused" Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:59Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.6.61.141:6443: connect: connection refused" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:59Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:25.808564231 +0000 UTC m=+610.323336991) (total time: 33.996377856s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.996377856s Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:59Z" level=error msg="insert error for key /registry/leases/kube-system/apiserver-sbiwrpumfte5iyfxhikdbh2e3u: timeout: context deadline exceeded" createRevision=320940109 key=/registry/leases/kube-system/apiserver-sbiwrpumfte5iyfxhikdbh2e3u previousRevision=653344687 Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:23:59Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:59.805302 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 4.654µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:23:59 prod-server-blue-k3s-01 k3s[983458]: E0909 03:23:59.805706 983458 controller.go:195] "Failed to update lease" err="Timeout: request did not complete within requested timeout - context deadline exceeded" Sep 09 03:24:00 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:00Z" level=info msg="Slow SQL (started: 2025-09-09 03:23:55.69870373 +0000 UTC m=+640.213476448) (total time: 4.993054365s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.993054365s Sep 09 03:24:00 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:00Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-operator-resource-lock: timeout: context deadline exceeded" createRevision=706 key=/registry/leases/kube-system/cilium-operator-resource-lock previousRevision=653344739 Sep 09 03:24:00 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:00.691931 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:00 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:00.692149 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:00 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:00Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:00 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:00.691975 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 11.548µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:00 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:00.693854 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:00 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:00.694046 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.162854ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result: Sep 09 03:24:01 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:01.817769 983458 controller.go:195] "Failed to update lease" err="Put \"https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-server-blue-k3s-01?timeout=10s\": context deadline exceeded" Sep 09 03:24:01 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:01.818096 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:01 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:01.818149 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:01 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:01.818188 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 10.406µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:01 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:01Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:51.826151174 +0000 UTC m=+636.340923905) (total time: 9.991966951s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.991966951s Sep 09 03:24:01 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:01Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-server-blue-k3s-01: context canceled" createRevision=320940478 key=/registry/leases/kube-node-lease/prod-server-blue-k3s-01 previousRevision=653344634 Sep 09 03:24:01 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:01.819940 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:01 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:01.820102 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.122087ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-server-blue-k3s-01" result: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:52.832780635 +0000 UTC m=+637.347553335) (total time: 9.952530985s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.952530985s Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.785685 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 15.343µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.785895 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.786040 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 24.748µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.786104 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 5.822µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.786203 983458 wrap.go:54] timeout or abort while handling: method=PUT URI="/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-4-79b5072b?timeout=10s" audit-ID="ea72cd85-1204-451e-9113-46ce01eed79b" Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.786259 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.62µs, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-4-79b5072b" result: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.786516 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 6.175µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.786537 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.786805 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.786813 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.786860 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 6.453µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.786926 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.788298 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.788358 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.788755 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.788920 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.788965 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 10.509µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.789694 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.789774 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 50.038µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.789800 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.790030 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.790058 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.790092 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.790129 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.769989ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ingress-blue-2-39068466" result: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.790327 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 3.514µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.790708 983458 wrap.go:54] timeout or abort while handling: method=PUT URI="/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-0-f0380b7a?timeout=10s" audit-ID="20358823-e6f1-4d51-9bda-29f4abbcd626" Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.790747 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.67µs, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-0-f0380b7a" result: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.791144 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 503.053µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.791256 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.791449 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 5.311µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.791479 983458 timeout.go:142] post-timeout activity - time-elapsed: 5.898053ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ops-blue-1-89a427cc" result: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.791701 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.791707 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.791721 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.791736 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.791801 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.792346 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.792552 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ops-blue-1-89a427cc: timeout: context deadline exceeded" createRevision=321006889 key=/registry/leases/kube-node-lease/prod-agent-ops-blue-1-89a427cc previousRevision=653344659 Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:52.805311222 +0000 UTC m=+637.320083924) (total time: 9.980393761s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.980393761s Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-3-aef87d06: context canceled" createRevision=320960233 key=/registry/leases/kube-node-lease/prod-agent-app-blue-3-aef87d06 previousRevision=653344685 Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:52.821506479 +0000 UTC m=+637.336279196) (total time: 9.964543478s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.964543478s Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-4-79b5072b: context canceled" createRevision=320960218 key=/registry/leases/kube-node-lease/prod-agent-app-blue-4-79b5072b previousRevision=653344692 Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:52.826102483 +0000 UTC m=+637.340875171) (total time: 9.960151779s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.960151779s Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ingress-blue-2-39068466: context canceled" createRevision=320947107 key=/registry/leases/kube-node-lease/prod-agent-ingress-blue-2-39068466 previousRevision=653344680 Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:52.840078571 +0000 UTC m=+637.354851253) (total time: 9.946570705s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.946570705s Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ingress-blue-0-ac9168ff: context canceled" createRevision=320947169 key=/registry/leases/kube-node-lease/prod-agent-ingress-blue-0-ac9168ff previousRevision=653344686 Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:52.82859356 +0000 UTC m=+637.343366238) (total time: 9.959545322s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.959545322s Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-10-1ef6ca3c: timeout: context deadline exceeded" createRevision=613356118 key=/registry/leases/kube-node-lease/prod-agent-app-blue-10-1ef6ca3c previousRevision=653344688 Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.793287 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:52.836416872 +0000 UTC m=+637.351189559) (total time: 9.953789182s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.953789182s Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.793356 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-1-a234b216: context canceled" createRevision=320959216 key=/registry/leases/kube-node-lease/prod-agent-app-blue-1-a234b216 previousRevision=653344667 Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:52.838544196 +0000 UTC m=+637.353316925) (total time: 9.951890675s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.951890675s Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.793406 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.101608ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-1-a234b216" result: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-9-63cdf3ff: context canceled" createRevision=613356165 key=/registry/leases/kube-node-lease/prod-agent-app-blue-9-63cdf3ff previousRevision=653344696 Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:52.835753491 +0000 UTC m=+637.350526192) (total time: 9.95512294s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.95512294s Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.793517 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.527894ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-10-1ef6ca3c" result: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ingress-blue-1-4323cca2: timeout: context deadline exceeded" createRevision=320946796 key=/registry/leases/kube-node-lease/prod-agent-ingress-blue-1-4323cca2 previousRevision=653344650 Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.793574 983458 timeout.go:142] post-timeout activity - time-elapsed: 7.83246ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-3-aef87d06" result: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.793723 983458 timeout.go:142] post-timeout activity - time-elapsed: 6.956242ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ingress-blue-0-ac9168ff" result: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:52.827023299 +0000 UTC m=+637.341795982) (total time: 9.964430961s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.964430961s Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-0-f0380b7a: context canceled" createRevision=320960246 key=/registry/leases/kube-node-lease/prod-agent-app-blue-0-f0380b7a previousRevision=653344668 Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.794094 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.428216ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ingress-blue-1-4323cca2" result: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.794563 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.256813ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-9-63cdf3ff" result: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.6.61.141:6443: connect: connection refused" Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.6.61.141:6443: connect: connection refused" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:53.006940928 +0000 UTC m=+637.521713640) (total time: 9.989059006s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.989059006s Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-2-9c964185: timeout: context deadline exceeded" createRevision=320960197 key=/registry/leases/kube-node-lease/prod-agent-app-blue-2-9c964185 previousRevision=653344675 Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.996563 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 9.924µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.996924 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 9.266µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.996972 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:53.050332247 +0000 UTC m=+637.565104968) (total time: 9.946572251s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.946572251s Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ops-blue-0-4b80a6a8: context canceled" createRevision=321006849 key=/registry/leases/kube-node-lease/prod-agent-ops-blue-0-4b80a6a8 previousRevision=653344656 Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.997609 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.997869 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:02 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.998000 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.998042 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 130.408µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:53.0309127 +0000 UTC m=+637.545685391) (total time: 9.967231682s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.967231682s Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-7-c3860444: context canceled" createRevision=320959253 key=/registry/leases/kube-node-lease/prod-agent-app-blue-7-c3860444 previousRevision=653344660 Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.998679 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.999062 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.999096 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.999069 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.999115 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.999142 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 6.738µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:53.042638254 +0000 UTC m=+637.557410945) (total time: 9.956469141s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.956469141s Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-3-47319b7a: timeout: context deadline exceeded" createRevision=624036915 key=/registry/leases/kube-node-lease/prod-agent-data-green-3-47319b7a previousRevision=653344684 Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.999726 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.999784 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.229µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:53.045390984 +0000 UTC m=+637.560163665) (total time: 9.954394175s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.954394175s Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:02Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-5-f52e9ef0: timeout: context deadline exceeded" createRevision=624037756 key=/registry/leases/kube-node-lease/prod-agent-data-green-5-f52e9ef0 previousRevision=653344682 Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:02.999867 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:03Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.000283 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.000538 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.000587 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 10.385µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:03Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:53.055116123 +0000 UTC m=+637.569888804) (total time: 9.945425168s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.945425168s Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:03Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-4-96deb91b: context canceled" createRevision=624037787 key=/registry/leases/kube-node-lease/prod-agent-data-green-4-96deb91b previousRevision=653344689 Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:03Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:53.052426466 +0000 UTC m=+637.567199145) (total time: 9.948182203s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.948182203s Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:03Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-5-bb69767d: timeout: context deadline exceeded" createRevision=320959392 key=/registry/leases/kube-node-lease/prod-agent-app-blue-5-bb69767d previousRevision=653344674 Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.000868 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:03Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.000909 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 6.862µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.001076 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.056527ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-3-47319b7a" result: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.001201 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.163407ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-7-c3860444" result: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.001256 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.001382 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.001419 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.001431 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.001439 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 26.623µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.001468 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.66609ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ops-blue-0-4b80a6a8" result: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:03Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:53.046538934 +0000 UTC m=+637.561311616) (total time: 9.954979634s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.954979634s Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:03Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-7-93605edd: context canceled" createRevision=624037614 key=/registry/leases/kube-node-lease/prod-agent-data-green-7-93605edd previousRevision=653344691 Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.001699 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.002515 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.002657 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.002688 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.002787 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.103862ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-5-f52e9ef0" result: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.002825 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.003167 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.003262 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 61.573µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.003450 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.003486 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 4.415µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:03Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:53.051830335 +0000 UTC m=+637.566603029) (total time: 9.951589741s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.951589741s Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:03Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-6-cd93959b: context canceled" createRevision=320959048 key=/registry/leases/kube-node-lease/prod-agent-app-blue-6-cd93959b previousRevision=653344672 Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.003678 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.835732ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-5-bb69767d" result: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.003712 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:03Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:53.059277505 +0000 UTC m=+637.574050226) (total time: 9.944395649s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.944395649s Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:03Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-0-9b98de05: context canceled" createRevision=623966046 key=/registry/leases/kube-node-lease/prod-agent-data-green-0-9b98de05 previousRevision=653344666 Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.003985 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.004017 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.738269ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-4-96deb91b" result: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.004396 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.004476 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 60.509µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:03Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:53.05694127 +0000 UTC m=+637.571713961) (total time: 9.947579692s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.947579692s Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:03Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-6-420706e2: context canceled" createRevision=624036980 key=/registry/leases/kube-node-lease/prod-agent-data-green-6-420706e2 previousRevision=653344661 Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.004676 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.004754 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.004808 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.392027ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-7-93605edd" result: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.005058 983458 timeout.go:142] post-timeout activity - time-elapsed: 9.020364ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-2-9c964185" result: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.005107 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.005793 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.005861 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.497403ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-0-9b98de05" result: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.006134 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.006249 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.877877ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-6-cd93959b" result: Sep 09 03:24:03 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:03.007007 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.511567ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-6-420706e2" result: Sep 09 03:24:05 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:05Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:24:05 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:05Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.6.61.141:6443: connect: connection refused" Sep 09 03:24:05 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:05Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.6.61.141:6443: connect: connection refused" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:06.000417 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:06.000479 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:06.000521 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 9.426µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: W0909 03:24:06.000571 983458 transport.go:356] Unable to cancel request for *otelhttp.Transport Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:06.000748 983458 kubelet_node_status.go:544] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"NetworkUnavailable\\\"},{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:23:55Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:23:55Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:23:55Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:23:55Z\\\",\\\"type\\\":\\\"Ready\\\"}]}}\" for node \"prod-server-blue-k3s-01\": Patch \"https://127.0.0.1:6443/api/v1/nodes/prod-server-blue-k3s-01/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:06Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:56.011459781 +0000 UTC m=+640.526232540) (total time: 9.989659086s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.989659086s Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:06Z" level=error msg="insert error for key /registry/minions/prod-server-blue-k3s-01: context canceled" createRevision=320940137 key=/registry/minions/prod-server-blue-k3s-01 previousRevision=653341971 Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:06.001934 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:06.002139 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.674032ms, PATCH "/api/v1/nodes/prod-server-blue-k3s-01/status" result: Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:06Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:32.921936499 +0000 UTC m=+617.436709175) (total time: 33.996406455s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.996406455s Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:06Z" level=error msg="insert error for key /registry/leases/kube-system/cert-manager-controller: timeout: context deadline exceeded" createRevision=303124872 key=/registry/leases/kube-system/cert-manager-controller previousRevision=653344707 Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:06.918519 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 15.262µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:06Z" level=error msg="error in txn compare: success:cert-manager-7f78bfd948-chmhv-external-cert-manager-controller\\020<\\032\\014\\010\\351\\377\\300\\305\\006\\020\\350\\317\\225\\350\\002\\\"\\013\\010\\264\\273\\376\\305\\006\\020\\260\\274\\235](\\007\\032\\000\\\"\\000\" > > failure: > : context deadline exceeded" Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:06Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:32.928464369 +0000 UTC m=+617.443237046) (total time: 33.993360497s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.993360497s Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:06Z" level=error msg="insert error for key /registry/leases/kube-system/cert-manager-cainjector-leader-election: timeout: context deadline exceeded" createRevision=303128948 key=/registry/leases/kube-system/cert-manager-cainjector-leader-election previousRevision=653344706 Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:06.921995 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 12.79µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:06 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:06Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:07Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.248804517 +0000 UTC m=+617.763577217) (total time: 33.755313113s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.755313113s Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:07Z" level=error msg="insert error for key /registry/events/enablers/oss-data-d7777fcfd-wcwqk.185fd6a024fd24aa: timeout: context deadline exceeded" createRevision=653336628 key=/registry/events/enablers/oss-data-d7777fcfd-wcwqk.185fd6a024fd24aa previousRevision=653336628 Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:07.004335 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 7.963µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:07Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:07Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:33.500991114 +0000 UTC m=+618.015763805) (total time: 33.996651113s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.996651113s Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:07Z" level=error msg="insert error for key /registry/external-secrets.io/externalsecrets/verify/idi-main: timeout: context deadline exceeded" createRevision=324683668 key=/registry/external-secrets.io/externalsecrets/verify/idi-main previousRevision=653344147 Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:07.498338 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 15.619µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:07Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:07.635585 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 16.458µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:07.635614 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:07Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:57.648911662 +0000 UTC m=+642.163684425) (total time: 9.986570824s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.986570824s Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:07Z" level=error msg="insert error for key /registry/minions/prod-agent-app-blue-4-79b5072b: context canceled" createRevision=320959816 key=/registry/minions/prod-agent-app-blue-4-79b5072b previousRevision=653341980 Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:07.637433 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:07.637516 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:07.638688 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.965459ms, PATCH "/api/v1/nodes/prod-agent-app-blue-4-79b5072b/status" result: Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:07.952412 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:07.952474 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:07.952438 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 12.753µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:07Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:57.971924078 +0000 UTC m=+642.486696754) (total time: 9.980467367s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.980467367s Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:07Z" level=error msg="insert error for key /registry/minions/prod-agent-data-green-5-f52e9ef0: context canceled" createRevision=624037426 key=/registry/minions/prod-agent-data-green-5-f52e9ef0 previousRevision=653342072 Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:07.954204 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:07 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:07.954460 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.148751ms, PATCH "/api/v1/nodes/prod-agent-data-green-5-f52e9ef0/status" result: Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.004525 983458 leaderelection.go:383] Failed to update lock: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/k3s-cloud-controller-manager?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers) Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.004663 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.004699 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.004728 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 9.054µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:08Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:03.05352604 +0000 UTC m=+647.568298772) (total time: 4.951097595s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.951097595s Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:08Z" level=error msg="insert error for key /registry/leases/kube-system/k3s-cloud-controller-manager: timeout: context deadline exceeded" createRevision=368 key=/registry/leases/kube-system/k3s-cloud-controller-manager previousRevision=653344738 Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:08Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.006599 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.006799 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.193348ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/k3s-cloud-controller-manager" result: Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:08Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:34.212275421 +0000 UTC m=+618.727048146) (total time: 33.991442463s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.991442463s Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:08Z" level=error msg="insert error for key /registry/events/default/dc-prod-aws-secretsmanager.18479ad829c235bd: context canceled" createRevision=508183431 key=/registry/events/default/dc-prod-aws-secretsmanager.18479ad829c235bd previousRevision=653342075 Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.204258 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 123.14µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.323151 983458 leaderelection.go:383] Failed to update lock: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.323296 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:08Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:03.330445931 +0000 UTC m=+647.845218691) (total time: 4.992764016s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.992764016s Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.323344 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.323343 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 10.612µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:08Z" level=error msg="insert error for key /registry/leases/kube-system/kube-scheduler: context canceled" createRevision=387 key=/registry/leases/kube-system/kube-scheduler previousRevision=653344736 Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.325049 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.325287 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.070274ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result: Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:08Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:08Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.6.61.141:6443: connect: connection refused" Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:08Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.6.61.141:6443: connect: connection refused" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.910988 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.911001 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 15.119µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.911040 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:08Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:03.916399577 +0000 UTC m=+648.431172267) (total time: 4.994711368s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.994711368s Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:08Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-operator-resource-lock: context canceled" createRevision=706 key=/registry/leases/kube-system/cilium-operator-resource-lock previousRevision=653344739 Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.912113 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:08 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:08.912343 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.342835ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result: Sep 09 03:24:09 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:09.625623 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 14.187µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:09 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:09Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:35.657588305 +0000 UTC m=+620.172361031) (total time: 33.967966096s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.967966096s Sep 09 03:24:09 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:09Z" level=error msg="insert error for key /registry/events/encryptor-system/encryptor-controller-manager-58b77b7b56-k7wpj.185fd6a0a1ba8a3c: context canceled" createRevision=653338846 key=/registry/events/encryptor-system/encryptor-controller-manager-58b77b7b56-k7wpj.185fd6a0a1ba8a3c previousRevision=653338846 Sep 09 03:24:09 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:09.632896 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 12.147µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:09 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:09Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:35.642007725 +0000 UTC m=+620.156780460) (total time: 33.99080791s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.99080791s Sep 09 03:24:09 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:09Z" level=error msg="insert error for key /registry/pods/encryptor-system/encryptor-controller-manager-58b77b7b56-k7wpj: context canceled" createRevision=508183156 key=/registry/pods/encryptor-system/encryptor-controller-manager-58b77b7b56-k7wpj previousRevision=653338933 Sep 09 03:24:09 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:09Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:35.962021679 +0000 UTC m=+620.476794416) (total time: 33.994108514s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.994108514s Sep 09 03:24:09 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:09Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-ingress-nginx-secondary-ingress-nginx-secondary-controller: timeout: context deadline exceeded" createRevision=537632387 key=/registry/leases/kube-system/cilium-l2announce-ingress-nginx-secondary-ingress-nginx-secondary-controller previousRevision=653344732 Sep 09 03:24:09 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:09.956305 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 15.891µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:09 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:09Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:10 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:10Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:36.699829316 +0000 UTC m=+621.214602079) (total time: 33.99442884s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.99442884s Sep 09 03:24:10 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:10Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master: timeout: context deadline exceeded" createRevision=537633548 key=/registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master previousRevision=653344740 Sep 09 03:24:10 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:10.694502 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 12.904µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:10 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:10Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:10 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:10Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:36.848268088 +0000 UTC m=+621.363040799) (total time: 33.994522307s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.994522307s Sep 09 03:24:10 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:10.842981 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 17.604µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:10 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:10Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master: context canceled" createRevision=537633548 key=/registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master previousRevision=653344740 Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:37.241157932 +0000 UTC m=+621.755930617) (total time: 33.970050115s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.970050115s Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=error msg="insert error for key /registry/events/kube-system/cilium-operator-6b658bf6bc-qlpsk.18637f45c32c713c: timeout: context deadline exceeded" createRevision=0 key=/registry/events/kube-system/cilium-operator-6b658bf6bc-qlpsk.18637f45c32c713c previousRevision=0 Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:11.211410 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 18.293µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=error msg="error in txn compare: success:Readiness probe failed: HTTP probe failed with statuscode: 500*\\\"\\n\\007kubelet\\022\\027prod-server-blue-k3s-012\\010\\010\\271\\273\\376\\305\\006\\020\\000:\\010\\010\\271\\273\\376\\305\\006\\020\\000@\\001J\\007WarningR\\000b\\000r\\007kubeletz\\027prod-server-blue-k3s-01\\032\\000\\\"\\000\" lease:3660 > > : timeout: context deadline exceeded" Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:11.212525 983458 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{cilium-operator-6b658bf6bc-qlpsk.18637f45c32c713c kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:cilium-operator-6b658bf6bc-qlpsk,UID:4a092ece-bd27-4684-a21a-49081a2868ac,APIVersion:v1,ResourceVersion:652933969,FieldPath:spec.containers{cilium-operator},},Reason:Unhealthy,Message:Readiness probe failed: HTTP probe failed with statuscode: 500,Source:EventSource{Component:kubelet,Host:prod-server-blue-k3s-01,},FirstTimestamp:2025-09-09 03:23:37.208566076 +0000 UTC m=+621.723338861,LastTimestamp:2025-09-09 03:23:37.208566076 +0000 UTC m=+621.723338861,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:prod-server-blue-k3s-01,}" Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:11.562440 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 6.265µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:37.571159621 +0000 UTC m=+622.085932331) (total time: 33.991248436s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.991248436s Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=error msg="insert error for key /registry/events/verify/id-verify-service-5c969886c6-gx7qz.185fd6a2076a8acd: context canceled" createRevision=653338925 key=/registry/events/verify/id-verify-service-5c969886c6-gx7qz.185fd6a2076a8acd previousRevision=653338925 Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:37.762270921 +0000 UTC m=+622.277043639) (total time: 33.991081463s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.991081463s Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:11.753464 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 15.239µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=error msg="insert error for key /registry/events/verify/idi-gateway-6766d6db77-j9x5w.185fd6a21345d5ae: context canceled" createRevision=653338916 key=/registry/events/verify/idi-gateway-6766d6db77-j9x5w.185fd6a21345d5ae previousRevision=653338916 Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:37.789390389 +0000 UTC m=+622.304163104) (total time: 33.969842012s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.969842012s Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=error msg="insert error for key /registry/events/sign/vsp-core-656c796f98-5td9l.18633a2039a5737e: timeout: context deadline exceeded" createRevision=653338928 key=/registry/events/sign/vsp-core-656c796f98-5td9l.18633a2039a5737e previousRevision=653338928 Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:11.759439 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 11.499µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.6.61.141:6443: connect: connection refused" Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.6.61.141:6443: connect: connection refused" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: W0909 03:24:11.818194 983458 transport.go:356] Unable to cancel request for *otelhttp.Transport Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:11.818365 983458 controller.go:195] "Failed to update lease" err="Put \"https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-server-blue-k3s-01?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: I0909 03:24:11.818416 983458 controller.go:115] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease" Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:11.818558 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 10.167µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:01.826176684 +0000 UTC m=+646.340949403) (total time: 9.992390299s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.992390299s Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-server-blue-k3s-01: context canceled" createRevision=320940478 key=/registry/leases/kube-node-lease/prod-server-blue-k3s-01 previousRevision=653344634 Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:11.818757 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:11.820327 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:11.820404 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:11.821603 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.920718ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-server-blue-k3s-01" result: Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:11.864859 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 15.941µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:37.867553003 +0000 UTC m=+622.382325720) (total time: 33.99732178s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.99732178s Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master: context canceled" createRevision=537633548 key=/registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master previousRevision=653344740 Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:37.878793001 +0000 UTC m=+622.393565677) (total time: 33.99684748s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.99684748s Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:11Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master: context canceled" createRevision=537633548 key=/registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master previousRevision=653344740 Sep 09 03:24:11 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:11.875744 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 11.433µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:38.097241078 +0000 UTC m=+622.612013797) (total time: 33.996386299s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.996386299s Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-ingress-nginx-ingress-nginx-controller: timeout: context deadline exceeded" createRevision=545796592 key=/registry/leases/kube-system/cilium-l2announce-ingress-nginx-ingress-nginx-controller previousRevision=653344728 Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.093868 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 9.73µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="error in txn compare: success:\\n\\036prod-agent-app-blue-0-f0380b7a\\020\\017\\032\\013\\010\\272\\273\\376\\305\\006\\020\\230\\326\\340\\033\\\"\\013\\010\\272\\273\\376\\305\\006\\020\\230\\326\\340\\033(7\\032\\000\\\"\\000\" > > failure: > : context deadline exceeded" Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:38.165139395 +0000 UTC m=+622.679912155) (total time: 33.970090674s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.970090674s Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="insert error for key /registry/events/dev-microblink-system/s67e56611a7b8d3605d024303s-6f9d4fc54d-qgl8k.186353d0daf2f956: timeout: context deadline exceeded" createRevision=652880172 key=/registry/events/dev-microblink-system/s67e56611a7b8d3605d024303s-6f9d4fc54d-qgl8k.186353d0daf2f956 previousRevision=653341872 Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.135418 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 11.427µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:38.638095998 +0000 UTC m=+623.152868717) (total time: 33.995381427s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.995381427s Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-ingress-nginx-ingress-nginx-controller: timeout: context deadline exceeded" createRevision=545796592 key=/registry/leases/kube-system/cilium-l2announce-ingress-nginx-ingress-nginx-controller previousRevision=653344728 Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.633913 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 16.766µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:02.793979429 +0000 UTC m=+647.308752158) (total time: 9.992635446s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.992635446s Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ops-blue-1-89a427cc: timeout: context deadline exceeded" createRevision=321006889 key=/registry/leases/kube-node-lease/prod-agent-ops-blue-1-89a427cc previousRevision=653344659 Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:02.837552421 +0000 UTC m=+647.352325102) (total time: 9.949387679s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.949387679s Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-3-aef87d06: timeout: context deadline exceeded" createRevision=320960233 key=/registry/leases/kube-node-lease/prod-agent-app-blue-3-aef87d06 previousRevision=653344685 Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:02.826385059 +0000 UTC m=+647.341157784) (total time: 9.960967035s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.960967035s Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ingress-blue-2-39068466: timeout: context deadline exceeded" createRevision=320947107 key=/registry/leases/kube-node-lease/prod-agent-ingress-blue-2-39068466 previousRevision=653344680 Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:02.822121015 +0000 UTC m=+647.336893693) (total time: 9.965558026s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.965558026s Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-4-79b5072b: timeout: context deadline exceeded" createRevision=320960218 key=/registry/leases/kube-node-lease/prod-agent-app-blue-4-79b5072b previousRevision=653344692 Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.787113 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:02.830957941 +0000 UTC m=+647.345730663) (total time: 9.957452605s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.957452605s Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.788484 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.787151 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 10.414µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.787408 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.787427 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 3.953µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ingress-blue-0-ac9168ff: timeout: context deadline exceeded" createRevision=320947169 key=/registry/leases/kube-node-lease/prod-agent-ingress-blue-0-ac9168ff previousRevision=653344686 Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.788634 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.788771 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 10.365µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.788944 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.789070 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.073µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:02.835202213 +0000 UTC m=+647.349974925) (total time: 9.953809529s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.953809529s Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-10-1ef6ca3c: timeout: context deadline exceeded" createRevision=613356118 key=/registry/leases/kube-node-lease/prod-agent-app-blue-10-1ef6ca3c previousRevision=653344688 Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.789198 983458 wrap.go:54] timeout or abort while handling: method=PUT URI="/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ingress-blue-2-39068466?timeout=10s" audit-ID="5e07bb03-6250-490a-abdb-bb37bf23de31" Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.789234 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.021µs, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ingress-blue-2-39068466" result: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.789313 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 7.283µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:02.832486248 +0000 UTC m=+647.347259025) (total time: 9.956949899s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.956949899s Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.789513 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 7.53µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-0-f0380b7a: context canceled" createRevision=320960246 key=/registry/leases/kube-node-lease/prod-agent-app-blue-0-f0380b7a previousRevision=653344668 Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.789593 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.789662 983458 wrap.go:54] timeout or abort while handling: method=PUT URI="/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-10-1ef6ca3c?timeout=10s" audit-ID="566faa3c-ed6b-49da-abc9-3c7c53c2dc82" Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.789672 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.789700 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.286µs, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-10-1ef6ca3c" result: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.789703 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.789810 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 5.827µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.789811 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.789856 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.790408 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.790496 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 51.269µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:02.836029942 +0000 UTC m=+647.350802656) (total time: 9.954470183s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.954470183s Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.790688 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:02.832040026 +0000 UTC m=+647.346812738) (total time: 9.958435106s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.958435106s Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-9-63cdf3ff: timeout: context deadline exceeded" createRevision=613356165 key=/registry/leases/kube-node-lease/prod-agent-app-blue-9-63cdf3ff previousRevision=653344696 Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-1-a234b216: timeout: context deadline exceeded" createRevision=320959216 key=/registry/leases/kube-node-lease/prod-agent-app-blue-1-a234b216 previousRevision=653344667 Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.790714 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 5.241µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.790729 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.790810 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.993742ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ops-blue-1-89a427cc" result: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.790812 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.791072 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.791182 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.791183 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.791569 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.792195 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.792254 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.792297 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.792459 983458 timeout.go:142] post-timeout activity - time-elapsed: 4.994177ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-3-aef87d06" result: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.792519 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.663973ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-4-79b5072b" result: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.792557 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.930099ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ingress-blue-0-ac9168ff" result: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.792830 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.300835ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-1-a234b216" result: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.793337 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.82865ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-0-f0380b7a" result: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.793429 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.676981ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-9-63cdf3ff" result: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.835366 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.214µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:02.840052458 +0000 UTC m=+647.354825150) (total time: 9.995274454s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.995274454s Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.835483 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ingress-blue-1-4323cca2: context canceled" createRevision=320946796 key=/registry/leases/kube-node-lease/prod-agent-ingress-blue-1-4323cca2 previousRevision=653344650 Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.836649 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.836711 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.837867 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.602408ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ingress-blue-1-4323cca2" result: Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:38.87422722 +0000 UTC m=+623.388999898) (total time: 33.968875225s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.968875225s Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="insert error for key /registry/events/verify/id-verify-ocr-service-56f975cbbb-srdn4.18623b097501d63c: timeout: context deadline exceeded" createRevision=653338813 key=/registry/events/verify/id-verify-ocr-service-56f975cbbb-srdn4.18623b097501d63c previousRevision=653338813 Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:12Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:12 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:12.843326 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 5.769µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.009031 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.009090 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.009049 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.3µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:03.022025647 +0000 UTC m=+647.536798326) (total time: 9.987082222s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.987082222s Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-5-f52e9ef0: context canceled" createRevision=624037756 key=/registry/leases/kube-node-lease/prod-agent-data-green-5-f52e9ef0 previousRevision=653344682 Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.011084 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.011315 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.359047ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-5-f52e9ef0" result: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.012062 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 6.751µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:03.053474571 +0000 UTC m=+647.568247250) (total time: 9.958468367s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.958468367s Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-6-420706e2: context canceled" createRevision=624036980 key=/registry/leases/kube-node-lease/prod-agent-data-green-6-420706e2 previousRevision=653344661 Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.012202 983458 wrap.go:54] timeout or abort while handling: method=PUT URI="/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-6-420706e2?timeout=10s" audit-ID="7beebc9d-040c-4be4-b275-19c98fa15b0d" Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.012233 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.418µs, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-6-420706e2" result: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:03.04131755 +0000 UTC m=+647.556090242) (total time: 9.99688843s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.99688843s Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-7-c3860444: timeout: context deadline exceeded" createRevision=320959253 key=/registry/leases/kube-node-lease/prod-agent-app-blue-7-c3860444 previousRevision=653344660 Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.038538 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.038584 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.038547 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 7.679µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.039684 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.039824 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.334459ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-7-c3860444" result: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:03.046619391 +0000 UTC m=+647.561392095) (total time: 9.994450337s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.994450337s Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ops-blue-0-4b80a6a8: timeout: context deadline exceeded" createRevision=321006849 key=/registry/leases/kube-node-lease/prod-agent-ops-blue-0-4b80a6a8 previousRevision=653344656 Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.041330 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.041369 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.041394 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 7.239µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.042189 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.042218 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 5.395µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:03.071424137 +0000 UTC m=+647.586196826) (total time: 9.970808552s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.970808552s Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-7-93605edd: timeout: context deadline exceeded" createRevision=624037614 key=/registry/leases/kube-node-lease/prod-agent-data-green-7-93605edd previousRevision=653344691 Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.042457 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.042802 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.043766 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.043796 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.4821ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-ops-blue-0-4b80a6a8" result: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.043893 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.750113ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-7-93605edd" result: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:03.055511088 +0000 UTC m=+647.570283782) (total time: 9.990459204s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.990459204s Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-4-96deb91b: timeout: context deadline exceeded" createRevision=624037787 key=/registry/leases/kube-node-lease/prod-agent-data-green-4-96deb91b previousRevision=653344689 Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.046322 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.046351 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.046381 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 7.873µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.047913 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.047986 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.588778ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-4-96deb91b" result: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.048332 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.048374 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 7.405µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:03.051967176 +0000 UTC m=+647.566739862) (total time: 9.996331292s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.996331292s Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-2-9c964185: context canceled" createRevision=320960197 key=/registry/leases/kube-node-lease/prod-agent-app-blue-2-9c964185 previousRevision=653344675 Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.053403 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.053441 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:03.054730829 +0000 UTC m=+647.569503514) (total time: 9.998805719s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.998805719s Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.053695 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-6-cd93959b: timeout: context deadline exceeded" createRevision=320959048 key=/registry/leases/kube-node-lease/prod-agent-app-blue-6-cd93959b previousRevision=653344672 Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.053742 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 29.863µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.053780 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.053810 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 5.281µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.054103 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:03.055858272 +0000 UTC m=+647.570631013) (total time: 9.998151946s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.998151946s Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:03.07440825 +0000 UTC m=+647.589180947) (total time: 9.979164012s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.979164012s Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-0-9b98de05: timeout: context deadline exceeded" createRevision=623966046 key=/registry/leases/kube-node-lease/prod-agent-data-green-0-9b98de05 previousRevision=653344666 Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-5-bb69767d: timeout: context deadline exceeded" createRevision=320959392 key=/registry/leases/kube-node-lease/prod-agent-app-blue-5-bb69767d previousRevision=653344674 Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.054271 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 79.324µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.054475 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.055070 983458 timeout.go:142] post-timeout activity - time-elapsed: 6.835746ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-2-9c964185" result: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.055356 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.055420 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.055841 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.056396 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.056799 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.300393ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-5-bb69767d" result: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.056956 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.127602ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-app-blue-6-cd93959b" result: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.057005 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.057573 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.057594 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 4.216µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:03.065624092 +0000 UTC m=+647.580396799) (total time: 9.992048223s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.992048223s Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-3-47319b7a: context canceled" createRevision=624036915 key=/registry/leases/kube-node-lease/prod-agent-data-green-3-47319b7a previousRevision=653344684 Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.058108 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.058137 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.276327ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-0-9b98de05" result: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.058682 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.060189 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.647236ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/prod-agent-data-green-3-47319b7a" result: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:39.772245965 +0000 UTC m=+624.287018679) (total time: 33.970235207s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.970235207s Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-ingress-nginx-secondary-ingress-nginx-secondary-controller: timeout: context deadline exceeded" createRevision=537632387 key=/registry/leases/kube-system/cilium-l2announce-ingress-nginx-secondary-ingress-nginx-secondary-controller previousRevision=653344732 Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.742985 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 11.459µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:39.770343958 +0000 UTC m=+624.285116653) (total time: 33.996211381s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.996211381s Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-ingress-nginx-secondary-ingress-nginx-secondary-controller: timeout: context deadline exceeded" createRevision=537632387 key=/registry/leases/kube-system/cilium-l2announce-ingress-nginx-secondary-ingress-nginx-secondary-controller previousRevision=653344732 Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:13.766949 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 4.449µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:13 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:13Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:14Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:40.4821731 +0000 UTC m=+624.996945870) (total time: 33.996293889s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.996293889s Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:14Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-ingress-nginx-ingress-nginx-controller: timeout: context deadline exceeded" createRevision=545796592 key=/registry/leases/kube-system/cilium-l2announce-ingress-nginx-ingress-nginx-controller previousRevision=653344728 Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:14Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:14.479197 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 8.629µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:14Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:04.561801029 +0000 UTC m=+649.076573724) (total time: 9.985493672s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.985493672s Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:14Z" level=error msg="insert error for key /registry/minions/prod-agent-data-green-6-420706e2: timeout: context deadline exceeded" createRevision=624036760 key=/registry/minions/prod-agent-data-green-6-420706e2 previousRevision=653342245 Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:14.547983 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:14.548046 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:14.548247 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 155.139µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:14Z" level=error msg="error in txn compare: success:\\n6volumes.kubernetes.io/controller-managed-attach-detach\\022\\004truer\\027wrangler.cattle.io/node\\212\\001\\231\\001\\n&k3s-supervisor@prod-server-blue-k3s-01\\022\\006Update\\032\\002v1\\\"\\010\\010\\347\\262\\314\\304\\006\\020\\0002\\010FieldsV1:M\\nK{\\\"f:metadata\\\":{\\\"f:finalizers\\\":{\\\".\\\":{},\\\"v:\\\\\\\"wrangler.cattle.io/node\\\\\\\"\\\":{}}}}B\\000\\212\\001\\267\\006\\n\\003k3s\\022\\006Update\\032\\002v1\\\"\\010\\010\\207\\263\\314\\304\\006\\020\\0002\\010FieldsV1:\\215\\006\\n\\212\\006{\\\"f:metadata\\\":{\\\"f:annotations\\\":{\\\".\\\":{},\\\"f:alpha.kubernetes.io/provided-node-ip\\\":{},\\\"f:k3s.io/hostname\\\":{},\\\"f:k3s.io/internal-ip\\\":{},\\\"f:k3s.io/node-args\\\":{},\\\"f:k3s.io/node-config-hash\\\":{},\\\"f:k3s.io/node-env\\\":{},\\\"f:node.alpha.kubernetes.io/ttl\\\":{},\\\"f:volumes.kubernetes.io/controller-managed-attach-detach\\\":{}},\\\"f:labels\\\":{\\\".\\\":{},\\\"f:beta.kubernetes.io/arch\\\":{},\\\"f:beta.kubernetes.io/instance-type\\\":{},\\\"f:beta.kubernetes.io/os\\\":{},\\\"f:cluster-name\\\":{},\\\"f:failure-domain.beta.kubernetes.io/zone\\\":{},\\\"f:kubernetes.io/arch\\\":{},\\\"f:kubernetes.io/hostname\\\":{},\\\"f:kubernetes.io/os\\\":{},\\\"f:node-group\\\":{},\\\"f:node-role\\\":{},\\\"f:node.kubernetes.io/instance-type\\\":{},\\\"f:topology.kubernetes.io/zone\\\":{}}},\\\"f:spec\\\":{\\\"f:podCIDR\\\":{},\\\"f:podCIDRs\\\":{\\\".\\\":{},\\\"v:\\\\\\\"10.128.34.0/24\\\\\\\"\\\":{}},\\\"f:providerID\\\":{}}}B\\000\\212\\001\\377\\001\\n\\027cilium-operator-generic\\022\\006Update\\032\\002v1\\\"\\010\\010\\214\\263\\314\\304\\006\\020\\0002\\010FieldsV1:\\273\\001\\n\\270\\001{\\\"f:status\\\":{\\\"f:conditions\\\":{\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"NetworkUnavailable\\\\\\\"}\\\":{\\\".\\\":{},\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{},\\\"f:type\\\":{}}}}}B\\006status\\212\\001Y\\n\\027cilium-operator-generic\\022\\006Update\\032\\002v1\\\"\\010\\010\\277\\301\\371\\305\\006\\020\\0002\\010FieldsV1:\\034\\n\\032{\\\"f:spec\\\":{\\\"f:taints\\\":{}}}B\\000\\212\\001\\273\\004\\n\\003k3s\\022\\006Update\\032\\002v1\\\"\\010\\010\\324\\273\\376\\305\\006\\020\\0002\\010FieldsV1:\\213\\004\\n\\210\\004{\\\"f:metadata\\\":{\\\"f:annotations\\\":{\\\"f:csi.volume.kubernetes.io/nodeid\\\":{}}},\\\"f:status\\\":{\\\"f:addresses\\\":{\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"Hostname\\\\\\\"}\\\":{\\\"f:address\\\":{}}},\\\"f:conditions\\\":{\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"DiskPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"MemoryPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"PIDPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"Ready\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{}}},\\\"f:images\\\":{},\\\"f:volumesAttached\\\":{},\\\"f:volumesInUse\\\":{}}}B\\006status\\022k\\n\\01610.128.34.0/24\\022\\000\\032&k3s://prod-agent-data-green-6-420706e2 \\000*\\035\\n\\tnode-role\\022\\004data\\032\\nNoSchedule:\\01610.128.34.0/24\\032\\330=\\n\\013\\n\\003cpu\\022\\004\\n\\00216\\n!\\n\\021ephemeral-storage\\022\\014\\n\\n29751268Ki\\n\\024\\n\\rhugepages-1Gi\\022\\003\\n\\0010\\n\\024\\n\\rhugepages-2Mi\\022\\003\\n\\0010\\n\\026\\n\\006memory\\022\\014\\n\\n32860600Ki\\n\\r\\n\\004pods\\022\\005\\n\\003110\\022\\013\\n\\003cpu\\022\\004\\n\\00216\\022\\\"\\n\\021ephemeral-storage\\022\\r\\n\\01328942033488\\022\\024\\n\\rhugepages-1Gi\\022\\003\\n\\0010\\022\\024\\n\\rhugepages-2Mi\\022\\003\\n\\0010\\022\\026\\n\\006memory\\022\\014\\n\\n32860600Ki\\022\\r\\n\\004pods\\022\\005\\n\\003110\\032\\000\\\"[\\n\\022NetworkUnavailable\\022\\005False\\032\\010\\010\\214\\263\\314\\304\\006\\020\\000\\\"\\010\\010\\214\\263\\314\\304\\006\\020\\000*\\nCiliumIsUp2\\036Cilium is running on this node\\\"p\\n\\016MemoryPressure\\022\\005False\\032\\010\\010\\324\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\347\\262\\314\\304\\006\\020\\000*\\032KubeletHasSufficientMemory2'kubelet has sufficient memory available\\\"a\\n\\014DiskPressure\\022\\005False\\032\\010\\010\\324\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\347\\262\\314\\304\\006\\020\\000*\\030KubeletHasNoDiskPressure2\\034kubelet has no disk pressure\\\"g\\n\\013PIDPressure\\022\\005False\\032\\010\\010\\324\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\347\\262\\314\\304\\006\\020\\000*\\027KubeletHasSufficientPID2$kubelet has sufficient PID available\\\"P\\n\\005Ready\\022\\004True\\032\\010\\010\\324\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\205\\263\\314\\304\\006\\020\\000*\\014KubeletReady2\\037kubelet is posting ready status*\\031\\n\\nInternalIP\\022\\01310.6.61.247*#\\n\\010Hostname\\022\\027prod-agent-data-green-62\\005\\n\\003\\010\\212P:\\334\\001\\n 930d76008d2c408e9ba36a050eb87a99\\022$a30b0a42-5455-c064-a707-0b4e2157331f\\032$45b37632-65b1-4cd2-bd78-a0c5920ff7c5\\\"\\0225.15.0-119-generic*\\022Ubuntu 22.04.4 LTS2\\030containerd://1.7.27-k3s1:\\rv1.30.14+k3s2B\\rv1.30.14+k3s2J\\005linuxR\\005amd64B\\251\\001\\npdocker.io/grafana/grafana-image-renderer@sha256:ad45676a30678ffb922399b48b076f800c4167b8feb8343b8f87f4a9a30aacd6\\n/docker.io/grafana/grafana-image-renderer:latest\\020\\307\\367\\371\\342\\002B\\324\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/oss-data@sha256:f64755b5ca9e390a255817398e435fff9af2353cfc3593593f27281f07d54f10\\nG868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/oss-data:main-77e3f49\\020\\201\\311\\251\\216\\002B\\225\\001\\ndpublic.ecr.aws/datadog/agent@sha256:465da828947d6c055eacf399df8d0f96423f6e029fd1218a806f091a0dc0200b\\n'public.ecr.aws/datadog/agent:7.62.0-jmx\\020\\207\\330\\325\\347\\001B\\334\\001\\n}868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops@sha256:e88bf7e606df57a9e7c3efa1f489132a72ad76f981bf7419e9598ce83ff9005c\\nU868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops:docker-wazuh-agent-a62b8c9-main\\020\\323\\304\\323\\335\\001B\\253\\002\\n\\256\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/tritonserver-cpu-onnxruntime/onprem@sha256:2e310dfa28f70f304c3abe48afb0376b5e2db7b2a36dcbe2ba837bc9a57147ed\\nrus-central1-docker.pkg.dev/document-verification-public/docver-gcm/tritonserver-cpu-onnxruntime/onprem:23.06-3.8.0\\020\\301\\305\\250\\233\\001B\\252\\002\\n\\254\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/tf-serving-kira-native-cpu/onprem@sha256:71a22b984b8a609cfadf3e1796e991ededa0c1e26985d83d14a161b1a6459e5b\\nsus-central1-docker.pkg.dev/document-verification-public/docver-gcm/tf-serving-kira-native-cpu/onprem:v2.13.0-3.15.1\\020\\311\\375\\253\\207\\001B\\203\\001\\n]quay.io/cilium/cilium@sha256:5a94b561f4651fcfd85970a50bc78b201cfbd6e2ab1a03848eab25a82832653a\\n\\035quay.io/cilium/cilium:v1.14.0\\020\\300\\317\\204WB\\242\\002\\n\\254\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/tf-serving-kira-native-cpu/onprem@sha256:4bfcd319098ec6d0f16346222fa2114bcdc2efcb15c58bf4e5cb6ed86564bb4d\\nlus-central1-docker.pkg.dev/document-verification-public/docver-gcm/tf-serving-kira-native-cpu/onprem:v2.13.0\\020\\304\\233\\302IB\\244\\002\\n\\256\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/tritonserver-cpu-onnxruntime/onprem@sha256:67f677d008d438602881b950dc2309b774d643affc0d2e0aeabdc91d6ff76c12\\nlus-central1-docker.pkg.dev/document-verification-public/docver-gcm/tritonserver-cpu-onnxruntime/onprem:23.06\\020\\212\\202\\305HB\\212\\001\\nadocker.io/grafana/grafana@sha256:8b37a2f028f164ce7b9889e1765b9d6ee23fec80f871d156fbf436d6198d32b7\\n docker.io/grafana/grafana:11.5.2\\020\\215\\307\\204HB\\212\\002\\n\\241\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-visual-anomaly@sha256:1e0a60549563c9742089a7c370dada22d3672611b71b6a833e7429b4f954604a\\n_us-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-visual-anomaly:1.2.1\\020\\305\\325\\227=B\\222\\001\\nequay.io/hpestorage/csi-driver@sha256:72c2e0477e750ea7301fa3543f951dd9f03e29aa8b7eba5e3084b2f1cc06c997\\n$quay.io/hpestorage/csi-driver:v2.5.2\\020\\201\\315\\3575B\\236\\001\\nddocker.io/bitnami/postgresql@sha256:aedda040a9d1b70df4b76cfb237dcf8f913afde5c41b998c61f8e6a0a7f1ee3a\\n1docker.io/bitnami/postgresql:15.2.0-debian-11-r30\\020\\331\\373\\247+B\\373\\001\\n\\231\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-runner@sha256:f4b783b60f9396bfd22a6b2249a060cfe59a2afb94d2103d563a87f5da7d5c96\\nXus-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-runner:3.15.0\\020\\366\\212\\352#B\\214\\002\\n\\241\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/embedding-store/onprem@sha256:2ccc03f7d575a928f879cf207ba789ad5a4eedd5a36afc643fba4dc9787e5d78\\naus-central1-docker.pkg.dev/document-verification-public/docver-gcm/embedding-store/onprem:v0.32.5\\020\\215\\266\\372 B\\372\\001\\n\\231\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-runner@sha256:0a172f4a12738999fcaa76df7082f2d1c966736d30f2f202a2a85b9d87f997cc\\nWus-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-runner:3.4.0\\020\\322\\226\\314 B\\214\\002\\n\\241\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/embedding-store/onprem@sha256:7fbbbacc4d025aab8664ddebcbbb32e5c16f85a8d57c43da0a86d30859b8a734\\naus-central1-docker.pkg.dev/document-verification-public/docver-gcm/embedding-store/onprem:v0.31.1\\020\\345\\260\\254\\036B\\251\\001\\njdocker.io/fabiocicerchia/nginx-lua@sha256:5bfbce1515934a8a085355379dd592d38dac3ee5f790971dd3ac2e53436cb685\\n6docker.io/fabiocicerchia/nginx-lua:1.29.1-alpine3.22.1\\020\\342\\320\\341\\026B\\251\\001\\njdocker.io/fabiocicerchia/nginx-lua@sha256:e21edef393d8a114ffd8d33292520aa960638cfe36946b09118290655b8acc61\\n6docker.io/fabiocicerchia/nginx-lua:1.25.2-alpine3.18.3\\020\\204\\271\\310\\024B\\252\\002\\n\\260\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/anomaly-detection-intermediary/onprem@sha256:1d9f6ca664c264a68f13a7507f0af7728c16deca5ae37ca6534c7f63feba13ee\\npus-central1-docker.pkg.dev/document-verification-public/docver-gcm/anomaly-detection-intermediary/onprem:v0.11.3\\020\\355\\371\\305\\024B\\214\\002\\n\\241\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/mlp-model-proxy/onprem@sha256:dfee307564fcd5cf7c856c1da56d9046248633e91c32b96cd818fa8a1f60e0f4\\naus-central1-docker.pkg.dev/document-verification-public/docver-gcm/mlp-model-proxy/onprem:v0.19.2\\020\\320\\351\\330\\023B\\375\\001\\n\\232\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-doc-ver@sha256:cca2da3c9f11afd33acaba8aaf51db636e1d0d4d8cb5f62e5b904d04c4139e23\\nYus-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-doc-ver:3.15.0\\020\\243\\350\\354\\022B\\374\\001\\n\\232\\001us-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-doc-ver@sha256:eccf730b6a51a23c71f0d52e10cf2f92e3831a89e0bd89afb0345d8340e7363d\\nXus-central1-docker.pkg.dev/document-verification-public/docver-gcm/web-api-doc-ver:3.4.0\\020\\231\\336\\313\\022B\\206\\001\\n_docker.io/bitnami/redis@sha256:69b26bf4461a273ee3ed2cbd0a053caf6e4e833066b7dd40c692e425e8706027\\n\\036docker.io/bitnami/redis:6.2.12\\020\\346\\360\\246\\021B\\246\\001\\nhdocker.io/bitnami/redis-sentinel@sha256:46c47870d2662426226d8c906a080fd9b18eeb5844bdfcf3cccdfa7486ea14b7\\n5docker.io/bitnami/redis-sentinel:7.0.11-debian-11-r18\\020\\304\\204\\365\\020B\\201\\001\\n_docker.io/rclone/rclone@sha256:34c729127386abec1c610b2aa024e39b4498dc2b4a72a0798ae21fbdc1b0493b\\n\\031docker.io/rclone/rclone:1\\020\\244\\207\\227\\rB\\254\\001\\nrdocker.io/rancher/mirrored-coredns-coredns@sha256:a11fafae1f8037cbbd66c5afa40ba2423936b72b4fd50a7034a7e8b955163594\\n1docker.io/rancher/mirrored-coredns-coredns:1.10.1\\020\\271\\225\\334\\007B\\303\\001\\n}registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:d7138bcc3aa5f267403d45ad4292c95397e421ea17a0035888850f424c7de25d\\n=registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0\\020\\277\\230\\206\\007B\\230\\001\\nhquay.io/prometheus/node-exporter@sha256:4032c6d5bfd752342c3e631c2f1de93ba6b86c41db6b167b9a35372c139e7706\\n'quay.io/prometheus/node-exporter:v1.8.2\\020\\370\\371\\336\\005B\\211\\001\\nadocker.io/library/busybox@sha256:95cf004f559831017cdf4628aaf1bb30133677be8702a8c5f2994629f637a209\\n docker.io/library/busybox:1.31.1\\020\\214\\325.B\\224\\001\\nhdocker.io/rancher/mirrored-pause@sha256:74c4244427b7312c5b901fe0f67cbc53683d06f4f24c6faee65d4182bf0fa893\\n$docker.io/rancher/mirrored-pause:3.6\\020\\227\\263\\022JHkubernetes.io/csi/csi.hpe.com^06766d9dd0b83f1e9100000000000000000000002dJHkubernetes.io/csi/csi.hpe.com^06766d9dd0b83f1e91000000000000000000000057RL\\nHkubernetes.io/csi/csi.hpe.com^06766d9dd0b83f1e91000000000000000000000057\\022\\000RL\\nHkubernetes.io/csi/csi.hpe.com^06766d9dd0b83f1e9100000000000000000000002d\\022\\000\\032\\000\\\"\\000\" > > failure: > : context deadline exceeded" Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:14.549191 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:14.550047 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.756697ms, PATCH "/api/v1/nodes/prod-agent-data-green-6-420706e2/status" result: Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:14Z" level=info msg="Connecting to proxy" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:14Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.6.61.141:6443: connect: connection refused" Sep 09 03:24:14 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:14Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.6.61.141:6443: connect: connection refused" url="wss://10.6.61.141:6443/v1-k3s/connect" Sep 09 03:24:15 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:15.290820 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:15 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:15.290882 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:15 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:15.290938 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 30.808µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:15 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:15Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:05.296039009 +0000 UTC m=+649.810811728) (total time: 9.994846897s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.994846897s Sep 09 03:24:15 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:15Z" level=error msg="insert error for key /registry/configmaps/ops/datadog-leader-election: timeout: context deadline exceeded" createRevision=4206015 key=/registry/configmaps/ops/datadog-leader-election previousRevision=653344733 Sep 09 03:24:15 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:15Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:15 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:15.291986 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:15 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:15.292137 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.350924ms, PUT "/api/v1/namespaces/ops/configmaps/datadog-leader-election" result: Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: W0909 03:24:16.034001 983458 transport.go:356] Unable to cancel request for *otelhttp.Transport Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.034181 983458 kubelet_node_status.go:544] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"NetworkUnavailable\\\"},{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:24:06Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:24:06Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:24:06Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2025-09-09T03:24:06Z\\\",\\\"type\\\":\\\"Ready\\\"}]}}\" for node \"prod-server-blue-k3s-01\": Patch \"https://127.0.0.1:6443/api/v1/nodes/prod-server-blue-k3s-01/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:06.044106339 +0000 UTC m=+650.558879058) (total time: 9.990474911s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.990474911s Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=error msg="insert error for key /registry/minions/prod-server-blue-k3s-01: context canceled" createRevision=320940137 key=/registry/minions/prod-server-blue-k3s-01 previousRevision=653341971 Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.034714 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.034767 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.034805 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 9.456µs, panicked: false, err: context canceled, panic-reason: Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.035878 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.036034 983458 timeout.go:142] post-timeout activity - time-elapsed: 1.602814ms, PATCH "/api/v1/nodes/prod-server-blue-k3s-01/status" result: Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:42.428909707 +0000 UTC m=+626.943682449) (total time: 33.988144488s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.988144488s Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=error msg="insert error for key /registry/leases/ingress-nginx/ingress-nginx-leader: timeout: context deadline exceeded" createRevision=98641045 key=/registry/leases/ingress-nginx/ingress-nginx-leader previousRevision=653344729 Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.417327 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 13.1µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:42.503268632 +0000 UTC m=+627.018041332) (total time: 33.971740475s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.971740475s Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=error msg="insert error for key /registry/leases/ingress-nginx-secondary/ingress-nginx-secondary-leader: timeout: context deadline exceeded" createRevision=474540760 key=/registry/leases/ingress-nginx-secondary/ingress-nginx-secondary-leader previousRevision=653344730 Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.475218 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 17.874µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:11.550403897 +0000 UTC m=+656.065176609) (total time: 4.994263032s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.994263032s Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=error msg="insert error for key /registry/leases/kube-system/k3s-cloud-controller-manager: timeout: context deadline exceeded" createRevision=368 key=/registry/leases/kube-system/k3s-cloud-controller-manager previousRevision=653344738 Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.544841 983458 leaderelection.go:383] Failed to update lock: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/k3s-cloud-controller-manager?timeout=5s": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.544948 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 31.524µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=error msg="error in txn compare: success: > failure: > : context deadline exceeded" Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.545215 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.546376 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.546428 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.547608 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.483195ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/k3s-cloud-controller-manager" result: Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.728904 983458 leaderelection.go:383] Failed to update lock: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers) Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.729149 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 19.24µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.729202 983458 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:11.735110392 +0000 UTC m=+656.249883109) (total time: 4.994001423s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.994001423s Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=error msg="insert error for key /registry/leases/kube-system/kube-scheduler: context canceled" createRevision=387 key=/registry/leases/kube-system/kube-scheduler previousRevision=653344736 Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.730972 983458 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.731051 983458 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.732414 983458 timeout.go:142] post-timeout activity - time-elapsed: 3.306428ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result: Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:42.866599693 +0000 UTC m=+627.381372453) (total time: 33.99810142s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=33.99810142s Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=error msg="insert error for key /registry/events/verify/idverify-gateway-69c9675676-qtskd.18637f4714206dd8: timeout: context deadline exceeded" createRevision=0 key=/registry/events/verify/idverify-gateway-69c9675676-qtskd.18637f4714206dd8 previousRevision=0 Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:16Z" level=error msg="error in txn compare: success: > : timeout: context deadline exceeded" Sep 09 03:24:16 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:16.865653 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 10.408µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:27.583116036 +0000 UTC m=+612.097888797) (total time: 50.067639479s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=50.067639479s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/masterleases/10.6.61.141: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=653336725 key=/registry/masterleases/10.6.61.141 previousRevision=653344639 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:17.652164 983458 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 3.506µs, panicked: false, err: context deadline exceeded, panic-reason: Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:17.652292 983458 wrap.go:54] timeout or abort while handling: method=PATCH URI="/api/v1/nodes/prod-agent-app-blue-4-79b5072b/status?timeout=10s" audit-ID="91010c70-379d-4774-8cab-2aa9e5b5b7be" Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:07.683998254 +0000 UTC m=+652.198771013) (total time: 9.968262729s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.968262729s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:17.652325 983458 timeout.go:142] post-timeout activity - time-elapsed: 2.769µs, PATCH "/api/v1/nodes/prod-agent-app-blue-4-79b5072b/status" result: Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/minions/prod-agent-app-blue-4-79b5072b: timeout: context deadline exceeded" createRevision=320959816 key=/registry/minions/prod-agent-app-blue-4-79b5072b previousRevision=653341980 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="error in txn compare: success:\\n6volumes.kubernetes.io/controller-managed-attach-detach\\022\\004truer\\027wrangler.cattle.io/node\\212\\001\\231\\001\\n&k3s-supervisor@prod-server-blue-k3s-00\\022\\006Update\\032\\002v1\\\"\\010\\010\\222\\253\\302\\266\\006\\020\\0002\\010FieldsV1:M\\nK{\\\"f:metadata\\\":{\\\"f:finalizers\\\":{\\\".\\\":{},\\\"v:\\\\\\\"wrangler.cattle.io/node\\\\\\\"\\\":{}}}}B\\000\\212\\001\\366\\001\\n\\016Go-http-client\\022\\006Update\\032\\002v1\\\"\\010\\010\\313\\253\\302\\266\\006\\020\\0002\\010FieldsV1:\\273\\001\\n\\270\\001{\\\"f:status\\\":{\\\"f:conditions\\\":{\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"NetworkUnavailable\\\\\\\"}\\\":{\\\".\\\":{},\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{},\\\"f:type\\\":{}}}}}B\\006status\\212\\001\\346\\005\\n\\003k3s\\022\\006Update\\032\\002v1\\\"\\010\\010\\310\\317\\267\\305\\006\\020\\0002\\010FieldsV1:\\274\\005\\n\\271\\005{\\\"f:metadata\\\":{\\\"f:annotations\\\":{\\\".\\\":{},\\\"f:alpha.kubernetes.io/provided-node-ip\\\":{},\\\"f:k3s.io/hostname\\\":{},\\\"f:k3s.io/internal-ip\\\":{},\\\"f:k3s.io/node-args\\\":{},\\\"f:k3s.io/node-config-hash\\\":{},\\\"f:k3s.io/node-env\\\":{},\\\"f:node.alpha.kubernetes.io/ttl\\\":{},\\\"f:volumes.kubernetes.io/controller-managed-attach-detach\\\":{}},\\\"f:labels\\\":{\\\".\\\":{},\\\"f:beta.kubernetes.io/arch\\\":{},\\\"f:beta.kubernetes.io/instance-type\\\":{},\\\"f:beta.kubernetes.io/os\\\":{},\\\"f:cluster-name\\\":{},\\\"f:kubernetes.io/arch\\\":{},\\\"f:kubernetes.io/hostname\\\":{},\\\"f:kubernetes.io/os\\\":{},\\\"f:node-group\\\":{},\\\"f:node-role\\\":{},\\\"f:node.kubernetes.io/instance-type\\\":{}}},\\\"f:spec\\\":{\\\"f:podCIDR\\\":{},\\\"f:podCIDRs\\\":{\\\".\\\":{},\\\"v:\\\\\\\"10.128.10.0/24\\\\\\\"\\\":{}},\\\"f:providerID\\\":{}}}B\\000\\212\\001Y\\n\\027cilium-operator-generic\\022\\006Update\\032\\002v1\\\"\\010\\010\\350\\300\\371\\305\\006\\020\\0002\\010FieldsV1:\\034\\n\\032{\\\"f:spec\\\":{\\\"f:taints\\\":{}}}B\\000\\212\\001\\352\\005\\n\\003k3s\\022\\006Update\\032\\002v1\\\"\\010\\010\\327\\273\\376\\305\\006\\020\\0002\\010FieldsV1:\\272\\005\\n\\267\\005{\\\"f:status\\\":{\\\"f:addresses\\\":{\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"Hostname\\\\\\\"}\\\":{\\\"f:address\\\":{}}},\\\"f:allocatable\\\":{\\\"f:ephemeral-storage\\\":{}},\\\"f:capacity\\\":{\\\"f:ephemeral-storage\\\":{}},\\\"f:conditions\\\":{\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"DiskPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"MemoryPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"PIDPressure\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{}},\\\"k:{\\\\\\\"type\\\\\\\":\\\\\\\"Ready\\\\\\\"}\\\":{\\\"f:lastHeartbeatTime\\\":{},\\\"f:lastTransitionTime\\\":{},\\\"f:message\\\":{},\\\"f:reason\\\":{},\\\"f:status\\\":{}}},\\\"f:images\\\":{}}}B\\006status\\022h\\n\\01610.128.10.0/24\\022\\000\\032$k3s://prod-agent-app-blue-4-79b5072b \\000*\\034\\n\\tnode-role\\022\\003app\\032\\nNoSchedule:\\01610.128.10.0/24\\032\\222_\\n\\013\\n\\003cpu\\022\\004\\n\\00216\\n\\\"\\n\\021ephemeral-storage\\022\\r\\n\\013152558036Ki\\n\\024\\n\\rhugepages-1Gi\\022\\003\\n\\0010\\n\\024\\n\\rhugepages-2Mi\\022\\003\\n\\0010\\n\\026\\n\\006memory\\022\\014\\n\\n32860592Ki\\n\\r\\n\\004pods\\022\\005\\n\\003110\\022\\013\\n\\003cpu\\022\\004\\n\\00216\\022#\\n\\021ephemeral-storage\\022\\016\\n\\014148408457305\\022\\024\\n\\rhugepages-1Gi\\022\\003\\n\\0010\\022\\024\\n\\rhugepages-2Mi\\022\\003\\n\\0010\\022\\026\\n\\006memory\\022\\014\\n\\n32860592Ki\\022\\r\\n\\004pods\\022\\005\\n\\003110\\032\\000\\\"[\\n\\022NetworkUnavailable\\022\\005False\\032\\010\\010\\313\\253\\302\\266\\006\\020\\000\\\"\\010\\010\\313\\253\\302\\266\\006\\020\\000*\\nCiliumIsUp2\\036Cilium is running on this node\\\"p\\n\\016MemoryPressure\\022\\005False\\032\\010\\010\\327\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\314\\225\\237\\303\\006\\020\\000*\\032KubeletHasSufficientMemory2'kubelet has sufficient memory available\\\"a\\n\\014DiskPressure\\022\\005False\\032\\010\\010\\327\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\314\\225\\237\\303\\006\\020\\000*\\030KubeletHasNoDiskPressure2\\034kubelet has no disk pressure\\\"g\\n\\013PIDPressure\\022\\005False\\032\\010\\010\\327\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\314\\225\\237\\303\\006\\020\\000*\\027KubeletHasSufficientPID2$kubelet has sufficient PID available\\\"P\\n\\005Ready\\022\\004True\\032\\010\\010\\327\\273\\376\\305\\006\\020\\000\\\"\\010\\010\\305\\317\\267\\305\\006\\020\\000*\\014KubeletReady2\\037kubelet is posting ready status*\\031\\n\\nInternalIP\\022\\01310.6.61.165*!\\n\\010Hostname\\022\\025prod-agent-app-blue-42\\005\\n\\003\\010\\212P:\\332\\001\\n 930d76008d2c408e9ba36a050eb87a99\\022$b7fe0a42-93cc-7bdf-2c37-c4a90247fb6a\\032$32d85074-bb01-4dc5-89a8-d9126a9426b3\\\"\\0225.15.0-119-generic*\\022Ubuntu 22.04.4 LTS2\\030containerd://1.7.17-k3s1:\\014v1.30.3+k3s1B\\014v1.30.3+k3s1J\\005linuxR\\005amd64B\\332\\001\\n\\205\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/fr-local-cb@sha256:33c39330003adb18c6f8b3ccb31edf917b4e0e8d9da6e92e83f275408a72984b\\nJ868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/fr-local-cb:main-eac4bec\\020\\354\\341\\310\\247\\017B\\225\\001\\ndpublic.ecr.aws/datadog/agent@sha256:5a9fec9959f86c72ea1e7d8c3e77e6782d9538a0f417cf75ed20585f031e18bb\\n'public.ecr.aws/datadog/agent:7.56.2-jmx\\020\\366\\300\\300\\362\\001B\\225\\001\\ndpublic.ecr.aws/datadog/agent@sha256:465da828947d6c055eacf399df8d0f96423f6e029fd1218a806f091a0dc0200b\\n'public.ecr.aws/datadog/agent:7.62.0-jmx\\020\\207\\330\\325\\347\\001B\\334\\001\\n}868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops@sha256:307c60e7c0a570a7bbad2a45e1315f5ec6c39f6aea2b3d7dc521b57219fd73f1\\nU868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops:docker-wazuh-agent-f8b1f9f-main\\020\\311\\253\\321\\342\\001B\\334\\001\\n}868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops@sha256:e88bf7e606df57a9e7c3efa1f489132a72ad76f981bf7419e9598ce83ff9005c\\nU868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/ops:docker-wazuh-agent-a62b8c9-main\\020\\323\\304\\323\\335\\001B\\225\\001\\ndpublic.ecr.aws/datadog/agent@sha256:30601ca41c4afc566d3096ef58155df2f08707ced7ec2039c3e31f98478cbe0a\\n'public.ecr.aws/datadog/agent:7.44.1-jmx\\020\\206\\352\\375\\332\\001B\\327\\001\\n\\201\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/keyraze@sha256:5f22250e0a7b0d7719c96ddb6758834f424c8fc0c2e9c44333fc4a751d10d6df\\nK868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/keyraze:main-prod-deed74d\\020\\234\\356\\250\\332\\001B\\367\\001\\n\\203\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/my-verify@sha256:df85c286be8c9eb33832b2c2e32ec271cb8c1f0b00f608715ba28135cf121a50\\ni868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/my-verify:VENGG-13259-Integrate-inhouse-forgery-af56720\\020\\365\\356\\200\\325\\001B\\344\\001\\n\\205\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-message@sha256:47566e664e91773ce767788ca565a23c4c7ec3775f247d4a88b072606a545c43\\nT868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-message:release_1.74.0-56a9cb5\\020\\236\\276\\265\\320\\001B\\352\\001\\n\\206\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-document@sha256:04e8535931db5678b952d7253389ccea91b5af5c0c9eefb1800fd7716b3a8b6f\\nY868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-document:release_1.72.0-new-a8cc0b4\\020\\377\\210\\247\\304\\001B\\346\\001\\n\\206\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-document@sha256:3e94f56ca6e92460258648d1752b4def1c25bcbd2b022ee32fef1116ed5c8c34\\nU868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-document:release_1.70.1-a0623fa\\020\\304\\206\\245\\304\\001B\\346\\001\\n\\206\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-document@sha256:a2e085f19e8c59b3c999768585ee5709de2b2bafcd0fec7c40cfb8184f1f6a71\\nU868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-document:release_1.69.0-d31b528\\020\\276\\331\\242\\304\\001B\\342\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-core@sha256:5ae7d4c1b380062bfe5d8aab06aea2cebef89d09575b6b36b730befb572dd60c\\nU868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-core:release_1.72.0-new-a8cc0b4\\020\\327\\364\\322\\275\\001B\\336\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-core@sha256:afb5e2f74c718ee7843785757941293a925b69bb7022b4bc28234104e277fb84\\nQ868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-core:release_1.72.1-8f208a6\\020\\201\\363\\322\\275\\001B\\342\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-core@sha256:424ebc45cb1bcf3407c3b5bd71c99c8243dbbae4baee338b58115c13c50fa39e\\nU868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-core:release_1.72.0-new-4cc1e39\\020\\301\\361\\322\\275\\001B\\336\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-core@sha256:a3c05fbf3dc38808d07b59c6019a923c825d592c04626018c8406b618c5af1d3\\nQ868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-core:release_1.70.2-1b196cb\\020\\235\\356\\317\\275\\001B\\336\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-core@sha256:159d25b01eb5c7b60db4a489c5cd9dcbd9804b1294007f4e332f4a8cfecec2cf\\nQ868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/vsp-core:release_1.70.0-8858da8\\020\\323\\314\\317\\275\\001B\\324\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:0dc95b72ac3fc9cb24109b15e0b43061adc5846fab1707851f0d7fc4a845118c\\nG868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:main-e16099e\\020\\354\\266\\243\\274\\001B\\324\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:0c789a7a006f94a593b8189f799e000aa1a7a64c682059c62cd8f22bce99fde0\\nG868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:main-f702d9d\\020\\330\\363\\237\\274\\001B\\340\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:4f240d438b7fe32a964782c78604b55e8eca87e14fd59e418f5a624d10c4fc86\\nS868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:chore_delay-logs-d8ed80b\\020\\363\\340\\240\\273\\001B\\324\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:d9a69d9af2161826a11323bd004344d44049790933d004ff8edd7e6b6732b8a0\\nG868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:main-57ab919\\020\\324\\335\\240\\273\\001B\\340\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:1a087a05b31ddb7711ecf60f250da7baf1a251f8808472ec7e29732a2309d054\\nS868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:chore_delay-logs-af95be2\\020\\270\\333\\240\\273\\001B\\340\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:033a4e0db0c0989f9e95c47ef1bb710f243e72821ca4633bfb2c5eb137da20f6\\nS868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:chore_delay-logs-203266f\\020\\273\\241\\222\\273\\001B\\340\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:f5d089586b7708f89eabfe23bcbe6d5beb4f4abd691d090a8a5540f8943d2b47\\nS868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:chore_delay-logs-8ffc3b1\\020\\266\\241\\222\\273\\001B\\367\\001\\n\\217\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-partnermanagement@sha256:5f21763d201b14c5b50cfa2d4aadda17635e49843cb614e81c3d478063c663b4\\n]868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-partnermanagement:k8s-readiness-32e110e\\020\\264\\212\\302\\270\\001B\\324\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:7344ce2523ab0fe2181586aeb992d9f0ebb3449113b5fe49fc50ac18f18e95d0\\nG868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:main-ea99f8a\\020\\305\\275\\204\\265\\001B\\324\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:7937be166e92e6f33d50ef3b13b6ccaa7a1b1cafe3b6e4a81a2e428df8421679\\nG868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:main-6cea337\\020\\301\\272\\204\\265\\001B\\324\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:cd52d2be71f7ef8696ea6323feef167293be1fbbc636353d5cb38cc5f8e98c65\\nG868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:main-6255db8\\020\\203\\220\\204\\265\\001B\\324\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:a7be267d611910ce1f0b3cf58fbf80e55c4ec354c2ffc37458e38c49444b345b\\nG868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:main-a47ab67\\020\\263\\205\\204\\265\\001B\\324\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:25bd3689d3052229459149c2a58e4dff19547a9246b7e9d24ce6d17d1e959d9d\\nG868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:main-52f2006\\020\\335\\202\\204\\265\\001B\\324\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:cf9a8d0a78297951b2652e9116ec95717cbef3bf2d4c97ab81948f7fd5445932\\nG868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:main-e29c9ad\\020\\312\\357\\203\\265\\001B\\341\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:d647d3144172eb60aafcf7f92619d7c038ce2948e612a8a80212e8c9f36580e7\\nT868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:hotfix_sdkchanges-f573d6e\\020\\210\\336\\203\\265\\001B\\341\\001\\n\\202\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main@sha256:cda6a128567dae731b6ca76d18068a00513056bb4a89706c9e1e6fed5d4dfaf9\\nT868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-main:hotfix_sdkchanges-3ac4ba5\\020\\232\\331\\203\\265\\001B\\361\\001\\n\\210\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness@sha256:116d6a8301bc6c1a0fab7464af6c41937239f23286dfead0667550b5b8896175\\n^868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness:hotfix_sentinelConfig-0bf0777\\020\\225\\267\\274\\264\\001B\\362\\001\\n\\210\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness@sha256:aa06eb689cd95f6b6fd34f3bad0de593cc8e09cb789d22a3b96181d915220123\\n_868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness:feature_redis-sentinel-42351a9\\020\\332\\244\\274\\264\\001B\\361\\001\\n\\210\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness@sha256:6f5f374248bf43d175ca5c52b58d3f9528eac719e1474834347e5bb978298579\\n^868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness:hotfix_sentinelConfig-3a99d7f\\020\\322\\244\\274\\264\\001B\\340\\001\\n\\210\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness@sha256:052f62d1487fcf8e071284586faf2281c5731171b952e799b2d11dc48bba7898\\nM868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness:main-471cd63\\020\\320\\244\\274\\264\\001B\\340\\001\\n\\210\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness@sha256:75a70b2aae2c0311113e357bed0893bed30f04efa0f085ec358c45e27995e2fb\\nM868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness:main-f1e2f87\\020\\317\\314\\271\\264\\001B\\340\\001\\n\\210\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness@sha256:83eb3b4e5e303032e478968f93cdfa7132bb89a2cb1d0c703233ff60b6a5bd76\\nM868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness:main-d65b986\\020\\341\\313\\271\\264\\001B\\350\\001\\n\\213\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/id-verify-service@sha256:0392325fa95bcbc923e09516c0f83d8c284fcffd816eacf17dd1a7a17e563a6c\\nR868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/id-verify-service:master-41f4f54\\020\\332\\277\\371\\262\\001B\\350\\001\\n\\213\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/id-verify-service@sha256:f22bb5e58e53af4d682d05a75f0a4ab0c5fa72fe53376f4d15548c08e96d1e7b\\nR868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/id-verify-service:master-8658ddc\\020\\367\\275\\371\\262\\001B\\350\\001\\n\\213\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/id-verify-service@sha256:fa5e3fb4e3cc16c65bbc0d89f63ce939e98a7e2282cef4503eae7291b5632557\\nR868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/id-verify-service:master-53fbf06\\020\\305\\260\\371\\262\\001B\\322\\001\\n\\201\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/asg-neo@sha256:69a3f62b738496edecf8f441dfe969d58e98ae5cef96f57d64d215e190b940f0\\nF868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/asg-neo:main-ee168ee\\020\\355\\261\\322\\261\\001B\\340\\001\\n\\210\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness@sha256:a0163de6e05fe7e7078fddbc4b297443bbc1197a6d4ecdbc6bf153bde27e3252\\nM868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness:main-8720248\\020\\226\\245\\337\\260\\001B\\340\\001\\n\\210\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness@sha256:7e8c9a5d1d8f9f543aee870e7bdef68d6839029c6059a29206acd14e7944e78a\\nM868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness:main-7b403e1\\020\\325\\231\\337\\260\\001B\\340\\001\\n\\210\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness@sha256:036f6910f716d550336de1caa6be75ea11903e8b49a4c5990c7f6b5f6687ce24\\nM868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness:main-03f3866\\020\\222\\313\\306\\260\\001B\\340\\001\\n\\210\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness@sha256:edd965bcdf3ee69252b5aed73b825ed6f2b312c94dd84399ac25ba4ff0cddd08\\nM868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness:main-fadb921\\020\\357\\251\\206\\260\\001B\\340\\001\\n\\210\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness@sha256:1f060e5023780ba6a8c8f979b4bc64e766526869a878d29a1896e2b165434e0c\\nM868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness:main-920230e\\020\\300\\251\\206\\260\\001B\\340\\001\\n\\210\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness@sha256:dd53fea65e1ad5ba4d257d34513a6ac1d7bab1f9565adfc181e0485e455849e5\\nM868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness:main-393d20a\\020\\346\\344\\355\\257\\001B\\340\\001\\n\\210\\001868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness@sha256:82b9304326b29ebe2417fdc816034074713842d96f02784cd51b8fbb33edad74\\nM868948896061.dkr.ecr.ap-southeast-3.amazonaws.com/idi-liveliness:main-09b4d06\\020\\225\\340\\355\\257\\001\\032\\000\\\"\\000\" > > failure: > : context deadline exceeded" Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:22.840848167 +0000 UTC m=+607.355620885) (total time: 54.842267399s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=54.842267399s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:12.833716123 +0000 UTC m=+657.348488806) (total time: 4.908011652s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.908011652s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-1-a234b216: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=320959216 key=/registry/leases/kube-node-lease/prod-agent-app-blue-1-a234b216 previousRevision=653344667 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/masterleases/10.6.61.142: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=653338986 key=/registry/masterleases/10.6.61.142 previousRevision=653344655 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:12.23176206 +0000 UTC m=+656.746534779) (total time: 5.51365662s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=5.51365662s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-system/cert-manager-controller: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=303124872 key=/registry/leases/kube-system/cert-manager-controller previousRevision=653344707 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:13.17837757 +0000 UTC m=+657.693150287) (total time: 4.571444719s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.571444719s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-ingress-nginx-secondary-ingress-nginx-secondary-controller: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=537632387 key=/registry/leases/kube-system/cilium-l2announce-ingress-nginx-secondary-ingress-nginx-secondary-controller previousRevision=653344732 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:12.851840778 +0000 UTC m=+657.366613488) (total time: 4.898321862s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.898321862s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-10-1ef6ca3c: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=613356118 key=/registry/leases/kube-node-lease/prod-agent-app-blue-10-1ef6ca3c previousRevision=653344688 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:13.039564859 +0000 UTC m=+657.554337577) (total time: 4.711291411s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.711291411s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-6-420706e2: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=624036980 key=/registry/leases/kube-node-lease/prod-agent-data-green-6-420706e2 previousRevision=653344661 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:13.088951219 +0000 UTC m=+657.603723903) (total time: 4.662351299s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.662351299s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-3-47319b7a: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=624036915 key=/registry/leases/kube-node-lease/prod-agent-data-green-3-47319b7a previousRevision=653344684 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:55.205433104 +0000 UTC m=+639.720205795) (total time: 22.545997681s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=22.545997681s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/ingress-nginx-secondary/ingress-nginx-secondary-leader: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=474540760 key=/registry/leases/ingress-nginx-secondary/ingress-nginx-secondary-leader previousRevision=653344730 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:13.018427675 +0000 UTC m=+657.533200392) (total time: 4.733561402s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.733561402s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-5-f52e9ef0: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=624037756 key=/registry/leases/kube-node-lease/prod-agent-data-green-5-f52e9ef0 previousRevision=653344682 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:13.084609726 +0000 UTC m=+657.599382403) (total time: 4.667909785s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.667909785s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=537633548 key=/registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master previousRevision=653344740 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:13.080404883 +0000 UTC m=+657.595177561) (total time: 4.672297452s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.672297452s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-7-93605edd: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=624037614 key=/registry/leases/kube-node-lease/prod-agent-data-green-7-93605edd previousRevision=653344691 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:13.082793733 +0000 UTC m=+657.597566449) (total time: 4.670930763s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.670930763s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:15.249978454 +0000 UTC m=+659.764751138) (total time: 2.503762729s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=2.503762729s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-ingress-nginx-ingress-nginx-controller: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=545796592 key=/registry/leases/kube-system/cilium-l2announce-ingress-nginx-ingress-nginx-controller previousRevision=653344728 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-5-bb69767d: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=320959392 key=/registry/leases/kube-node-lease/prod-agent-app-blue-5-bb69767d previousRevision=653344674 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:16.147335063 +0000 UTC m=+660.662107780) (total time: 1.606556292s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=1.606556292s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-ingress-nginx-secondary-ingress-nginx-secondary-controller: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=537632387 key=/registry/leases/kube-system/cilium-l2announce-ingress-nginx-secondary-ingress-nginx-secondary-controller previousRevision=653344732 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:16.375605677 +0000 UTC m=+660.890378396) (total time: 1.37835222s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=1.37835222s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-ingress-nginx-ingress-nginx-controller: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=545796592 key=/registry/leases/kube-system/cilium-l2announce-ingress-nginx-ingress-nginx-controller previousRevision=653344728 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:12.83358621 +0000 UTC m=+657.348358892) (total time: 4.92129908s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.92129908s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-9-63cdf3ff: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=613356165 key=/registry/leases/kube-node-lease/prod-agent-app-blue-9-63cdf3ff previousRevision=653344696 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:11.861144449 +0000 UTC m=+656.375917207) (total time: 5.894438606s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=5.894438606s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-server-blue-k3s-01: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=320940478 key=/registry/leases/kube-node-lease/prod-server-blue-k3s-01 previousRevision=653344634 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:07.672016633 +0000 UTC m=+652.186789330) (total time: 10.08587949s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=10.08587949s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:02.918310538 +0000 UTC m=+647.433083259) (total time: 14.844200585s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=14.844200585s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/ingress-nginx/ingress-nginx-leader: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=98641045 key=/registry/leases/ingress-nginx/ingress-nginx-leader previousRevision=653344729 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:08.238444996 +0000 UTC m=+652.753217714) (total time: 9.524171561s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.524171561s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:07.018106301 +0000 UTC m=+651.532878998) (total time: 10.744879764s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=10.744879764s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:16.438744833 +0000 UTC m=+660.953517592) (total time: 1.324359793s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=1.324359793s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:08.135345822 +0000 UTC m=+652.650118552) (total time: 9.627700524s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=9.627700524s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/ingress-nginx/ingress-nginx-leader: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=98641045 key=/registry/leases/ingress-nginx/ingress-nginx-leader previousRevision=653344729 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:56.183565446 +0000 UTC m=+640.698338153) (total time: 21.579847798s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=21.579847798s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:12.811906813 +0000 UTC m=+657.326679571) (total time: 4.952706172s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.952706172s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ops-blue-1-89a427cc: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=321006889 key=/registry/leases/kube-node-lease/prod-agent-ops-blue-1-89a427cc previousRevision=653344659 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:55.3160377 +0000 UTC m=+639.830810383) (total time: 22.448997027s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=22.448997027s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:09.672012517 +0000 UTC m=+654.186785237) (total time: 8.093033258s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=8.093033258s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:13.076510741 +0000 UTC m=+657.591283419) (total time: 4.688614112s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.688614112s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-6-cd93959b: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=320959048 key=/registry/leases/kube-node-lease/prod-agent-app-blue-6-cd93959b previousRevision=653344672 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:15.115372697 +0000 UTC m=+659.630145504) (total time: 2.649867338s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=2.649867338s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-4-96deb91b: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=624037787 key=/registry/leases/kube-node-lease/prod-agent-data-green-4-96deb91b previousRevision=653344689 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:59.812945226 +0000 UTC m=+644.327717984) (total time: 17.95233069s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=17.95233069s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-system/apiserver-sbiwrpumfte5iyfxhikdbh2e3u: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=320940109 key=/registry/leases/kube-system/apiserver-sbiwrpumfte5iyfxhikdbh2e3u previousRevision=653344687 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:15.260556709 +0000 UTC m=+659.775329416) (total time: 2.504605261s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=2.504605261s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=537633548 key=/registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master previousRevision=653344740 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:13.083560957 +0000 UTC m=+657.598333678) (total time: 4.681772448s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.681772448s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-2-9c964185: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=320960197 key=/registry/leases/kube-node-lease/prod-agent-app-blue-2-9c964185 previousRevision=653344675 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:13.080481253 +0000 UTC m=+657.595253931) (total time: 4.685224946s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.685224946s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:13.08600918 +0000 UTC m=+657.600781915) (total time: 4.679741096s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.679741096s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-data-green-0-9b98de05: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=623966046 key=/registry/leases/kube-node-lease/prod-agent-data-green-0-9b98de05 previousRevision=653344666 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ops-blue-0-4b80a6a8: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=321006849 key=/registry/leases/kube-node-lease/prod-agent-ops-blue-0-4b80a6a8 previousRevision=653344656 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:16.510454219 +0000 UTC m=+661.025226939) (total time: 1.25549512s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=1.25549512s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/ingress-nginx-secondary/ingress-nginx-secondary-leader: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=474540760 key=/registry/leases/ingress-nginx-secondary/ingress-nginx-secondary-leader previousRevision=653344730 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:14.786266456 +0000 UTC m=+659.301039175) (total time: 2.97971332s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=2.97971332s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=537633548 key=/registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master previousRevision=653344740 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:12.850624369 +0000 UTC m=+657.365397062) (total time: 4.915501374s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.915501374s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ingress-blue-2-39068466: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=320947107 key=/registry/leases/kube-node-lease/prod-agent-ingress-blue-2-39068466 previousRevision=653344680 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:13.282283278 +0000 UTC m=+657.797056016) (total time: 4.483961016s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.483961016s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-operator-resource-lock: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=706 key=/registry/leases/kube-system/cilium-operator-resource-lock previousRevision=653344739 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:16.24373286 +0000 UTC m=+660.758505601) (total time: 1.522393141s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=1.522393141s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=537633548 key=/registry/leases/kube-system/cilium-l2announce-redis-ha-redis-master previousRevision=653344740 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:11.220368331 +0000 UTC m=+655.735141087) (total time: 6.546103097s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=6.546103097s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:53.334189193 +0000 UTC m=+637.848961922) (total time: 24.432107074s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=24.432107074s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/encryptor-system/93f9fcca.encryptor.dev: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=91237952 key=/registry/leases/encryptor-system/93f9fcca.encryptor.dev previousRevision=653344741 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:52.633665307 +0000 UTC m=+637.148438066) (total time: 25.133567313s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=25.133567313s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-system/k3s: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=234 key=/registry/leases/kube-system/k3s previousRevision=653344737 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:12.856063368 +0000 UTC m=+657.370836073) (total time: 4.912452455s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.912452455s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-ingress-blue-1-4323cca2: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=320946796 key=/registry/leases/kube-node-lease/prod-agent-ingress-blue-1-4323cca2 previousRevision=653344650 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:12.155181454 +0000 UTC m=+656.669954161) (total time: 5.613571859s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=5.613571859s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:12.220878309 +0000 UTC m=+656.735651044) (total time: 5.548333076s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=5.548333076s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-system/cert-manager-cainjector-leader-election: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=303128948 key=/registry/leases/kube-system/cert-manager-cainjector-leader-election previousRevision=653344706 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:15.296727214 +0000 UTC m=+659.811499976) (total time: 2.47266371s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=2.47266371s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/configmaps/ops/datadog-leader-election: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=4206015 key=/registry/configmaps/ops/datadog-leader-election previousRevision=653344733 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:24:11.6060138 +0000 UTC m=+656.120786517) (total time: 6.163361503s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=6.163361503s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=info msg="Slow SQL (started: 2025-09-09 03:24:13.066501646 +0000 UTC m=+657.581274366) (total time: 4.702960297s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=4.702960297s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/kube-node-lease/prod-agent-app-blue-7-c3860444: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=320959253 key=/registry/leases/kube-node-lease/prod-agent-app-blue-7-c3860444 previousRevision=653344660 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=warning msg="Slow SQL (started: 2025-09-09 03:23:58.507351277 +0000 UTC m=+643.022124022) (total time: 19.263871533s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id" duration=19.263871533s Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=error msg="insert error for key /registry/leases/ingress-nginx/ingress-nginx-leader: ERROR: duplicate key value violates unique constraint \"kine_name_prev_revision_uindex\" (SQLSTATE 23505)" createRevision=98641045 key=/registry/leases/ingress-nginx/ingress-nginx-leader previousRevision=653344729 Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: E0909 03:24:17.772389 983458 leaderelection.go:308] Failed to release lock: Operation cannot be fulfilled on leases.coordination.k8s.io "k3s": the object has been modified; please apply your changes to the latest version and try again Sep 09 03:24:17 prod-server-blue-k3s-01 k3s[983458]: time="2025-09-09T03:24:17Z" level=fatal msg="leaderelection lost for k3s" Sep 09 03:24:18 prod-server-blue-k3s-01 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE Sep 09 03:24:18 prod-server-blue-k3s-01 systemd[1]: k3s.service: Failed with result 'exit-code'. Sep 09 03:24:18 prod-server-blue-k3s-01 systemd[1]: k3s.service: Unit process 1494444 (containerd-shim) remains running after unit stopped. Sep 09 03:24:18 prod-server-blue-k3s-01 systemd[1]: k3s.service: Unit process 2743661 (containerd-shim) remains running after unit stopped. Sep 09 03:24:18 prod-server-blue-k3s-01 systemd[1]: k3s.service: Unit process 1493406 (containerd-shim) remains running after unit stopped. Sep 09 03:24:18 prod-server-blue-k3s-01 systemd[1]: k3s.service: Unit process 572824 (containerd-shim) remains running after unit stopped. Sep 09 03:24:18 prod-server-blue-k3s-01 systemd[1]: k3s.service: Unit process 697953 (containerd-shim) remains running after unit stopped. Sep 09 03:24:18 prod-server-blue-k3s-01 systemd[1]: k3s.service: Unit process 698000 (containerd-shim) remains running after unit stopped. Sep 09 03:24:18 prod-server-blue-k3s-01 systemd[1]: k3s.service: Unit process 773509 (containerd-shim) remains running after unit stopped. Sep 09 03:24:18 prod-server-blue-k3s-01 systemd[1]: k3s.service: Consumed 1min 42.541s CPU time. Sep 09 03:24:23 prod-server-blue-k3s-01 systemd[1]: k3s.service: Scheduled restart job, restart counter is at 2. Sep 09 03:24:23 prod-server-blue-k3s-01 systemd[1]: Stopped Lightweight Kubernetes. Sep 09 03:24:23 prod-server-blue-k3s-01 systemd[1]: k3s.service: Consumed 1min 42.560s CPU time.