kubectl describe pod -n pulsar pulsar-bookie-2 Name: pulsar-bookie-2 Namespace: pulsar Priority: 0 Node: ip-10-163-195-148.us-west-2.compute.internal/10.163.195.148 Start Time: Tue, 14 Nov 2023 16:40:46 +0000 Labels: app=pulsar cluster=pulsar component=bookie controller-revision-hash=pulsar-bookie-79489dd45c release=pulsar statefulset.kubernetes.io/pod-name=pulsar-bookie-2 Annotations: prometheus.io/port: 8000 prometheus.io/scrape: true Status: Running IP: 10.163.201.26 IPs: IP: 10.163.201.26 Controlled By: StatefulSet/pulsar-bookie Init Containers: pulsar-bookkeeper-verify-clusterid: Container ID: containerd://e004ef3d6b1c622bb81ec5fcac1a4cb92a3959cad18959bfd9ccf5feb3a632df Image: artifactory.cwantools.io:5000/apachepulsar/pulsar:2.9.2 Image ID: artifactory.cwantools.io:5000/apachepulsar/pulsar@sha256:73d5c73509c8be9b324888baaab988393c365814ce744cf03ae33d98692da9c2 Port: Host Port: Command: sh -c Args: set -e; bin/apply-config-from-env.py conf/bookkeeper.conf;until bin/bookkeeper shell whatisinstanceid; do sleep 3; done; State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 14 Nov 2023 16:41:41 +0000 Finished: Tue, 14 Nov 2023 16:41:43 +0000 Ready: True Restart Count: 0 Environment Variables from: pulsar-bookie ConfigMap Optional: false Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-l5jfr (ro) Containers: pulsar-bookie: Container ID: containerd://b6e3b33f27c6d46400f9eda75a57c91652be5806e0fd8cb1e3174c4570e1f9a4 Image: artifactory.cwantools.io:5000/apachepulsar/pulsar:2.9.2 Image ID: artifactory.cwantools.io:5000/apachepulsar/pulsar@sha256:73d5c73509c8be9b324888baaab988393c365814ce744cf03ae33d98692da9c2 Ports: 3181/TCP, 8000/TCP Host Ports: 0/TCP, 0/TCP Command: sh -c Args: bin/apply-config-from-env.py conf/bookkeeper.conf; OPTS="${OPTS} -Dlog4j2.formatMsgNoLookups=true" exec bin/pulsar bookie; State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 2 Started: Tue, 14 Nov 2023 17:03:24 +0000 Finished: Tue, 14 Nov 2023 17:03:26 +0000 Ready: False Restart Count: 9 Requests: cpu: 200m memory: 128Mi Liveness: http-get http://:8000/api/v1/bookie/state delay=10s timeout=5s period=30s #success=1 #failure=60 Readiness: http-get http://:8000/api/v1/bookie/is_ready delay=10s timeout=5s period=30s #success=1 #failure=60 Environment Variables from: pulsar-bookie ConfigMap Optional: false Environment: Mounts: /pulsar/data/bookkeeper/journal from pulsar-bookie-journal (rw) /pulsar/data/bookkeeper/ledgers from pulsar-bookie-ledgers (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-l5jfr (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: pulsar-bookie-journal: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pulsar-bookie-journal-pulsar-bookie-2 ReadOnly: false pulsar-bookie-ledgers: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pulsar-bookie-ledgers-pulsar-bookie-2 ReadOnly: false kube-api-access-l5jfr: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: karpenter.sh/capacity-type=on-demand Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 23m default-scheduler Successfully assigned pulsar/pulsar-bookie-2 to ip-10-163-195-148.us-west-2.compute.internal Warning FailedAttachVolume 23m attachdetach-controller Multi-Attach error for volume "pvc-ac1cac87-ad2d-4165-aa22-d9f2f19615a4" Volume is already exclusively attached to one node and can't be attached to another Warning FailedAttachVolume 23m attachdetach-controller Multi-Attach error for volume "pvc-f9066e2d-5a72-49c3-a6f8-34618b7f4d7d" Volume is already exclusively attached to one node and can't be attached to another Normal SuccessfulAttachVolume 23m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-f9066e2d-5a72-49c3-a6f8-34618b7f4d7d" Normal SuccessfulAttachVolume 23m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-ac1cac87-ad2d-4165-aa22-d9f2f19615a4" Normal Pulling 23m kubelet Pulling image "artifactory.cwantools.io:5000/apachepulsar/pulsar:2.9.2" Normal Pulled 22m kubelet Successfully pulled image "artifactory.cwantools.io:5000/apachepulsar/pulsar:2.9.2" in 26.55001331s (26.550028058s including waiting) Normal Created 22m kubelet Created container pulsar-bookkeeper-verify-clusterid Normal Started 22m kubelet Started container pulsar-bookkeeper-verify-clusterid Normal Pulled 21m (x4 over 22m) kubelet Container image "artifactory.cwantools.io:5000/apachepulsar/pulsar:2.9.2" already present on machine Normal Created 21m (x4 over 22m) kubelet Created container pulsar-bookie Normal Started 21m (x4 over 22m) kubelet Started container pulsar-bookie Warning BackOff 3m14s (x95 over 22m) kubelet Back-off restarting failed container