2025-02-19 03:55:33,472 INFO akka.remote.Remoting ] - Remoting started; listening on addresses :[akka.tcp://flink@example-cluster-v2.flink-operator:6123] 2025-02-19 03:55:33,873 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils ] - Actor system started at akka.tcp://flink@example-cluster-v2.flink-operator:6123 2025-02-19 03:55:33,963 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager ] - Loading delegation token providers 2025-02-19 03:55:33,968 INFO org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvider ] - Hadoop FS is not available (not packaged with this application): NoClassDefFoundError : "org/apache/hadoop/conf/Configuration". 2025-02-19 03:55:33,969 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager ] - Delegation token provider hadoopfs loaded and initialized 2025-02-19 03:55:33,973 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager ] - Delegation token provider hbase loaded and initialized 2025-02-19 03:55:33,974 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: external-resource-gpu 2025-02-19 03:55:33,974 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: metrics-datadog 2025-02-19 03:55:33,974 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: metrics-graphite 2025-02-19 03:55:33,975 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: metrics-influx 2025-02-19 03:55:33,975 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: metrics-jmx 2025-02-19 03:55:33,975 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: metrics-prometheus 2025-02-19 03:55:33,976 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: metrics-slf4j 2025-02-19 03:55:33,976 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: metrics-statsd 2025-02-19 03:55:33,977 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: flink-s3-fs-hadoop 2025-02-19 03:55:33,982 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager ] - Delegation token provider s3 loaded and initialized 2025-02-19 03:55:33,982 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager ] - Delegation token providers loaded successfully 2025-02-19 03:55:33,983 INFO org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository ] - Loading delegation token receivers 2025-02-19 03:55:34,064 INFO org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository ] - Delegation token receiver hadoopfs loaded and initialized 2025-02-19 03:55:34,065 INFO org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository ] - Delegation token receiver hbase loaded and initialized 2025-02-19 03:55:34,065 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: external-resource-gpu 2025-02-19 03:55:34,066 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: metrics-datadog 2025-02-19 03:55:34,066 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: metrics-graphite 2025-02-19 03:55:34,066 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: metrics-influx 2025-02-19 03:55:34,067 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: metrics-jmx 2025-02-19 03:55:34,067 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: metrics-prometheus 2025-02-19 03:55:34,067 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: metrics-slf4j 2025-02-19 03:55:34,067 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: metrics-statsd 2025-02-19 03:55:34,068 INFO org.apache.flink.core.plugin.DefaultPluginManager ] - Plugin loader with ID found, reusing it: flink-s3-fs-hadoop 2025-02-19 03:55:34,070 INFO org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository ] - Delegation token receiver s3 loaded and initialized 2025-02-19 03:55:34,070 INFO org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository ] - Delegation token receivers loaded successfully 2025-02-19 03:55:34,070 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager ] - Checking provider and receiver instances consistency 2025-02-19 03:55:34,071 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager ] - Provider and receiver instances are consistent 2025-02-19 03:55:34,071 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager ] - Obtaining delegation tokens 2025-02-19 03:55:34,074 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager ] - Delegation tokens obtained successfully 2025-02-19 03:55:34,074 WARN org.apache.flink.runtime.security.token.DefaultDelegationTokenManager ] - No tokens obtained so skipping notifications 2025-02-19 03:55:34,088 INFO org.apache.flink.configuration.Configuration ] - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address' 2025-02-19 03:55:34,171 INFO org.apache.flink.runtime.blob.BlobServer ] - Created BLOB server storage directory /tmp/jm_91a14533899567cf539de7d3621ac4b2/blobStorage 2025-02-19 03:55:34,177 INFO org.apache.flink.runtime.blob.BlobServer ] - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000 2025-02-19 03:55:34,272 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl ] - No metrics reporter configured, no metrics will be exposed/reported. 2025-02-19 03:55:34,277 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils ] - Trying to start actor system, external address example-cluster-v2.flink-operator:0, bind address 0.0.0.0:0. 2025-02-19 03:55:34,379 INFO akka.event.slf4j.Slf4jLogger ] - Slf4jLogger started 2025-02-19 03:55:34,385 INFO akka.remote.RemoteActorRefProvider ] - Akka Cluster not in use - enabling unsafe features anyway because `akka.remote.use-unsafe-remote-features-outside-cluster` has been enabled. 2025-02-19 03:55:34,385 INFO akka.remote.Remoting ] - Starting remoting 2025-02-19 03:55:34,487 INFO akka.remote.Remoting ] - Remoting started; listening on addresses :[akka.tcp://flink-metrics@example-cluster-v2.flink-operator:33155] 2025-02-19 03:55:34,573 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils ] - Actor system started at akka.tcp://flink-metrics@example-cluster-v2.flink-operator:33155 2025-02-19 03:55:34,674 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService ] - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/rpc/MetricQueryService . 2025-02-19 03:55:34,775 INFO org.apache.flink.runtime.dispatcher.FileExecutionGraphInfoStore ] - Initializing FileExecutionGraphInfoStore: Storage directory /tmp/executionGraphStore-0e667b91-c332-4916-b809-215901c26868, expiration time 3600000, maximum cache size 52428800 bytes. 2025-02-19 03:55:35,265 INFO org.apache.flink.configuration.Configuration ] - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address' 2025-02-19 03:55:35,374 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint ] - Upload directory /tmp/flink-web-631db4a7-2852-4dc0-bf5f-dd59d718807e/flink-web-upload does not exist. 2025-02-19 03:55:35,375 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint ] - Created directory /tmp/flink-web-631db4a7-2852-4dc0-bf5f-dd59d718807e/flink-web-upload for file uploads. 2025-02-19 03:55:35,379 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint ] - Starting rest endpoint. 2025-02-19 03:55:35,969 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils ] - Determined location of main cluster component log file: /opt/flink/log/flink--kubernetes-session-0-example-cluster-v2-5b9cb75884-x898h.log 2025-02-19 03:55:35,970 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils ] - Determined location of main cluster component stdout file: /opt/flink/log/flink--kubernetes-session-0-example-cluster-v2-5b9cb75884-x898h.out 2025-02-19 03:55:36,473 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint ] - Rest endpoint listening at example-cluster-v2.flink-operator:8081 2025-02-19 03:55:36,475 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint ] - http://example-cluster-v2.flink-operator:8081 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000 2025-02-19 03:55:36,476 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint ] - Web frontend listening at http://example-cluster-v2.flink-operator:8081. 2025-02-19 03:55:36,580 INFO org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunner ] - DefaultDispatcherRunner was granted leadership with leader id 00000000-0000-0000-0000-000000000000. Creating new DispatcherLeaderProcess. 2025-02-19 03:55:36,662 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess ] - Start SessionDispatcherLeaderProcess. 2025-02-19 03:55:36,664 INFO org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl ] - Starting resource manager service. 2025-02-19 03:55:36,671 INFO org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl ] - Resource manager service is granted leadership with session id 00000000-0000-0000-0000-000000000000. 2025-02-19 03:55:36,672 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess ] - Recover all persisted job graphs that are not finished, yet. 2025-02-19 03:55:36,676 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess ] - Successfully recovered 0 persisted job graphs. 2025-02-19 03:55:36,873 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService ] - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/rpc/dispatcher_0 . 2025-02-19 03:55:39,561 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: blob.server.port, 6124 2025-02-19 03:55:39,562 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: kubernetes.jobmanager.annotations, flinkdeployment.flink.apache.org/generation:9 2025-02-19 03:55:39,562 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: kubernetes.jobmanager.replicas, 1 2025-02-19 03:55:39,562 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: logging.level, DEBUG 2025-02-19 03:55:39,562 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: jobmanager.rpc.address, example-cluster-v2.flink-operator 2025-02-19 03:55:39,562 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: kubernetes.service-account, flink 2025-02-19 03:55:39,563 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: kubernetes.cluster-id, example-cluster-v2 2025-02-19 03:55:39,563 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: kubernetes.taskmanager.cpu.amount, 0.5 2025-02-19 03:55:39,564 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: parallelism.default, 1 2025-02-19 03:55:39,564 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: kubernetes.namespace, flink-operator 2025-02-19 03:55:39,564 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: taskmanager.numberOfTaskSlots, 2 2025-02-19 03:55:39,566 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: fs.s3a.aws.credentials.provider, com.amazonaws.auth.WebIdentityTokenCredentialsProvider 2025-02-19 03:55:39,566 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: kubernetes.rest-service.exposed.type, ClusterIP 2025-02-19 03:55:39,566 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: kubernetes.jobmanager.owner.reference, uid:e8a496b7-8e08-46d3-a374-4f5479da71d1,name:example-cluster-v2,controller:false,blockOwnerDeletion:true,apiVersion:flink.apache.org/v1beta1,kind:FlinkDeployment 2025-02-19 03:55:39,567 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: kubernetes.container.image.ref, flink:1.17 2025-02-19 03:55:39,567 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: taskmanager.memory.process.size, 2 gb 2025-02-19 03:55:39,568 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint 2025-02-19 03:55:39,569 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: kubernetes.pod-template-file.taskmanager, /tmp/flink_op_generated_podTemplate_9853055710725970067.yaml 2025-02-19 03:55:39,569 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: fs.s3.impl, org.apache.hadoop.fs.s3a.S3AFileSystem 2025-02-19 03:55:39,569 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: web.cancel.enable, false 2025-02-19 03:55:39,569 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: execution.target, kubernetes-session 2025-02-19 03:55:39,570 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: jobmanager.memory.process.size, 2 gb 2025-02-19 03:55:39,570 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: taskmanager.rpc.port, 6122 2025-02-19 03:55:39,570 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: kubernetes.jobmanager.cpu.amount, 0.5 2025-02-19 03:55:39,570 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: internal.cluster.execution-mode, NORMAL 2025-02-19 03:55:39,571 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: $internal.flink.version, v1_17 2025-02-19 03:55:39,571 INFO org.apache.flink.configuration.GlobalConfiguration ] - Loading configuration property: kubernetes.pod-template-file.jobmanager, /tmp/flink_op_generated_podTemplate_3363561739573238384.yaml 2025-02-19 03:55:39,579 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService ] - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at akka://flink/user/rpc/resourcemanager_1 . 2025-02-19 03:55:39,664 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager ] - Starting the resource manager. 2025-02-19 03:55:39,678 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager ] - Starting tokens update task 2025-02-19 03:55:39,679 WARN org.apache.flink.runtime.security.token.DefaultDelegationTokenManager ] - No tokens obtained so skipping notifications 2025-02-19 03:55:39,679 WARN org.apache.flink.runtime.security.token.DefaultDelegationTokenManager ] - Tokens update task not started because either no tokens obtained or none of the tokens specified its renewal date 2025-02-19 03:55:43,866 INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver ] - Recovered 0 pods from previous attempts, current attempt id is 1. 2025-02-19 03:55:43,866 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager ] - Recovered 0 workers from previous attempt. 2025-02-19 03:55:48,674 WARN org.apache.flink.runtime.webmonitor.retriever.impl.RpcGatewayRetriever ] - Error while retrieving the leader gateway. Retrying to connect to akka.tcp://flink@example-cluster-v2.flink-operator:6123/user/rpc/resourcemanager_*. 2025-02-19 03:55:48,682 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagersHandler ] - Exception occurred in REST handler: Cannot connect to ResourceManager right now. Please try to refresh.