/opt/flink/bin/config-parser-utils.sh: line 45: /opt/flink/conf/flink-conf.yaml: Read-only file system Starting kubernetes-session as a console application on host flink-trades-788d66787-k2h2f. 2024-08-21 16:56:32,390 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -------------------------------------------------------------------------------- 2024-08-21 16:56:32,392 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Preconfiguration: 2024-08-21 16:56:32,392 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - RESOURCE_PARAMS extraction logs: jvm_params: -Xmx4315938809 -Xms4315938809 -XX:MaxMetaspaceSize=268435456 dynamic_configs: -D jobmanager.memory.off-heap.size=134217728b -D jobmanager.memory.jvm-overhead.min=524288007b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=4315938809b -D jobmanager.memory.jvm-overhead.max=524288007b logs: WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. INFO [] - Using legacy YAML parser to load flink configuration file from /opt/flink/conf/flink-conf.yaml. INFO [] - Loading configuration property: blob.server.port, 6124 INFO [] - Loading configuration property: taskmanager.memory.process.size, 20000 mb INFO [] - Loading configuration property: kubernetes.jobmanager.annotations, flinkdeployment.flink.apache.org/generation:3 INFO [] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint INFO [] - Loading configuration property: kubernetes.jobmanager.replicas, 1 INFO [] - Loading configuration property: jobmanager.rpc.address, flink-trades.flink-operator INFO [] - Loading configuration property: web.cancel.enable, false INFO [] - Loading configuration property: execution.target, kubernetes-session INFO [] - Loading configuration property: jobmanager.memory.process.size, 5000 mb INFO [] - Loading configuration property: kubernetes.service-account, flink INFO [] - Loading configuration property: kubernetes.cluster-id, flink-trades INFO [] - Loading configuration property: taskmanager.rpc.port, 6122 INFO [] - Loading configuration property: kubernetes.jobmanager.cpu.amount, 5.0 INFO [] - Loading configuration property: kubernetes.taskmanager.cpu.amount, 10.0 INFO [] - Loading configuration property: internal.cluster.execution-mode, NORMAL INFO [] - Loading configuration property: parallelism.default, 1 INFO [] - Loading configuration property: kubernetes.namespace, flink-operator INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1 INFO [] - Loading configuration property: kubernetes.rest-service.exposed.type, ClusterIP INFO [] - Loading configuration property: kubernetes.jobmanager.owner.reference, blockOwnerDeletion:true,controller:false,name:flink-trades,uid:2edf921d-f233-48c0-b5ca-6798c468e4e5,kind:FlinkDeployment,apiVersion:flink.apache.org/v1beta1 INFO [] - Loading configuration property: $internal.flink.version, v1_19 INFO [] - Loading configuration property: kubernetes.pod-template-file.jobmanager, /tmp/flink_op_generated_podTemplate_10469306648539520813.yaml INFO [] - Loading configuration property: kubernetes.container.image.ref, 862064919227.dkr.ecr.me-south-1.amazonaws.com/mdu/analytics/flink:test-print2 INFO [] - Final Master Memory configuration: INFO [] - Total Process Memory: 4.883gb (5242880000 bytes) INFO [] - Total Flink Memory: 4.145gb (4450156537 bytes) INFO [] - JVM Heap: 4.020gb (4315938809 bytes) INFO [] - Off-heap: 128.000mb (134217728 bytes) INFO [] - JVM Metaspace: 256.000mb (268435456 bytes) INFO [] - JVM Overhead: 500.000mb (524288007 bytes) 2024-08-21 16:56:32,392 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -------------------------------------------------------------------------------- 2024-08-21 16:56:32,392 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting KubernetesSessionClusterEntrypoint (Version: 1.19.1, Scala: 2.12, Rev:5edb5a9, Date:2024-06-06T14:45:33+02:00) 2024-08-21 16:56:32,392 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current user: flink 2024-08-21 16:56:32,392 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current Hadoop/Kerberos user: 2024-08-21 16:56:32,392 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM: OpenJDK 64-Bit Server VM - Eclipse Adoptium - 11/11.0.24+8 2024-08-21 16:56:32,392 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Arch: amd64 2024-08-21 16:56:32,392 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum heap size: 4116 MiBytes 2024-08-21 16:56:32,393 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME: /opt/java/openjdk 2024-08-21 16:56:32,393 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No Hadoop Dependency available 2024-08-21 16:56:32,393 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM Options: 2024-08-21 16:56:32,393 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xmx4315938809 2024-08-21 16:56:32,393 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xms4315938809 2024-08-21 16:56:32,393 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -XX:MaxMetaspaceSize=268435456 2024-08-21 16:56:32,393 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -XX:+IgnoreUnrecognizedVMOptions 2024-08-21 16:56:32,393 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog.file=/opt/flink/log/flink--kubernetes-session-0-flink-trades-788d66787-k2h2f.log 2024-08-21 16:56:32,393 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties 2024-08-21 16:56:32,393 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties 2024-08-21 16:56:32,393 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml 2024-08-21 16:56:32,393 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program Arguments: 2024-08-21 16:56:32,394 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D 2024-08-21 16:56:32,394 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.off-heap.size=134217728b 2024-08-21 16:56:32,394 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D 2024-08-21 16:56:32,394 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-overhead.min=524288007b 2024-08-21 16:56:32,394 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D 2024-08-21 16:56:32,394 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-metaspace.size=268435456b 2024-08-21 16:56:32,395 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D 2024-08-21 16:56:32,395 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.heap.size=4315938809b 2024-08-21 16:56:32,395 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D 2024-08-21 16:56:32,395 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-overhead.max=524288007b 2024-08-21 16:56:32,395 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath: /opt/flink/lib/flink-cep-1.19.1.jar:/opt/flink/lib/flink-connector-files-1.19.1.jar:/opt/flink/lib/flink-csv-1.19.1.jar:/opt/flink/lib/flink-json-1.19.1.jar:/opt/flink/lib/flink-scala_2.12-1.19.1.jar:/opt/flink/lib/flink-table-api-java-uber-1.19.1.jar:/opt/flink/lib/flink-table-planner-loader-1.19.1.jar:/opt/flink/lib/flink-table-runtime-1.19.1.jar:/opt/flink/lib/log4j-1.2-api-2.17.1.jar:/opt/flink/lib/log4j-api-2.17.1.jar:/opt/flink/lib/log4j-core-2.17.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.17.1.jar:/opt/flink/lib/flink-dist-1.19.1.jar:::: 2024-08-21 16:56:32,395 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -------------------------------------------------------------------------------- 2024-08-21 16:56:32,396 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered UNIX signal handlers for [TERM, HUP, INT] 2024-08-21 16:56:32,403 INFO org.apache.flink.configuration.GlobalConfiguration [] - Using legacy YAML parser to load flink configuration file from /opt/flink/conf/flink-conf.yaml. 2024-08-21 16:56:32,403 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: blob.server.port, 6124 2024-08-21 16:56:32,403 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 20000 mb 2024-08-21 16:56:32,404 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.jobmanager.annotations, flinkdeployment.flink.apache.org/generation:3 2024-08-21 16:56:32,404 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint 2024-08-21 16:56:32,404 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.jobmanager.replicas, 1 2024-08-21 16:56:32,404 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, flink-trades.flink-operator 2024-08-21 16:56:32,404 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: web.cancel.enable, false 2024-08-21 16:56:32,404 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.target, kubernetes-session 2024-08-21 16:56:32,404 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 5000 mb 2024-08-21 16:56:32,404 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.service-account, flink 2024-08-21 16:56:32,404 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.cluster-id, flink-trades 2024-08-21 16:56:32,404 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.rpc.port, 6122 2024-08-21 16:56:32,404 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.jobmanager.cpu.amount, 5.0 2024-08-21 16:56:32,404 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.taskmanager.cpu.amount, 10.0 2024-08-21 16:56:32,405 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: internal.cluster.execution-mode, NORMAL 2024-08-21 16:56:32,405 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1 2024-08-21 16:56:32,405 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.namespace, flink-operator 2024-08-21 16:56:32,405 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1 2024-08-21 16:56:32,405 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.rest-service.exposed.type, ClusterIP 2024-08-21 16:56:32,405 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.jobmanager.owner.reference, blockOwnerDeletion:true,controller:false,name:flink-trades,uid:2edf921d-f233-48c0-b5ca-6798c468e4e5,kind:FlinkDeployment,apiVersion:flink.apache.org/v1beta1 2024-08-21 16:56:32,405 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: $internal.flink.version, v1_19 2024-08-21 16:56:32,405 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.pod-template-file.jobmanager, /tmp/flink_op_generated_podTemplate_10469306648539520813.yaml 2024-08-21 16:56:32,405 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.container.image.ref, 862064919227.dkr.ecr.me-south-1.amazonaws.com/mdu/analytics/flink:test-print2 2024-08-21 16:56:32,405 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading dynamic configuration property: jobmanager.memory.off-heap.size, 134217728b 2024-08-21 16:56:32,405 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading dynamic configuration property: jobmanager.memory.jvm-overhead.min, 524288007b 2024-08-21 16:56:32,405 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading dynamic configuration property: jobmanager.memory.jvm-metaspace.size, 268435456b 2024-08-21 16:56:32,405 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading dynamic configuration property: jobmanager.memory.heap.size, 4315938809b 2024-08-21 16:56:32,406 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading dynamic configuration property: jobmanager.memory.jvm-overhead.max, 524288007b 2024-08-21 16:56:32,513 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting KubernetesSessionClusterEntrypoint. 2024-08-21 16:56:32,531 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install default filesystem. 2024-08-21 16:56:32,533 INFO org.apache.flink.core.fs.FileSystem [] - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available. 2024-08-21 16:56:32,544 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: external-resource-gpu 2024-08-21 16:56:32,548 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: metrics-datadog 2024-08-21 16:56:32,549 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: metrics-graphite 2024-08-21 16:56:32,549 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: metrics-influx 2024-08-21 16:56:32,549 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: metrics-jmx 2024-08-21 16:56:32,549 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: metrics-prometheus 2024-08-21 16:56:32,549 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: metrics-slf4j 2024-08-21 16:56:32,549 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID not found, creating it: metrics-statsd 2024-08-21 16:56:32,560 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install security context. 2024-08-21 16:56:32,569 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath. 2024-08-21 16:56:32,572 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-12613856461396464947.conf. 2024-08-21 16:56:32,576 INFO org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath. 2024-08-21 16:56:32,577 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Initializing cluster services. 2024-08-21 16:56:32,581 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Using working directory: WorkingDirectory(/tmp/jm_84f8b21289ec89e9c6c5ca045c75b807). 2024-08-21 16:56:32,742 INFO org.apache.flink.runtime.rpc.pekko.PekkoRpcServiceUtils [] - Trying to start actor system, external address flink-trades.flink-operator:6123, bind address 0.0.0.0:6123. 2024-08-21 16:56:33,256 INFO org.apache.pekko.event.slf4j.Slf4jLogger [] - Slf4jLogger started 2024-08-21 16:56:33,288 INFO org.apache.pekko.remote.RemoteActorRefProvider [] - Pekko Cluster not in use - enabling unsafe features anyway because `pekko.remote.use-unsafe-remote-features-outside-cluster` has been enabled. 2024-08-21 16:56:33,288 INFO org.apache.pekko.remote.Remoting [] - Starting remoting 2024-08-21 16:56:33,397 INFO org.apache.pekko.remote.Remoting [] - Remoting started; listening on addresses :[pekko.tcp://flink@flink-trades.flink-operator:6123] 2024-08-21 16:56:33,470 INFO org.apache.flink.runtime.rpc.pekko.PekkoRpcServiceUtils [] - Actor system started at pekko.tcp://flink@flink-trades.flink-operator:6123 2024-08-21 16:56:33,481 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - Loading delegation token providers 2024-08-21 16:56:33,483 INFO org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvider [] - Hadoop FS is not available (not packaged with this application): NoClassDefFoundError : "org/apache/hadoop/conf/Configuration". 2024-08-21 16:56:33,483 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - Delegation token provider hadoopfs loaded and initialized 2024-08-21 16:56:33,484 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - Delegation token provider hbase loaded and initialized 2024-08-21 16:56:33,485 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: external-resource-gpu 2024-08-21 16:56:33,485 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: metrics-datadog 2024-08-21 16:56:33,485 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: metrics-graphite 2024-08-21 16:56:33,485 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: metrics-influx 2024-08-21 16:56:33,485 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: metrics-jmx 2024-08-21 16:56:33,485 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: metrics-prometheus 2024-08-21 16:56:33,485 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: metrics-slf4j 2024-08-21 16:56:33,485 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: metrics-statsd 2024-08-21 16:56:33,485 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - Delegation token providers loaded successfully 2024-08-21 16:56:33,486 INFO org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository [] - Loading delegation token receivers 2024-08-21 16:56:33,487 INFO org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository [] - Delegation token receiver hadoopfs loaded and initialized 2024-08-21 16:56:33,487 INFO org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository [] - Delegation token receiver hbase loaded and initialized 2024-08-21 16:56:33,487 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: external-resource-gpu 2024-08-21 16:56:33,487 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: metrics-datadog 2024-08-21 16:56:33,487 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: metrics-graphite 2024-08-21 16:56:33,487 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: metrics-influx 2024-08-21 16:56:33,488 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: metrics-jmx 2024-08-21 16:56:33,488 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: metrics-prometheus 2024-08-21 16:56:33,488 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: metrics-slf4j 2024-08-21 16:56:33,488 INFO org.apache.flink.core.plugin.DefaultPluginManager [] - Plugin loader with ID found, reusing it: metrics-statsd 2024-08-21 16:56:33,488 INFO org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository [] - Delegation token receivers loaded successfully 2024-08-21 16:56:33,488 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - Checking provider and receiver instances consistency 2024-08-21 16:56:33,488 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - Provider and receiver instances are consistent 2024-08-21 16:56:33,488 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - Obtaining delegation tokens 2024-08-21 16:56:33,490 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - Delegation tokens obtained successfully 2024-08-21 16:56:33,490 WARN org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - No tokens obtained so skipping notifications 2024-08-21 16:56:33,496 INFO org.apache.flink.configuration.Configuration [] - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address' 2024-08-21 16:56:33,501 INFO org.apache.flink.runtime.blob.BlobServer [] - Created BLOB server storage directory /tmp/jm_84f8b21289ec89e9c6c5ca045c75b807/blobStorage 2024-08-21 16:56:33,504 INFO org.apache.flink.runtime.blob.BlobServer [] - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000 2024-08-21 16:56:33,515 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl [] - No metrics reporter configured, no metrics will be exposed/reported. 2024-08-21 16:56:33,515 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl [] - No trace reporter configured, no metrics will be exposed/reported. 2024-08-21 16:56:33,518 INFO org.apache.flink.runtime.rpc.pekko.PekkoRpcServiceUtils [] - Trying to start actor system, external address flink-trades.flink-operator:0, bind address 0.0.0.0:0. 2024-08-21 16:56:33,532 INFO org.apache.pekko.event.slf4j.Slf4jLogger [] - Slf4jLogger started 2024-08-21 16:56:33,535 INFO org.apache.pekko.remote.RemoteActorRefProvider [] - Pekko Cluster not in use - enabling unsafe features anyway because `pekko.remote.use-unsafe-remote-features-outside-cluster` has been enabled. 2024-08-21 16:56:33,535 INFO org.apache.pekko.remote.Remoting [] - Starting remoting 2024-08-21 16:56:33,552 INFO org.apache.pekko.remote.Remoting [] - Remoting started; listening on addresses :[pekko.tcp://flink-metrics@flink-trades.flink-operator:42787] 2024-08-21 16:56:33,557 INFO org.apache.flink.runtime.rpc.pekko.PekkoRpcServiceUtils [] - Actor system started at pekko.tcp://flink-metrics@flink-trades.flink-operator:42787 2024-08-21 16:56:33,567 INFO org.apache.flink.runtime.rpc.pekko.PekkoRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at pekko://flink-metrics/user/rpc/MetricQueryService . 2024-08-21 16:56:33,583 INFO org.apache.flink.runtime.dispatcher.FileExecutionGraphInfoStore [] - Initializing FileExecutionGraphInfoStore: Storage directory /tmp/executionGraphStore-c4623187-4946-48fc-ac14-e413102ca386, expiration time 3600000, maximum cache size 52428800 bytes. 2024-08-21 16:56:33,623 INFO org.apache.flink.configuration.Configuration [] - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address' 2024-08-21 16:56:33,639 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Upload directory /tmp/flink-web-a40d277d-fce5-46ff-bea6-d81ca39b6eb9/flink-web-upload does not exist. 2024-08-21 16:56:33,639 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Created directory /tmp/flink-web-a40d277d-fce5-46ff-bea6-d81ca39b6eb9/flink-web-upload for file uploads. 2024-08-21 16:56:33,641 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Starting rest endpoint. 2024-08-21 16:56:33,711 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of main cluster component log file: /opt/flink/log/flink--kubernetes-session-0-flink-trades-788d66787-k2h2f.log 2024-08-21 16:56:33,711 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of main cluster component stdout file: /opt/flink/log/flink--kubernetes-session-0-flink-trades-788d66787-k2h2f.out 2024-08-21 16:56:33,785 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Rest endpoint listening at flink-trades.flink-operator:8081 2024-08-21 16:56:33,787 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - http://flink-trades.flink-operator:8081 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000 2024-08-21 16:56:33,787 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Web frontend listening at http://flink-trades.flink-operator:8081. 2024-08-21 16:56:33,796 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (1.953gb (2097152031 bytes)) is greater than its max value 1024.000mb (1073741824 bytes), max value will be used instead 2024-08-21 16:56:33,802 WARN org.apache.flink.kubernetes.entrypoint.KubernetesResourceManagerFactory [] - Configured size for 'taskmanager.memory.process.size' is ignored. Total memory size for TaskManagers are dynamically decided in fine-grained resource management. 2024-08-21 16:56:33,805 INFO org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunner [] - DefaultDispatcherRunner was granted leadership with leader id 00000000-0000-0000-0000-000000000000. Creating new DispatcherLeaderProcess. 2024-08-21 16:56:33,808 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Start SessionDispatcherLeaderProcess. 2024-08-21 16:56:33,809 INFO org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl [] - Starting resource manager service. 2024-08-21 16:56:33,812 INFO org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl [] - Resource manager service is granted leadership with session id 00000000-0000-0000-0000-000000000000. 2024-08-21 16:56:33,813 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Recover all persisted job graphs that are not finished, yet. 2024-08-21 16:56:33,813 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Successfully recovered 0 persisted job graphs. 2024-08-21 16:56:33,823 INFO org.apache.flink.runtime.rpc.pekko.PekkoRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at pekko://flink/user/rpc/dispatcher_0 . 2024-08-21 16:56:34,104 INFO org.apache.flink.configuration.GlobalConfiguration [] - Using legacy YAML parser to load flink configuration file from /opt/flink/conf/flink-conf.yaml. 2024-08-21 16:56:34,105 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: blob.server.port, 6124 2024-08-21 16:56:34,105 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 20000 mb 2024-08-21 16:56:34,105 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.jobmanager.annotations, flinkdeployment.flink.apache.org/generation:3 2024-08-21 16:56:34,105 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint 2024-08-21 16:56:34,105 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.jobmanager.replicas, 1 2024-08-21 16:56:34,105 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, flink-trades.flink-operator 2024-08-21 16:56:34,105 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: web.cancel.enable, false 2024-08-21 16:56:34,105 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.target, kubernetes-session 2024-08-21 16:56:34,105 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 5000 mb 2024-08-21 16:56:34,105 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.service-account, flink 2024-08-21 16:56:34,105 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.cluster-id, flink-trades 2024-08-21 16:56:34,105 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.rpc.port, 6122 2024-08-21 16:56:34,106 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.jobmanager.cpu.amount, 5.0 2024-08-21 16:56:34,106 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.taskmanager.cpu.amount, 10.0 2024-08-21 16:56:34,106 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: internal.cluster.execution-mode, NORMAL 2024-08-21 16:56:34,106 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1 2024-08-21 16:56:34,106 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.namespace, flink-operator 2024-08-21 16:56:34,106 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1 2024-08-21 16:56:34,106 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.rest-service.exposed.type, ClusterIP 2024-08-21 16:56:34,106 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.jobmanager.owner.reference, blockOwnerDeletion:true,controller:false,name:flink-trades,uid:2edf921d-f233-48c0-b5ca-6798c468e4e5,kind:FlinkDeployment,apiVersion:flink.apache.org/v1beta1 2024-08-21 16:56:34,106 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: $internal.flink.version, v1_19 2024-08-21 16:56:34,106 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.pod-template-file.jobmanager, /tmp/flink_op_generated_podTemplate_10469306648539520813.yaml 2024-08-21 16:56:34,106 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.container.image.ref, 862064919227.dkr.ecr.me-south-1.amazonaws.com/mdu/analytics/flink:test-print2 2024-08-21 16:56:34,109 INFO org.apache.flink.runtime.rpc.pekko.PekkoRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at pekko://flink/user/rpc/resourcemanager_1 . 2024-08-21 16:56:34,116 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Starting the resource manager. 2024-08-21 16:56:34,119 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Starting the slot manager. 2024-08-21 16:56:34,120 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - Starting tokens update task 2024-08-21 16:56:34,120 WARN org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - No tokens obtained so skipping notifications 2024-08-21 16:56:34,120 WARN org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - Tokens update task not started because either no tokens obtained or none of the tokens specified its renewal date 2024-08-21 16:56:34,588 INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Recovered 0 pods from previous attempts, current attempt id is 1. 2024-08-21 16:56:34,588 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Recovered 0 workers from previous attempt. 2024-08-21 16:56:41,825 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Exception occurred in REST handler: Job 5d246a975c7b43520000000000000001 not found 2024-08-21 16:56:41,875 ERROR org.apache.flink.runtime.rest.handler.job.JobExceptionsHandler [] - Exception occurred in REST handler: Job 5d246a975c7b43520000000000000001 not found 2024-08-21 16:58:40,275 ERROR org.apache.flink.runtime.rest.handler.job.JobCancellationHandler [] - Exception occurred in REST handler: Job could not be found. 2024-08-21 16:58:44,274 ERROR org.apache.flink.runtime.rest.handler.job.JobCancellationHandler [] - Exception occurred in REST handler: Job could not be found. 2024-08-21 16:58:50,334 ERROR org.apache.flink.runtime.rest.handler.job.JobCancellationHandler [] - Exception occurred in REST handler: Job could not be found. 2024-08-21 16:58:54,320 ERROR org.apache.flink.runtime.rest.handler.job.JobCancellationHandler [] - Exception occurred in REST handler: Job could not be found. 2024-08-21 16:59:07,617 INFO org.apache.flink.client.ClientUtils [] - Starting program (detached: true) 2024-08-21 16:59:07,618 INFO org.apache.flink.client.ClientUtils [] - Starting program (detached: true) 2024-08-21 16:59:07,675 INFO org.apache.flink.client.python.PythonEnvUtils [] - Starting Python process with environment variables: {PATH=/opt/flink/bin:/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT=tcp://10.249.108.78:443, FLINK_PLUGINS_DIR=/opt/flink/plugins, FLINK_TRADES_REST_PORT_8081_TCP_ADDR=10.249.244.127, GPG_KEY=6378E37EB3AAEA188B9CB0D396C2914BB78A5EA1, FLINK_TRADES_REST_SERVICE_PORT=8081, FLINK_TRADES_REST_PORT_8081_TCP=tcp://10.249.244.127:8081, FLINK_CONF_DIR=/opt/flink/conf, FLINK_ENV_JAVA_OPTS=-XX:+IgnoreUnrecognizedVMOptions, PWD=/opt/flink, FLINK_TRADES_REST_SERVICE_PORT_REST=8081, KUBERNETES_PORT_443_TCP=tcp://10.249.0.1:443, LANGUAGE=en_US:en, PYTHONPATH=/tmp/pyflink/98184c3a-056e-4362-9147-15c61d8f8cf9/5864e436-249f-4f7d-ace4-3803cf2f0200:/opt/flink/opt/python/cloudpickle-2.2.0-src.zip:/opt/flink/opt/python/py4j-0.10.9.7-src.zip:/opt/flink/opt/python/pyflink.zip:/opt/flink/usrlib/, FLINK_TRADES_REST_PORT_8081_TCP_PROTO=tcp, MAX_LOG_FILE_NUMBER=10, DYNAMIC_PARAMETERS=-D jobmanager.memory.off-heap.size=134217728b -D jobmanager.memory.jvm-overhead.min=524288007b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=4315938809b -D jobmanager.memory.jvm-overhead.max=524288007b, FLINK_TRADES_REST_PORT=tcp://10.249.244.127:8081, LC_ALL=en_US.UTF-8, KUBERNETES_SERVICE_PORT_HTTPS=443, SHLVL=0, FLINK_BIN_DIR=/opt/flink/bin, FLINK_OPERATOR_WEBHOOK_SERVICE_SERVICE_HOST=10.249.108.78, FLINK_TGZ_URL=https://dlcdn.apache.org/flink/flink-1.19.1/flink-1.19.1-bin-scala_2.12.tgz, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT_443_TCP_PORT=443, JAVA_RUN=/opt/java/openjdk/bin/java, KUBERNETES_PORT=tcp://10.249.0.1:443, ROOT_LOG_LEVEL=INFO, JAVA_HOME=/opt/java/openjdk, CHECK_GPG=true, _POD_IP_ADDRESS=100.72.94.76, KUBERNETES_SERVICE_HOST=10.249.0.1, LANG=en_US.UTF-8, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT_443_TCP_PROTO=tcp, FLINK_ASC_URL=https://downloads.apache.org/flink/flink-1.19.1/flink-1.19.1-bin-scala_2.12.tgz.asc, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT_443_TCP_ADDR=10.249.108.78, FLINK_OPERATOR_WEBHOOK_SERVICE_SERVICE_PORT=443, JAVA_VERSION=jdk-11.0.24+8, FLINK_OPT_DIR=/opt/flink/opt, KUBERNETES_PORT_443_TCP_ADDR=10.249.0.1, FLINK_HOME=/opt/flink, FLINK_TRADES_REST_SERVICE_HOST=10.249.244.127, KUBERNETES_PORT_443_TCP_PROTO=tcp, FLINK_LIB_DIR=/opt/flink/lib, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT_443_TCP=tcp://10.249.108.78:443, PYFLINK_GATEWAY_PORT=45931, KUBERNETES_SERVICE_PORT=443, FLINK_INHERITED_LOGS= RESOURCE_PARAMS extraction logs: jvm_params: -Xmx4315938809 -Xms4315938809 -XX:MaxMetaspaceSize=268435456 dynamic_configs: -D jobmanager.memory.off-heap.size=134217728b -D jobmanager.memory.jvm-overhead.min=524288007b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=4315938809b -D jobmanager.memory.jvm-overhead.max=524288007b logs: WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. INFO [] - Using legacy YAML parser to load flink configuration file from /opt/flink/conf/flink-conf.yaml. INFO [] - Loading configuration property: blob.server.port, 6124 INFO [] - Loading configuration property: taskmanager.memory.process.size, 20000 mb INFO [] - Loading configuration property: kubernetes.jobmanager.annotations, flinkdeployment.flink.apache.org/generation:3 INFO [] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint INFO [] - Loading configuration property: kubernetes.jobmanager.replicas, 1 INFO [] - Loading configuration property: jobmanager.rpc.address, flink-trades.flink-operator INFO [] - Loading configuration property: web.cancel.enable, false INFO [] - Loading configuration property: execution.target, kubernetes-session INFO [] - Loading configuration property: jobmanager.memory.process.size, 5000 mb INFO [] - Loading configuration property: kubernetes.service-account, flink INFO [] - Loading configuration property: kubernetes.cluster-id, flink-trades INFO [] - Loading configuration property: taskmanager.rpc.port, 6122 INFO [] - Loading configuration property: kubernetes.jobmanager.cpu.amount, 5.0 INFO [] - Loading configuration property: kubernetes.taskmanager.cpu.amount, 10.0 INFO [] - Loading configuration property: internal.cluster.execution-mode, NORMAL INFO [] - Loading configuration property: parallelism.default, 1 INFO [] - Loading configuration property: kubernetes.namespace, flink-operator INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1 INFO [] - Loading configuration property: kubernetes.rest-service.exposed.type, ClusterIP INFO [] - Loading configuration property: kubernetes.jobmanager.owner.reference, blockOwnerDeletion:true,controller:false,name:flink-trades,uid:2edf921d-f233-48c0-b5ca-6798c468e4e5,kind:FlinkDeployment,apiVersion:flink.apache.org/v1beta1 INFO [] - Loading configuration property: $internal.flink.version, v1_19 INFO [] - Loading configuration property: kubernetes.pod-template-file.jobmanager, /tmp/flink_op_generated_podTemplate_10469306648539520813.yaml INFO [] - Loading configuration property: kubernetes.container.image.ref, 862064919227.dkr.ecr.me-south-1.amazonaws.com/mdu/analytics/flink:test-print2 INFO [] - Final Master Memory configuration: INFO [] - Total Process Memory: 4.883gb (5242880000 bytes) INFO [] - Total Flink Memory: 4.145gb (4450156537 bytes) INFO [] - JVM Heap: 4.020gb (4315938809 bytes) INFO [] - Off-heap: 128.000mb (134217728 bytes) INFO [] - JVM Metaspace: 256.000mb (268435456 bytes) INFO [] - JVM Overhead: 500.000mb (524288007 bytes) , GOSU_VERSION=1.11, FLINK_TRADES_REST_PORT_8081_TCP_PORT=8081, HOSTNAME=flink-trades-788d66787-k2h2f, JVM_ARGS= -Xmx4315938809 -Xms4315938809 -XX:MaxMetaspaceSize=268435456, LD_PRELOAD=:/usr/lib/x86_64-linux-gnu/libjemalloc.so, KUBERNETES_PORT_443_TCP_PORT=443, HOME=/opt/flink}, command: python -u /opt/flink/usrlib/test.py 2024-08-21 16:59:07,675 INFO org.apache.flink.client.python.PythonEnvUtils [] - Starting Python process with environment variables: {PATH=/opt/flink/bin:/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT=tcp://10.249.108.78:443, FLINK_PLUGINS_DIR=/opt/flink/plugins, FLINK_TRADES_REST_PORT_8081_TCP_ADDR=10.249.244.127, GPG_KEY=6378E37EB3AAEA188B9CB0D396C2914BB78A5EA1, FLINK_TRADES_REST_SERVICE_PORT=8081, FLINK_TRADES_REST_PORT_8081_TCP=tcp://10.249.244.127:8081, FLINK_CONF_DIR=/opt/flink/conf, FLINK_ENV_JAVA_OPTS=-XX:+IgnoreUnrecognizedVMOptions, PWD=/opt/flink, FLINK_TRADES_REST_SERVICE_PORT_REST=8081, KUBERNETES_PORT_443_TCP=tcp://10.249.0.1:443, LANGUAGE=en_US:en, PYTHONPATH=/tmp/pyflink/ec657e49-0382-4c58-a64b-bccc56d27b61/33333b32-c018-4d66-bcc1-4ad566c459a8:/opt/flink/opt/python/cloudpickle-2.2.0-src.zip:/opt/flink/opt/python/py4j-0.10.9.7-src.zip:/opt/flink/opt/python/pyflink.zip:/opt/flink/usrlib/, FLINK_TRADES_REST_PORT_8081_TCP_PROTO=tcp, MAX_LOG_FILE_NUMBER=10, DYNAMIC_PARAMETERS=-D jobmanager.memory.off-heap.size=134217728b -D jobmanager.memory.jvm-overhead.min=524288007b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=4315938809b -D jobmanager.memory.jvm-overhead.max=524288007b, FLINK_TRADES_REST_PORT=tcp://10.249.244.127:8081, LC_ALL=en_US.UTF-8, KUBERNETES_SERVICE_PORT_HTTPS=443, SHLVL=0, FLINK_BIN_DIR=/opt/flink/bin, FLINK_OPERATOR_WEBHOOK_SERVICE_SERVICE_HOST=10.249.108.78, FLINK_TGZ_URL=https://dlcdn.apache.org/flink/flink-1.19.1/flink-1.19.1-bin-scala_2.12.tgz, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT_443_TCP_PORT=443, JAVA_RUN=/opt/java/openjdk/bin/java, KUBERNETES_PORT=tcp://10.249.0.1:443, ROOT_LOG_LEVEL=INFO, JAVA_HOME=/opt/java/openjdk, CHECK_GPG=true, _POD_IP_ADDRESS=100.72.94.76, KUBERNETES_SERVICE_HOST=10.249.0.1, LANG=en_US.UTF-8, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT_443_TCP_PROTO=tcp, FLINK_ASC_URL=https://downloads.apache.org/flink/flink-1.19.1/flink-1.19.1-bin-scala_2.12.tgz.asc, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT_443_TCP_ADDR=10.249.108.78, FLINK_OPERATOR_WEBHOOK_SERVICE_SERVICE_PORT=443, JAVA_VERSION=jdk-11.0.24+8, FLINK_OPT_DIR=/opt/flink/opt, KUBERNETES_PORT_443_TCP_ADDR=10.249.0.1, FLINK_HOME=/opt/flink, FLINK_TRADES_REST_SERVICE_HOST=10.249.244.127, KUBERNETES_PORT_443_TCP_PROTO=tcp, FLINK_LIB_DIR=/opt/flink/lib, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT_443_TCP=tcp://10.249.108.78:443, PYFLINK_GATEWAY_PORT=36139, KUBERNETES_SERVICE_PORT=443, FLINK_INHERITED_LOGS= RESOURCE_PARAMS extraction logs: jvm_params: -Xmx4315938809 -Xms4315938809 -XX:MaxMetaspaceSize=268435456 dynamic_configs: -D jobmanager.memory.off-heap.size=134217728b -D jobmanager.memory.jvm-overhead.min=524288007b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=4315938809b -D jobmanager.memory.jvm-overhead.max=524288007b logs: WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. INFO [] - Using legacy YAML parser to load flink configuration file from /opt/flink/conf/flink-conf.yaml. INFO [] - Loading configuration property: blob.server.port, 6124 INFO [] - Loading configuration property: taskmanager.memory.process.size, 20000 mb INFO [] - Loading configuration property: kubernetes.jobmanager.annotations, flinkdeployment.flink.apache.org/generation:3 INFO [] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint INFO [] - Loading configuration property: kubernetes.jobmanager.replicas, 1 INFO [] - Loading configuration property: jobmanager.rpc.address, flink-trades.flink-operator INFO [] - Loading configuration property: web.cancel.enable, false INFO [] - Loading configuration property: execution.target, kubernetes-session INFO [] - Loading configuration property: jobmanager.memory.process.size, 5000 mb INFO [] - Loading configuration property: kubernetes.service-account, flink INFO [] - Loading configuration property: kubernetes.cluster-id, flink-trades INFO [] - Loading configuration property: taskmanager.rpc.port, 6122 INFO [] - Loading configuration property: kubernetes.jobmanager.cpu.amount, 5.0 INFO [] - Loading configuration property: kubernetes.taskmanager.cpu.amount, 10.0 INFO [] - Loading configuration property: internal.cluster.execution-mode, NORMAL INFO [] - Loading configuration property: parallelism.default, 1 INFO [] - Loading configuration property: kubernetes.namespace, flink-operator INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1 INFO [] - Loading configuration property: kubernetes.rest-service.exposed.type, ClusterIP INFO [] - Loading configuration property: kubernetes.jobmanager.owner.reference, blockOwnerDeletion:true,controller:false,name:flink-trades,uid:2edf921d-f233-48c0-b5ca-6798c468e4e5,kind:FlinkDeployment,apiVersion:flink.apache.org/v1beta1 INFO [] - Loading configuration property: $internal.flink.version, v1_19 INFO [] - Loading configuration property: kubernetes.pod-template-file.jobmanager, /tmp/flink_op_generated_podTemplate_10469306648539520813.yaml INFO [] - Loading configuration property: kubernetes.container.image.ref, 862064919227.dkr.ecr.me-south-1.amazonaws.com/mdu/analytics/flink:test-print2 INFO [] - Final Master Memory configuration: INFO [] - Total Process Memory: 4.883gb (5242880000 bytes) INFO [] - Total Flink Memory: 4.145gb (4450156537 bytes) INFO [] - JVM Heap: 4.020gb (4315938809 bytes) INFO [] - Off-heap: 128.000mb (134217728 bytes) INFO [] - JVM Metaspace: 256.000mb (268435456 bytes) INFO [] - JVM Overhead: 500.000mb (524288007 bytes) , GOSU_VERSION=1.11, FLINK_TRADES_REST_PORT_8081_TCP_PORT=8081, HOSTNAME=flink-trades-788d66787-k2h2f, JVM_ARGS= -Xmx4315938809 -Xms4315938809 -XX:MaxMetaspaceSize=268435456, LD_PRELOAD=:/usr/lib/x86_64-linux-gnu/libjemalloc.so, KUBERNETES_PORT_443_TCP_PORT=443, HOME=/opt/flink}, command: python -u /opt/flink/usrlib/test.py 2024-08-21 16:59:07,680 INFO org.apache.flink.client.python.PythonDriver [] - --------------------------- Python Process Started -------------------------- 2024-08-21 16:59:07,680 INFO org.apache.flink.client.python.PythonDriver [] - --------------------------- Python Process Started -------------------------- 2024-08-21 16:59:08,947 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.ddl.SqlCreateCatalog does not contain a setter for field catalogName 2024-08-21 16:59:08,948 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.apache.flink.sql.parser.ddl.SqlCreateCatalog cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution. 2024-08-21 16:59:08,949 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.ddl.SqlCreateView does not contain a setter for field viewName 2024-08-21 16:59:08,949 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.apache.flink.sql.parser.ddl.SqlCreateView cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution. 2024-08-21 16:59:08,949 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.ddl.SqlAlterViewRename does not contain a getter for field newViewIdentifier 2024-08-21 16:59:08,949 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.ddl.SqlAlterViewRename does not contain a setter for field newViewIdentifier 2024-08-21 16:59:08,949 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.apache.flink.sql.parser.ddl.SqlAlterViewRename cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution. 2024-08-21 16:59:08,950 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.ddl.SqlAlterViewProperties does not contain a setter for field propertyList 2024-08-21 16:59:08,950 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.apache.flink.sql.parser.ddl.SqlAlterViewProperties cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution. 2024-08-21 16:59:08,950 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.ddl.SqlAlterViewAs does not contain a setter for field newQuery 2024-08-21 16:59:08,951 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.apache.flink.sql.parser.ddl.SqlAlterViewAs cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution. 2024-08-21 16:59:08,951 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.ddl.SqlAddPartitions does not contain a setter for field ifPartitionNotExists 2024-08-21 16:59:08,951 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.apache.flink.sql.parser.ddl.SqlAddPartitions cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution. 2024-08-21 16:59:08,951 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.ddl.SqlDropPartitions does not contain a setter for field ifExists 2024-08-21 16:59:08,952 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.apache.flink.sql.parser.ddl.SqlDropPartitions cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution. 2024-08-21 16:59:08,952 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.dql.SqlShowPartitions does not contain a getter for field tableIdentifier 2024-08-21 16:59:08,952 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.dql.SqlShowPartitions does not contain a setter for field tableIdentifier 2024-08-21 16:59:08,952 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.apache.flink.sql.parser.dql.SqlShowPartitions cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution. 2024-08-21 16:59:08,953 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.dml.SqlTruncateTable does not contain a getter for field tableNameIdentifier 2024-08-21 16:59:08,953 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.dml.SqlTruncateTable does not contain a setter for field tableNameIdentifier 2024-08-21 16:59:08,953 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.apache.flink.sql.parser.dml.SqlTruncateTable cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution. 2024-08-21 16:59:08,953 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.dql.SqlShowFunctions does not contain a setter for field requireUser 2024-08-21 16:59:08,953 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.apache.flink.sql.parser.dql.SqlShowFunctions cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution. 2024-08-21 16:59:08,954 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.dql.SqlShowProcedures does not contain a getter for field databaseName 2024-08-21 16:59:08,954 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.dql.SqlShowProcedures does not contain a setter for field databaseName 2024-08-21 16:59:08,954 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.apache.flink.sql.parser.dql.SqlShowProcedures cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution. 2024-08-21 16:59:08,955 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.ddl.SqlReplaceTableAs does not contain a setter for field tableName 2024-08-21 16:59:08,955 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.apache.flink.sql.parser.ddl.SqlReplaceTableAs cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution. 2024-08-21 16:59:08,956 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.apache.flink.sql.parser.dql.SqlShowDatabases does not contain a setter for field preposition 2024-08-21 16:59:08,956 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.apache.flink.sql.parser.dql.SqlShowDatabases cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution. 2024-08-21 16:59:09,948 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job 753fc574fd3d8b24491dea29d82753c5 is submitted. 2024-08-21 16:59:09,948 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=753fc574fd3d8b24491dea29d82753c5. 2024-08-21 16:59:09,948 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job 753fc574fd3d8b24491dea29d82753c5 is submitted. 2024-08-21 16:59:09,948 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=753fc574fd3d8b24491dea29d82753c5. 2024-08-21 16:59:10,126 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received JobGraph submission 'insert-into_default_catalog.default_database.print' (753fc574fd3d8b24491dea29d82753c5). 2024-08-21 16:59:10,128 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting job 'insert-into_default_catalog.default_database.print' (753fc574fd3d8b24491dea29d82753c5). 2024-08-21 16:59:10,138 INFO org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner [] - JobMasterServiceLeadershipRunner for job 753fc574fd3d8b24491dea29d82753c5 was granted leadership with leader id 00000000-0000-0000-0000-000000000000. Creating new JobMasterServiceProcess. 2024-08-21 16:59:10,145 INFO org.apache.flink.runtime.rpc.pekko.PekkoRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at pekko://flink/user/rpc/jobmanager_2 . 2024-08-21 16:59:10,150 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Initializing job 'insert-into_default_catalog.default_database.print' (753fc574fd3d8b24491dea29d82753c5). 2024-08-21 16:59:10,165 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received JobGraph submission 'insert-into_default_catalog.default_database.print' (753fc574fd3d8b24491dea29d82753c5). 2024-08-21 16:59:10,167 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart back off time strategy NoRestartBackoffTimeStrategy for insert-into_default_catalog.default_database.print (753fc574fd3d8b24491dea29d82753c5). Traceback (most recent call last): 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - Traceback (most recent call last): File "/opt/flink/usrlib/test.py", line 34, in 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - File "/opt/flink/usrlib/test.py", line 34, in stmt_set.execute() 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - stmt_set.execute() File "/opt/flink/opt/python/pyflink.zip/pyflink/table/statement_set.py", line 158, in execute 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - File "/opt/flink/opt/python/pyflink.zip/pyflink/table/statement_set.py", line 158, in execute File "/opt/flink/opt/python/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__ 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - File "/opt/flink/opt/python/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__ File "/opt/flink/opt/python/pyflink.zip/pyflink/util/exceptions.py", line 158, in deco 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - File "/opt/flink/opt/python/pyflink.zip/pyflink/util/exceptions.py", line 158, in deco pyflink.util.exceptions.TableException: org.apache.flink.table.api.TableException: Failed to execute sql 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - pyflink.util.exceptions.TableException: org.apache.flink.table.api.TableException: Failed to execute sql at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1060) 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1060) at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:876) 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:876) at org.apache.flink.table.api.internal.StatementSetImpl.execute(StatementSetImpl.java:109) 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.table.api.internal.StatementSetImpl.execute(StatementSetImpl.java:109) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) at org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282) 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282) at org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) 2024-08-21 16:59:10,174 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79) at org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238) at java.base/java.lang.Thread.run(Unknown Source) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.flink.util.FlinkException: Failed to execute job 'insert-into_default_catalog.default_database.print'. 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - Caused by: org.apache.flink.util.FlinkException: Failed to execute job 'insert-into_default_catalog.default_database.print'. at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2455) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2455) at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:188) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:188) at org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:110) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:110) at org.apache.flink.table.executor.python.ChainingOptimizingExecutor.executeAsync(ChainingOptimizingExecutor.java:88) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.table.executor.python.ChainingOptimizingExecutor.executeAsync(ChainingOptimizingExecutor.java:88) at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1032) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1032) ... 13 more 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - ... 13 more Caused by: org.apache.flink.runtime.client.DuplicateJobSubmissionException: Job has already been submitted. 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - Caused by: org.apache.flink.runtime.client.DuplicateJobSubmissionException: Job has already been submitted. at org.apache.flink.runtime.client.DuplicateJobSubmissionException.of(DuplicateJobSubmissionException.java:31) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.runtime.client.DuplicateJobSubmissionException.of(DuplicateJobSubmissionException.java:31) at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$submitJob$3(Dispatcher.java:538) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$submitJob$3(Dispatcher.java:538) at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source) at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRunAsync$4(PekkoRpcActor.java:451) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRunAsync$4(PekkoRpcActor.java:451) at org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRunAsync(PekkoRpcActor.java:451) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRunAsync(PekkoRpcActor.java:451) at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:218) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:218) at org.apache.flink.runtime.rpc.pekko.FencedPekkoRpcActor.handleRpcMessage(FencedPekkoRpcActor.java:85) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.runtime.rpc.pekko.FencedPekkoRpcActor.handleRpcMessage(FencedPekkoRpcActor.java:85) at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:168) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:168) at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33) at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29) at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) 2024-08-21 16:59:10,175 INFO org.apache.flink.client.python.PythonDriver [] - at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) at org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547) at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545) at org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229) at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590) at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557) at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:280) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:280) at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:241) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:241) at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253) at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) 2024-08-21 16:59:10,176 INFO org.apache.flink.client.python.PythonDriver [] - 2024-08-21 16:59:10,192 INFO org.apache.flink.client.python.PythonDriver [] - --------------------------- Python Process Exited --------------------------- 2024-08-21 16:59:10,196 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Created execution graph 46d1dd2c67f21363438cb461eb231750 for job 753fc574fd3d8b24491dea29d82753c5. 2024-08-21 16:59:10,193 ERROR org.apache.flink.client.python.PythonDriver [] - Run python process failed java.lang.RuntimeException: Python process exits with code: 1 at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:124) ~[c1b8fb3f-1542-43be-89c1-7b08f12771bd_flink-python-1.19.1.jar:1.19.1] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?] at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?] at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.19.1.jar:1.19.1] at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist-1.19.1.jar:1.19.1] at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:108) ~[flink-dist-1.19.1.jar:1.19.1] at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84) ~[flink-dist-1.19.1.jar:1.19.1] at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70) ~[flink-dist-1.19.1.jar:1.19.1] at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:108) ~[flink-dist-1.19.1.jar:1.19.1] at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?] 2024-08-21 16:59:10,201 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Exception occurred in REST handler: Could not execute application. 2024-08-21 16:59:10,210 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running initialization on master for job insert-into_default_catalog.default_database.print (753fc574fd3d8b24491dea29d82753c5). 2024-08-21 16:59:10,210 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Successfully ran initialization on master in 0 ms. 2024-08-21 16:59:10,222 INFO org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built 1 new pipelined regions in 1 ms, total 1 pipelined regions currently. 2024-08-21 16:59:10,226 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - State backend is set to heap memory org.apache.flink.runtime.state.hashmap.HashMapStateBackend@56e3aba5 2024-08-21 16:59:10,227 INFO org.apache.flink.runtime.state.StateBackendLoader [] - State backend loader loads the state backend as HashMapStateBackend 2024-08-21 16:59:10,227 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Checkpoint storage is set to 'jobmanager' 2024-08-21 16:59:10,248 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No checkpoint found during restore. 2024-08-21 16:59:10,253 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.RestartPipelinedRegionFailoverStrategy@704ac401 for insert-into_default_catalog.default_database.print (753fc574fd3d8b24491dea29d82753c5). 2024-08-21 16:59:10,261 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting execution of job 'insert-into_default_catalog.default_database.print' (753fc574fd3d8b24491dea29d82753c5) under job master id 00000000000000000000000000000000. 2024-08-21 16:59:10,263 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting scheduling with scheduling strategy [org.apache.flink.runtime.scheduler.strategy.PipelinedRegionSchedulingStrategy] 2024-08-21 16:59:10,263 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job insert-into_default_catalog.default_database.print (753fc574fd3d8b24491dea29d82753c5) switched from state CREATED to RUNNING. 2024-08-21 16:59:10,265 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: datagen[2] -> Sink: print[4] (1/1) (46d1dd2c67f21363438cb461eb231750_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from CREATED to SCHEDULED. 2024-08-21 16:59:10,275 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Connecting to ResourceManager pekko.tcp://flink@flink-trades.flink-operator:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000) 2024-08-21 16:59:10,278 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Resolved ResourceManager address, beginning registration 2024-08-21 16:59:10,279 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 00000000000000000000000000000000@pekko.tcp://flink@flink-trades.flink-operator:6123/user/rpc/jobmanager_2 for job 753fc574fd3d8b24491dea29d82753c5. 2024-08-21 16:59:10,282 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registered job manager 00000000000000000000000000000000@pekko.tcp://flink@flink-trades.flink-operator:6123/user/rpc/jobmanager_2 for job 753fc574fd3d8b24491dea29d82753c5. 2024-08-21 16:59:10,285 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - JobManager successfully registered at ResourceManager, leader id: 00000000000000000000000000000000. 2024-08-21 16:59:10,286 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Received resource requirements from job 753fc574fd3d8b24491dea29d82753c5: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}] 2024-08-21 16:59:10,321 INFO org.apache.flink.client.python.PythonDriver [] - --------------------------- Python Process Exited --------------------------- 2024-08-21 16:59:10,324 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Exception occurred in REST handler: No jobs included in application. 2024-08-21 16:59:10,356 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Matching resource requirements against available resources. Missing resources: Job 753fc574fd3d8b24491dea29d82753c5 ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1} Current resources: (none) 2024-08-21 16:59:10,427 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - need request 1 new workers, current worker number 0, declared worker number 1 2024-08-21 16:59:10,428 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (2.059gb (2210864278 bytes)) is greater than its max value 1024.000mb (1073741824 bytes), max value will be used instead 2024-08-21 16:59:10,428 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Requesting new worker with resource spec WorkerResourceSpec {cpuCores=10.0, taskHeapSize=8.891gb (9546235758 bytes), taskOffHeapSize=0 bytes, networkMemSize=1.828gb (1962934301 bytes), managedMemSize=7.313gb (7851737205 bytes), numSlots=1}, current pending count: 1. 2024-08-21 16:59:10,433 INFO org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled external resources: [] 2024-08-21 16:59:10,437 INFO org.apache.flink.configuration.Configuration [] - Config uses fallback configuration key 'kubernetes.service-account' instead of key 'kubernetes.taskmanager.service-account' 2024-08-21 16:59:10,445 INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Creating new TaskManager pod with name flink-trades-taskmanager-1-1 and resource <20000,10.0>. 2024-08-21 16:59:10,612 INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Pod flink-trades-taskmanager-1-1 is created. 2024-08-21 16:59:10,623 INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Received new TaskManager pod: flink-trades-taskmanager-1-1 2024-08-21 16:59:10,623 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Requested worker flink-trades-taskmanager-1-1 with resource spec WorkerResourceSpec {cpuCores=10.0, taskHeapSize=8.891gb (9546235758 bytes), taskOffHeapSize=0 bytes, networkMemSize=1.828gb (1962934301 bytes), managedMemSize=7.313gb (7851737205 bytes), numSlots=1}. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.jboss.netty.util.internal.ByteBufferUtil (file:/tmp/flink-rpc-akkaff894eb8-e061-45fe-918d-473e6ecf5bfd.jar) to method java.nio.DirectByteBuffer.cleaner() WARNING: Please consider reporting this to the maintainers of org.jboss.netty.util.internal.ByteBufferUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2024-08-21 16:59:15,499 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering TaskManager with ResourceID flink-trades-taskmanager-1-1 (pekko.tcp://flink@100.72.172.243:6122/user/rpc/taskmanager_0) at ResourceManager 2024-08-21 16:59:15,513 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Registering task executor flink-trades-taskmanager-1-1 under 87b3bc6ca7d6cb6e4e7c2a800db5a015 at the slot manager. 2024-08-21 16:59:15,515 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DefaultSlotStatusSyncer [] - Starting allocation of slot 0dc5c8067aefa9e2dc74fb937a1fdfa7 from flink-trades-taskmanager-1-1 for job 753fc574fd3d8b24491dea29d82753c5 with resource profile ResourceProfile{cpuCores=1E+1, taskHeapMemory=8.891gb (9546235758 bytes), taskOffHeapMemory=0 bytes, managedMemory=7.313gb (7851737205 bytes), networkMemory=1.828gb (1962934301 bytes)}. 2024-08-21 16:59:15,517 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Worker flink-trades-taskmanager-1-1 is registered. 2024-08-21 16:59:15,518 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Worker flink-trades-taskmanager-1-1 with resource spec WorkerResourceSpec {cpuCores=10.0, taskHeapSize=8.891gb (9546235758 bytes), taskOffHeapSize=0 bytes, networkMemSize=1.828gb (1962934301 bytes), managedMemSize=7.313gb (7851737205 bytes), numSlots=1} was requested in current attempt. Current pending count after registering: 0. 2024-08-21 16:59:15,567 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: datagen[2] -> Sink: print[4] (1/1) (46d1dd2c67f21363438cb461eb231750_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from SCHEDULED to DEPLOYING. 2024-08-21 16:59:15,574 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: datagen[2] -> Sink: print[4] (1/1) (attempt #0) with attempt id 46d1dd2c67f21363438cb461eb231750_cbc357ccb763df2852fee8c4fc7d55f2_0_0 and vertex id cbc357ccb763df2852fee8c4fc7d55f2_0 to flink-trades-taskmanager-1-1 @ ip-100-72-172-243.me-south-1.compute.internal (dataPort=34441) with allocation id 0dc5c8067aefa9e2dc74fb937a1fdfa7 2024-08-21 16:59:15,781 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: datagen[2] -> Sink: print[4] (1/1) (46d1dd2c67f21363438cb461eb231750_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from DEPLOYING to INITIALIZING. 2024-08-21 16:59:16,133 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: datagen[2] -> Sink: print[4] (1/1) (46d1dd2c67f21363438cb461eb231750_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from INITIALIZING to RUNNING. 2024-08-21 16:59:17,508 INFO org.apache.flink.client.ClientUtils [] - Starting program (detached: true) 2024-08-21 16:59:17,529 INFO org.apache.flink.client.python.PythonEnvUtils [] - Starting Python process with environment variables: {PATH=/opt/flink/bin:/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT=tcp://10.249.108.78:443, FLINK_PLUGINS_DIR=/opt/flink/plugins, FLINK_TRADES_REST_PORT_8081_TCP_ADDR=10.249.244.127, GPG_KEY=6378E37EB3AAEA188B9CB0D396C2914BB78A5EA1, FLINK_TRADES_REST_SERVICE_PORT=8081, FLINK_TRADES_REST_PORT_8081_TCP=tcp://10.249.244.127:8081, FLINK_CONF_DIR=/opt/flink/conf, FLINK_ENV_JAVA_OPTS=-XX:+IgnoreUnrecognizedVMOptions, PWD=/opt/flink, FLINK_TRADES_REST_SERVICE_PORT_REST=8081, KUBERNETES_PORT_443_TCP=tcp://10.249.0.1:443, LANGUAGE=en_US:en, PYTHONPATH=/tmp/pyflink/b655c77d-c382-43f8-a176-3729c7a38724/d0dbe28d-90e2-440d-b538-11384171c4a7:/opt/flink/opt/python/cloudpickle-2.2.0-src.zip:/opt/flink/opt/python/py4j-0.10.9.7-src.zip:/opt/flink/opt/python/pyflink.zip:/opt/flink/usrlib/, FLINK_TRADES_REST_PORT_8081_TCP_PROTO=tcp, MAX_LOG_FILE_NUMBER=10, DYNAMIC_PARAMETERS=-D jobmanager.memory.off-heap.size=134217728b -D jobmanager.memory.jvm-overhead.min=524288007b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=4315938809b -D jobmanager.memory.jvm-overhead.max=524288007b, FLINK_TRADES_REST_PORT=tcp://10.249.244.127:8081, LC_ALL=en_US.UTF-8, KUBERNETES_SERVICE_PORT_HTTPS=443, SHLVL=0, FLINK_BIN_DIR=/opt/flink/bin, FLINK_OPERATOR_WEBHOOK_SERVICE_SERVICE_HOST=10.249.108.78, FLINK_TGZ_URL=https://dlcdn.apache.org/flink/flink-1.19.1/flink-1.19.1-bin-scala_2.12.tgz, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT_443_TCP_PORT=443, JAVA_RUN=/opt/java/openjdk/bin/java, KUBERNETES_PORT=tcp://10.249.0.1:443, ROOT_LOG_LEVEL=INFO, JAVA_HOME=/opt/java/openjdk, CHECK_GPG=true, _POD_IP_ADDRESS=100.72.94.76, KUBERNETES_SERVICE_HOST=10.249.0.1, LANG=en_US.UTF-8, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT_443_TCP_PROTO=tcp, FLINK_ASC_URL=https://downloads.apache.org/flink/flink-1.19.1/flink-1.19.1-bin-scala_2.12.tgz.asc, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT_443_TCP_ADDR=10.249.108.78, FLINK_OPERATOR_WEBHOOK_SERVICE_SERVICE_PORT=443, JAVA_VERSION=jdk-11.0.24+8, FLINK_OPT_DIR=/opt/flink/opt, KUBERNETES_PORT_443_TCP_ADDR=10.249.0.1, FLINK_HOME=/opt/flink, FLINK_TRADES_REST_SERVICE_HOST=10.249.244.127, KUBERNETES_PORT_443_TCP_PROTO=tcp, FLINK_LIB_DIR=/opt/flink/lib, FLINK_OPERATOR_WEBHOOK_SERVICE_PORT_443_TCP=tcp://10.249.108.78:443, PYFLINK_GATEWAY_PORT=32935, KUBERNETES_SERVICE_PORT=443, FLINK_INHERITED_LOGS= RESOURCE_PARAMS extraction logs: jvm_params: -Xmx4315938809 -Xms4315938809 -XX:MaxMetaspaceSize=268435456 dynamic_configs: -D jobmanager.memory.off-heap.size=134217728b -D jobmanager.memory.jvm-overhead.min=524288007b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=4315938809b -D jobmanager.memory.jvm-overhead.max=524288007b logs: WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. INFO [] - Using legacy YAML parser to load flink configuration file from /opt/flink/conf/flink-conf.yaml. INFO [] - Loading configuration property: blob.server.port, 6124 INFO [] - Loading configuration property: taskmanager.memory.process.size, 20000 mb INFO [] - Loading configuration property: kubernetes.jobmanager.annotations, flinkdeployment.flink.apache.org/generation:3 INFO [] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint INFO [] - Loading configuration property: kubernetes.jobmanager.replicas, 1 INFO [] - Loading configuration property: jobmanager.rpc.address, flink-trades.flink-operator INFO [] - Loading configuration property: web.cancel.enable, false INFO [] - Loading configuration property: execution.target, kubernetes-session INFO [] - Loading configuration property: jobmanager.memory.process.size, 5000 mb INFO [] - Loading configuration property: kubernetes.service-account, flink INFO [] - Loading configuration property: kubernetes.cluster-id, flink-trades INFO [] - Loading configuration property: taskmanager.rpc.port, 6122 INFO [] - Loading configuration property: kubernetes.jobmanager.cpu.amount, 5.0 INFO [] - Loading configuration property: kubernetes.taskmanager.cpu.amount, 10.0 INFO [] - Loading configuration property: internal.cluster.execution-mode, NORMAL INFO [] - Loading configuration property: parallelism.default, 1 INFO [] - Loading configuration property: kubernetes.namespace, flink-operator INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1 INFO [] - Loading configuration property: kubernetes.rest-service.exposed.type, ClusterIP INFO [] - Loading configuration property: kubernetes.jobmanager.owner.reference, blockOwnerDeletion:true,controller:false,name:flink-trades,uid:2edf921d-f233-48c0-b5ca-6798c468e4e5,kind:FlinkDeployment,apiVersion:flink.apache.org/v1beta1 INFO [] - Loading configuration property: $internal.flink.version, v1_19 INFO [] - Loading configuration property: kubernetes.pod-template-file.jobmanager, /tmp/flink_op_generated_podTemplate_10469306648539520813.yaml INFO [] - Loading configuration property: kubernetes.container.image.ref, 862064919227.dkr.ecr.me-south-1.amazonaws.com/mdu/analytics/flink:test-print2 INFO [] - Final Master Memory configuration: INFO [] - Total Process Memory: 4.883gb (5242880000 bytes) INFO [] - Total Flink Memory: 4.145gb (4450156537 bytes) INFO [] - JVM Heap: 4.020gb (4315938809 bytes) INFO [] - Off-heap: 128.000mb (134217728 bytes) INFO [] - JVM Metaspace: 256.000mb (268435456 bytes) INFO [] - JVM Overhead: 500.000mb (524288007 bytes) , GOSU_VERSION=1.11, FLINK_TRADES_REST_PORT_8081_TCP_PORT=8081, HOSTNAME=flink-trades-788d66787-k2h2f, JVM_ARGS= -Xmx4315938809 -Xms4315938809 -XX:MaxMetaspaceSize=268435456, LD_PRELOAD=:/usr/lib/x86_64-linux-gnu/libjemalloc.so, KUBERNETES_PORT_443_TCP_PORT=443, HOME=/opt/flink}, command: python -u /opt/flink/usrlib/test.py 2024-08-21 16:59:17,530 INFO org.apache.flink.client.python.PythonDriver [] - --------------------------- Python Process Started -------------------------- 2024-08-21 16:59:17,890 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job 4efa73f63b25b276d423d94d76d19b65 is submitted. 2024-08-21 16:59:17,891 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=4efa73f63b25b276d423d94d76d19b65. 2024-08-21 16:59:18,049 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received JobGraph submission 'insert-into_default_catalog.default_database.print' (4efa73f63b25b276d423d94d76d19b65). 2024-08-21 16:59:18,050 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting job 'insert-into_default_catalog.default_database.print' (4efa73f63b25b276d423d94d76d19b65). 2024-08-21 16:59:18,051 INFO org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner [] - JobMasterServiceLeadershipRunner for job 4efa73f63b25b276d423d94d76d19b65 was granted leadership with leader id 00000000-0000-0000-0000-000000000000. Creating new JobMasterServiceProcess. 2024-08-21 16:59:18,052 INFO org.apache.flink.runtime.rpc.pekko.PekkoRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at pekko://flink/user/rpc/jobmanager_3 . 2024-08-21 16:59:18,053 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Initializing job 'insert-into_default_catalog.default_database.print' (4efa73f63b25b276d423d94d76d19b65). 2024-08-21 16:59:18,054 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart back off time strategy NoRestartBackoffTimeStrategy for insert-into_default_catalog.default_database.print (4efa73f63b25b276d423d94d76d19b65). 2024-08-21 16:59:18,055 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Created execution graph 1bff6480adcdb5c7415b54866b00ff07 for job 4efa73f63b25b276d423d94d76d19b65. 2024-08-21 16:59:18,055 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running initialization on master for job insert-into_default_catalog.default_database.print (4efa73f63b25b276d423d94d76d19b65). 2024-08-21 16:59:18,055 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Successfully ran initialization on master in 0 ms. 2024-08-21 16:59:18,056 INFO org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built 1 new pipelined regions in 0 ms, total 1 pipelined regions currently. 2024-08-21 16:59:18,056 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - State backend is set to heap memory org.apache.flink.runtime.state.hashmap.HashMapStateBackend@230c5da 2024-08-21 16:59:18,056 INFO org.apache.flink.runtime.state.StateBackendLoader [] - State backend loader loads the state backend as HashMapStateBackend 2024-08-21 16:59:18,056 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Checkpoint storage is set to 'jobmanager' 2024-08-21 16:59:18,056 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No checkpoint found during restore. 2024-08-21 16:59:18,056 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.RestartPipelinedRegionFailoverStrategy@4bb33ca3 for insert-into_default_catalog.default_database.print (4efa73f63b25b276d423d94d76d19b65). 2024-08-21 16:59:18,057 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting execution of job 'insert-into_default_catalog.default_database.print' (4efa73f63b25b276d423d94d76d19b65) under job master id 00000000000000000000000000000000. 2024-08-21 16:59:18,058 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting scheduling with scheduling strategy [org.apache.flink.runtime.scheduler.strategy.PipelinedRegionSchedulingStrategy] 2024-08-21 16:59:18,058 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job insert-into_default_catalog.default_database.print (4efa73f63b25b276d423d94d76d19b65) switched from state CREATED to RUNNING. 2024-08-21 16:59:18,058 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: datagen[5] -> Sink: print[6] (1/1) (1bff6480adcdb5c7415b54866b00ff07_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from CREATED to SCHEDULED. 2024-08-21 16:59:18,058 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Connecting to ResourceManager pekko.tcp://flink@flink-trades.flink-operator:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000) 2024-08-21 16:59:18,059 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Resolved ResourceManager address, beginning registration 2024-08-21 16:59:18,059 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 00000000000000000000000000000000@pekko.tcp://flink@flink-trades.flink-operator:6123/user/rpc/jobmanager_3 for job 4efa73f63b25b276d423d94d76d19b65. 2024-08-21 16:59:18,060 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registered job manager 00000000000000000000000000000000@pekko.tcp://flink@flink-trades.flink-operator:6123/user/rpc/jobmanager_3 for job 4efa73f63b25b276d423d94d76d19b65. 2024-08-21 16:59:18,061 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - JobManager successfully registered at ResourceManager, leader id: 00000000000000000000000000000000. 2024-08-21 16:59:18,061 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Received resource requirements from job 4efa73f63b25b276d423d94d76d19b65: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}] 2024-08-21 16:59:18,121 INFO org.apache.flink.client.python.PythonDriver [] - --------------------------- Python Process Exited --------------------------- 2024-08-21 16:59:18,126 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Matching resource requirements against available resources. Missing resources: Job 4efa73f63b25b276d423d94d76d19b65 ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1} Current resources: TaskManager flink-trades-taskmanager-1-1 Available: ResourceProfile{cpuCores=0, taskHeapMemory=0 bytes, taskOffHeapMemory=0 bytes, managedMemory=0 bytes, networkMemory=0 bytes} Total: ResourceProfile{cpuCores=1E+1, taskHeapMemory=8.891gb (9546235758 bytes), taskOffHeapMemory=0 bytes, managedMemory=7.313gb (7851737205 bytes), networkMemory=1.828gb (1962934301 bytes)} 2024-08-21 16:59:18,195 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - need request 1 new workers, current worker number 1, declared worker number 2 2024-08-21 16:59:18,195 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (2.059gb (2210864278 bytes)) is greater than its max value 1024.000mb (1073741824 bytes), max value will be used instead 2024-08-21 16:59:18,195 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Requesting new worker with resource spec WorkerResourceSpec {cpuCores=10.0, taskHeapSize=8.891gb (9546235758 bytes), taskOffHeapSize=0 bytes, networkMemSize=1.828gb (1962934301 bytes), managedMemSize=7.313gb (7851737205 bytes), numSlots=1}, current pending count: 1. 2024-08-21 16:59:18,196 INFO org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled external resources: [] 2024-08-21 16:59:18,196 INFO org.apache.flink.configuration.Configuration [] - Config uses fallback configuration key 'kubernetes.service-account' instead of key 'kubernetes.taskmanager.service-account' 2024-08-21 16:59:18,198 INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Creating new TaskManager pod with name flink-trades-taskmanager-1-2 and resource <20000,10.0>. 2024-08-21 16:59:18,219 INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Pod flink-trades-taskmanager-1-2 is created. 2024-08-21 16:59:18,229 INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Received new TaskManager pod: flink-trades-taskmanager-1-2 2024-08-21 16:59:18,229 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Requested worker flink-trades-taskmanager-1-2 with resource spec WorkerResourceSpec {cpuCores=10.0, taskHeapSize=8.891gb (9546235758 bytes), taskOffHeapSize=0 bytes, networkMemSize=1.828gb (1962934301 bytes), managedMemSize=7.313gb (7851737205 bytes), numSlots=1}. 2024-08-21 16:59:22,944 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering TaskManager with ResourceID flink-trades-taskmanager-1-2 (pekko.tcp://flink@100.72.94.195:6122/user/rpc/taskmanager_0) at ResourceManager 2024-08-21 16:59:22,955 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Registering task executor flink-trades-taskmanager-1-2 under 3fb05ec2c49b6859ac0e08d29bc0301b at the slot manager. 2024-08-21 16:59:22,955 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DefaultSlotStatusSyncer [] - Starting allocation of slot f05093dc302258f816fae2f4e6c6eebb from flink-trades-taskmanager-1-2 for job 4efa73f63b25b276d423d94d76d19b65 with resource profile ResourceProfile{cpuCores=1E+1, taskHeapMemory=8.891gb (9546235758 bytes), taskOffHeapMemory=0 bytes, managedMemory=7.313gb (7851737205 bytes), networkMemory=1.828gb (1962934301 bytes)}. 2024-08-21 16:59:22,956 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Worker flink-trades-taskmanager-1-2 is registered. 2024-08-21 16:59:22,956 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Worker flink-trades-taskmanager-1-2 with resource spec WorkerResourceSpec {cpuCores=10.0, taskHeapSize=8.891gb (9546235758 bytes), taskOffHeapSize=0 bytes, networkMemSize=1.828gb (1962934301 bytes), managedMemSize=7.313gb (7851737205 bytes), numSlots=1} was requested in current attempt. Current pending count after registering: 0. 2024-08-21 16:59:22,999 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: datagen[5] -> Sink: print[6] (1/1) (1bff6480adcdb5c7415b54866b00ff07_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from SCHEDULED to DEPLOYING. 2024-08-21 16:59:23,007 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: datagen[5] -> Sink: print[6] (1/1) (attempt #0) with attempt id 1bff6480adcdb5c7415b54866b00ff07_cbc357ccb763df2852fee8c4fc7d55f2_0_0 and vertex id cbc357ccb763df2852fee8c4fc7d55f2_0 to flink-trades-taskmanager-1-2 @ ip-100-72-94-195.me-south-1.compute.internal (dataPort=38585) with allocation id f05093dc302258f816fae2f4e6c6eebb 2024-08-21 16:59:23,173 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: datagen[5] -> Sink: print[6] (1/1) (1bff6480adcdb5c7415b54866b00ff07_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from DEPLOYING to INITIALIZING. 2024-08-21 16:59:23,501 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: datagen[5] -> Sink: print[6] (1/1) (1bff6480adcdb5c7415b54866b00ff07_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from INITIALIZING to RUNNING.