2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,942 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Sort[13] -> PythonGroupWindowAggregate[14] -> Calc[15] -> PythonCalc[16] -> Calc[17] -> Sort[18] -> Sink: Filesystem (5/6) (fa96833f499eaa2e9f9c4fb3ce349c82_bd3f3dbc94135cffd6910d66b9048a6c_4_2) switched from RUNNING to CANCELING.
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,942 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Discarding the results produced by task execution fa96833f499eaa2e9f9c4fb3ce349c82_4150b807e25f98bebfeb73f2fab67d53_2_2.
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,942 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 2 (#2) of source Source: _order_events_data_lake[5].
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,942 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Discarding the results produced by task execution fa96833f499eaa2e9f9c4fb3ce349c82_6cdc5bb954874d922eaee11a8e7b5dd5_2_2.
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,942 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Discarding the results produced by task execution fa96833f499eaa2e9f9c4fb3ce349c82_4150b807e25f98bebfeb73f2fab67d53_3_2.
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,942 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Sort[13] -> PythonGroupWindowAggregate[14] -> Calc[15] -> PythonCalc[16] -> Calc[17] -> Sort[18] -> Sink: Filesystem (1/6) (fa96833f499eaa2e9f9c4fb3ce349c82_bd3f3dbc94135cffd6910d66b9048a6c_0_2) switched from RUNNING to CANCELING.
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,942 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Discarding the results produced by task execution fa96833f499eaa2e9f9c4fb3ce349c82_bc764cd8ddf7a0cff126f51c16239658_1_2.
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,942 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 1 (#2) of source Source: _delivery_events_data_lake[1].
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,942 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 2 (#2) of source Source: _delivery_events_data_lake[1].
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,942 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Discarding the results produced by task execution fa96833f499eaa2e9f9c4fb3ce349c82_bc764cd8ddf7a0cff126f51c16239658_2_2.
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,942 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Discarding the results produced by task execution fa96833f499eaa2e9f9c4fb3ce349c82_bc764cd8ddf7a0cff126f51c16239658_0_2.
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,942 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 0 (#2) of source Source: _delivery_events_data_lake[1].
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,941 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job pyflink-batch-job-queued-orders-liza (7ff4835dc8de5084beed3f5d46d3e2be) switched from state RUNNING to RESTARTING.
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,941 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - 20 tasks will be restarted to recover the failed task fa96833f499eaa2e9f9c4fb3ce349c82_bd3f3dbc94135cffd6910d66b9048a6c_3_4.
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,940 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job 7ff4835dc8de5084beed3f5d46d3e2be: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=5}]
2023-08-16T11:06:15+02:00		... 1 more
2023-08-16T11:06:15+02:00		at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.shaded.netty4.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.shaded.netty4.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.shaded.netty4.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00	Caused by: java.net.ConnectException: finishConnect(..) failed: No route to host
2023-08-16T11:06:15+02:00	Caused by: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: No route to host: /xx.xx.xx.xxx:32805
2023-08-16T11:06:15+02:00		... 16 more
2023-08-16T11:06:15+02:00		at org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:186) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.runtime.io.network.netty.NettyConnectionManager.createPartitionRequestClient(NettyConnectionManager.java:95) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.createPartitionRequestClient(PartitionRequestClientFactory.java:97) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.connectWithRetries(PartitionRequestClientFactory.java:136) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.connect(PartitionRequestClientFactory.java:173) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00	Caused by: org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Connecting to remote task manager '/xx.xx.xx.xxx:32805 [ pyflink-batch-job-queued-orders-liza-taskmanager-1-4 ] ' has failed. This might indicate that the remote task manager has been lost.
2023-08-16T11:06:15+02:00		at java.lang.Thread.run(Unknown Source) ~[?:?]
2023-08-16T11:06:15+02:00		at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.runtime.taskmanager.InputGateWithMetrics.requestPartitions(InputGateWithMetrics.java:115) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.requestPartitions(SingleInputGate.java:312) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.internalRequestPartitions(SingleInputGate.java:342) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00		at org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:190) ~[flink-dist-1.17.1.jar:1.17.1]
2023-08-16T11:06:15+02:00	org.apache.flink.runtime.io.network.partition.consumer.PartitionConnectionException: Connection for partition 8371b498e03cc0d55c5ce6dcfd96174c#5@fa96833f499eaa2e9f9c4fb3ce349c82_4150b807e25f98bebfeb73f2fab67d53_5_2 not reachable.
2023-08-16T11:06:15+02:00	2023-08-16 09:06:15,938 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Sort[13] -> PythonGroupWindowAggregate[14] -> Calc[15] -> PythonCalc[16] -> Calc[17] -> Sort[18] -> Sink: Filesystem (4/6) (fa96833f499eaa2e9f9c4fb3ce349c82_bd3f3dbc94135cffd6910d66b9048a6c_3_4) switched from RUNNING to FAILED on pyflink-batch-job-queued-orders-liza-taskmanager-2-1 @ ip-xx-xx-xx-xxx.eu-west-1.compute.internal (dataPort=41621).
2023-08-16T11:06:05+02:00	2023-08-16 09:06:05,703 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Sort[13] -> PythonGroupWindowAggregate[14] -> Calc[15] -> PythonCalc[16] -> Calc[17] -> Sort[18] -> Sink: Filesystem (4/6) (fa96833f499eaa2e9f9c4fb3ce349c82_bd3f3dbc94135cffd6910d66b9048a6c_3_4) switched from INITIALIZING to RUNNING.
2023-08-16T11:06:03+02:00	2023-08-16 09:06:03,865 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Sort[13] -> PythonGroupWindowAggregate[14] -> Calc[15] -> PythonCalc[16] -> Calc[17] -> Sort[18] -> Sink: Filesystem (4/6) (fa96833f499eaa2e9f9c4fb3ce349c82_bd3f3dbc94135cffd6910d66b9048a6c_3_4) switched from DEPLOYING to INITIALIZING.