2023-07-05 22:20:57.070 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] [rank: 2] Global seed set to 84
0it [00:00, ?it/s]7.567 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] 2023-07-05 22:20:57 ERROR [__main__] main failed
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] Traceback (most recent call last):
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/ddp_trainer.py", line 86, in <module>
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     main()
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return self.main(*args, **kwargs)
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     rv = self.invoke(ctx)
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return ctx.invoke(self.callback, **ctx.params)
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return __callback(*args, **kwargs)
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/ddp_trainer.py", line 76, in main
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     model.train(**kwargs)
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/src/trainer.py", line 224, in train
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     trainer.fit(self.t5_model, self.data_module)
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     call._call_and_handle_interrupt(
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 89, in launch
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     self._call_children_scripts()
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 94, in _call_children_scripts
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     self._check_can_spawn_children()
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 134, in _check_can_spawn_children
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     raise RuntimeError(
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] RuntimeError: Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen. Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user, 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented.
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] Traceback (most recent call last):
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/ddp_trainer.py", line 86, in <module>
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     main()
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return self.main(*args, **kwargs)
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     rv = self.invoke(ctx)
2023-07-05 22:20:57.569 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return ctx.invoke(self.callback, **ctx.params)
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return __callback(*args, **kwargs)
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/ddp_trainer.py", line 76, in main
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     model.train(**kwargs)
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/src/trainer.py", line 224, in train
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     trainer.fit(self.t5_model, self.data_module)
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     call._call_and_handle_interrupt(
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 89, in launch
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     self._call_children_scripts()
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 94, in _call_children_scripts
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     self._check_can_spawn_children()
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 134, in _check_can_spawn_children
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     raise RuntimeError(
2023-07-05 22:20:57.570 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] RuntimeError: Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen. Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user, 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented.
0it [00:00, ?it/s]8.784 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]
2023-07-05 22:20:58.785 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] 2023-07-05 22:20:58 ERROR [__main__] main failed
2023-07-05 22:20:58.785 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] Traceback (most recent call last):
2023-07-05 22:20:58.785 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/ddp_trainer.py", line 86, in <module>
2023-07-05 22:20:58.785 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     main()
2023-07-05 22:20:58.785 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
2023-07-05 22:20:58.785 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return self.main(*args, **kwargs)
2023-07-05 22:20:58.785 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main
2023-07-05 22:20:58.785 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     rv = self.invoke(ctx)
2023-07-05 22:20:58.785 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
2023-07-05 22:20:58.785 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return ctx.invoke(self.callback, **ctx.params)
2023-07-05 22:20:58.785 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke
2023-07-05 22:20:58.785 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return __callback(*args, **kwargs)
2023-07-05 22:20:58.785 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/ddp_trainer.py", line 76, in main
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     model.train(**kwargs)
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/src/trainer.py", line 224, in train
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     trainer.fit(self.t5_model, self.data_module)
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     call._call_and_handle_interrupt(
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 89, in launch
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     self._call_children_scripts()
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 94, in _call_children_scripts
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     self._check_can_spawn_children()
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 134, in _check_can_spawn_children
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     raise RuntimeError(
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] RuntimeError: Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen. Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user, 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented.
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] Traceback (most recent call last):
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/ddp_trainer.py", line 86, in <module>
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     main()
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return self.main(*args, **kwargs)
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     rv = self.invoke(ctx)
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return ctx.invoke(self.callback, **ctx.params)
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return __callback(*args, **kwargs)
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/ddp_trainer.py", line 76, in main
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     model.train(**kwargs)
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/src/trainer.py", line 224, in train
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     trainer.fit(self.t5_model, self.data_module)
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     call._call_and_handle_interrupt(
2023-07-05 22:20:58.786 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
2023-07-05 22:20:58.787 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
2023-07-05 22:20:58.787 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 89, in launch
2023-07-05 22:20:58.787 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     self._call_children_scripts()
2023-07-05 22:20:58.787 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 94, in _call_children_scripts
2023-07-05 22:20:58.787 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     self._check_can_spawn_children()
2023-07-05 22:20:58.787 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 134, in _check_can_spawn_children
2023-07-05 22:20:58.787 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     raise RuntimeError(
2023-07-05 22:20:58.787 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] RuntimeError: Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen. Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user, 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented.
2023-07-05 22:20:59.572 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] [rank: 3] Global seed set to 84
2023-07-05 22:20:59.707 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] [rank: 0] Global seed set to 84
2023-07-05 22:20:59.708 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/16
0it [00:00, ?it/s]1.261 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] 2023-07-05 22:21:01 ERROR [__main__] main failed
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] Traceback (most recent call last):
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/ddp_trainer.py", line 86, in <module>
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     main()
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return self.main(*args, **kwargs)
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     rv = self.invoke(ctx)
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return ctx.invoke(self.callback, **ctx.params)
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return __callback(*args, **kwargs)
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/ddp_trainer.py", line 76, in main
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     model.train(**kwargs)
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/src/trainer.py", line 224, in train
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     trainer.fit(self.t5_model, self.data_module)
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     call._call_and_handle_interrupt(
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 89, in launch
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     self._call_children_scripts()
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 94, in _call_children_scripts
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     self._check_can_spawn_children()
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 134, in _check_can_spawn_children
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     raise RuntimeError(
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] RuntimeError: Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen. Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user, 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented.
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f] Traceback (most recent call last):
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/ddp_trainer.py", line 86, in <module>
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     main()
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return self.main(*args, **kwargs)
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     rv = self.invoke(ctx)
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return ctx.invoke(self.callback, **ctx.params)
2023-07-05 22:21:01.263 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke
2023-07-05 22:21:01.264 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return __callback(*args, **kwargs)
2023-07-05 22:21:01.264 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/ddp_trainer.py", line 76, in main
2023-07-05 22:21:01.264 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     model.train(**kwargs)
2023-07-05 22:21:01.264 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/src/trainer.py", line 224, in train
2023-07-05 22:21:01.264 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     trainer.fit(self.t5_model, self.data_module)
2023-07-05 22:21:01.264 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
2023-07-05 22:21:01.264 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     call._call_and_handle_interrupt(
2023-07-05 22:21:01.264 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
2023-07-05 22:21:01.264 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
2023-07-05 22:21:01.264 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 89, in launch
2023-07-05 22:21:01.264 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]     self._call_children_scripts()
2023-07-05 22:21:01.264 [285/train/control-285-start-1100 (pid 8066)] [74a88de0-0e4b-4c77-b22c-fc955325c66f]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 94, in _call_children_scripts