0it [00:00, ?it/s][rank: 1] Global seed set to 84art-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] 2023-07-05 20:00:08.539 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] [rank: 2] Global seed set to 84 2023-07-05 20:00:08.971 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] 2023-07-05 20:00:08 ERROR [__main__] main failed 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] Traceback (most recent call last): 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 86, in 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] main() 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__ 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return self.main(*args, **kwargs) 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] rv = self.invoke(ctx) 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return ctx.invoke(self.callback, **ctx.params) 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return __callback(*args, **kwargs) 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 76, in main 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] model.train(**kwargs) 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/src/trainer.py", line 207, in train 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] trainer = pl.Trainer( 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/utilities/argparse.py", line 340, in insert_env_defaults 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return fn(self, **kwargs) 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 414, in __init__ 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._accelerator_connector = AcceleratorConnector( 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 224, in __init__ 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._lazy_init_strategy() 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 795, in _lazy_init_strategy 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.strategy.set_world_ranks() 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 212, in set_world_ranks 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.cluster_environment.set_global_rank(self.node_rank * self.num_processes + self.local_rank) 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] TypeError: unsupported operand type(s) for +: 'int' and 'str' 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] Traceback (most recent call last): 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 86, in 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] main() 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__ 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return self.main(*args, **kwargs) 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] rv = self.invoke(ctx) 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return ctx.invoke(self.callback, **ctx.params) 2023-07-05 20:00:08.972 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return __callback(*args, **kwargs) 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 76, in main 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] model.train(**kwargs) 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/src/trainer.py", line 207, in train 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] trainer = pl.Trainer( 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/utilities/argparse.py", line 340, in insert_env_defaults 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return fn(self, **kwargs) 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 414, in __init__ 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._accelerator_connector = AcceleratorConnector( 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 224, in __init__ 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._lazy_init_strategy() 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 795, in _lazy_init_strategy 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.strategy.set_world_ranks() 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 212, in set_world_ranks 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.cluster_environment.set_global_rank(self.node_rank * self.num_processes + self.local_rank) 2023-07-05 20:00:08.973 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] TypeError: unsupported operand type(s) for +: 'int' and 'str' 2023-07-05 20:00:10.205 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] 2023-07-05 20:00:10 ERROR [__main__] main failed 2023-07-05 20:00:10.205 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] Traceback (most recent call last): 2023-07-05 20:00:10.205 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 86, in 2023-07-05 20:00:10.205 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] main() 2023-07-05 20:00:10.205 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__ 2023-07-05 20:00:10.205 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return self.main(*args, **kwargs) 2023-07-05 20:00:10.205 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main 2023-07-05 20:00:10.205 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] rv = self.invoke(ctx) 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return ctx.invoke(self.callback, **ctx.params) 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return __callback(*args, **kwargs) 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 76, in main 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] model.train(**kwargs) 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/src/trainer.py", line 207, in train 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] trainer = pl.Trainer( 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/utilities/argparse.py", line 340, in insert_env_defaults 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return fn(self, **kwargs) 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 414, in __init__ 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._accelerator_connector = AcceleratorConnector( 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 224, in __init__ 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._lazy_init_strategy() 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 795, in _lazy_init_strategy 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.strategy.set_world_ranks() 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 212, in set_world_ranks 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.cluster_environment.set_global_rank(self.node_rank * self.num_processes + self.local_rank) 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] TypeError: unsupported operand type(s) for +: 'int' and 'str' 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] Traceback (most recent call last): 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 86, in 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] main() 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__ 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return self.main(*args, **kwargs) 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] rv = self.invoke(ctx) 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return ctx.invoke(self.callback, **ctx.params) 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return __callback(*args, **kwargs) 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 76, in main 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] model.train(**kwargs) 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/src/trainer.py", line 207, in train 2023-07-05 20:00:10.206 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] trainer = pl.Trainer( 2023-07-05 20:00:10.207 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/utilities/argparse.py", line 340, in insert_env_defaults 2023-07-05 20:00:10.207 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return fn(self, **kwargs) 2023-07-05 20:00:10.207 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 414, in __init__ 2023-07-05 20:00:10.207 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._accelerator_connector = AcceleratorConnector( 2023-07-05 20:00:10.207 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 224, in __init__ 2023-07-05 20:00:10.207 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._lazy_init_strategy() 2023-07-05 20:00:10.207 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 795, in _lazy_init_strategy 2023-07-05 20:00:10.207 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.strategy.set_world_ranks() 2023-07-05 20:00:10.207 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 212, in set_world_ranks 2023-07-05 20:00:10.207 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.cluster_environment.set_global_rank(self.node_rank * self.num_processes + self.local_rank) 2023-07-05 20:00:10.207 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] TypeError: unsupported operand type(s) for +: 'int' and 'str' 2023-07-05 20:00:11.062 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] [rank: 3] Global seed set to 84 2023-07-05 20:00:11.192 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] [rank: 0] Global seed set to 84 0it [00:05, ?it/s]1.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] 2023-07-05 20:00:11 ERROR [__main__] main failed 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] Traceback (most recent call last): 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 86, in 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] main() 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__ 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return self.main(*args, **kwargs) 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] rv = self.invoke(ctx) 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return ctx.invoke(self.callback, **ctx.params) 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return __callback(*args, **kwargs) 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 76, in main 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] model.train(**kwargs) 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/src/trainer.py", line 224, in train 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] trainer.fit(self.t5_model, self.data_module) 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] call._call_and_handle_interrupt( 2023-07-05 20:00:11.196 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs) 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 90, in launch 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return function(*args, **kwargs) 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 645, in _fit_impl 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._run(model, ckpt_path=self.ckpt_path) 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1034, in _run 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.strategy.setup_environment() 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 153, in setup_environment 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.setup_distributed() 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 200, in setup_distributed 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.set_world_ranks() 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 212, in set_world_ranks 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.cluster_environment.set_global_rank(self.node_rank * self.num_processes + self.local_rank) 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] TypeError: unsupported operand type(s) for +: 'int' and 'str' 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] Traceback (most recent call last): 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 86, in 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] main() 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__ 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return self.main(*args, **kwargs) 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] rv = self.invoke(ctx) 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return ctx.invoke(self.callback, **ctx.params) 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return __callback(*args, **kwargs) 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 76, in main 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] model.train(**kwargs) 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/src/trainer.py", line 224, in train 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] trainer.fit(self.t5_model, self.data_module) 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] call._call_and_handle_interrupt( 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs) 2023-07-05 20:00:11.197 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 90, in launch 2023-07-05 20:00:11.198 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return function(*args, **kwargs) 2023-07-05 20:00:11.198 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 645, in _fit_impl 2023-07-05 20:00:11.198 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._run(model, ckpt_path=self.ckpt_path) 2023-07-05 20:00:11.198 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1034, in _run 2023-07-05 20:00:11.198 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.strategy.setup_environment() 2023-07-05 20:00:11.198 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 153, in setup_environment 2023-07-05 20:00:11.198 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.setup_distributed() 2023-07-05 20:00:11.198 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 200, in setup_distributed 2023-07-05 20:00:11.198 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.set_world_ranks() 2023-07-05 20:00:11.198 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 212, in set_world_ranks 2023-07-05 20:00:11.198 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.cluster_environment.set_global_rank(self.node_rank * self.num_processes + self.local_rank) 2023-07-05 20:00:11.198 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] TypeError: unsupported operand type(s) for +: 'int' and 'str' 2023-07-05 20:00:11.734 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] failed: 2023-07-05 20:00:12.725 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] 2023-07-05 20:00:12 ERROR [__main__] main failed 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] Traceback (most recent call last): 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 86, in 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] main() 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__ 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return self.main(*args, **kwargs) 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] rv = self.invoke(ctx) 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return ctx.invoke(self.callback, **ctx.params) 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return __callback(*args, **kwargs) 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 76, in main 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] model.train(**kwargs) 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/src/trainer.py", line 207, in train 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] trainer = pl.Trainer( 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/utilities/argparse.py", line 340, in insert_env_defaults 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return fn(self, **kwargs) 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 414, in __init__ 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._accelerator_connector = AcceleratorConnector( 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 224, in __init__ 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._lazy_init_strategy() 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 795, in _lazy_init_strategy 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.strategy.set_world_ranks() 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 212, in set_world_ranks 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.cluster_environment.set_global_rank(self.node_rank * self.num_processes + self.local_rank) 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] TypeError: unsupported operand type(s) for +: 'int' and 'str' 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] Traceback (most recent call last): 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 86, in 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] main() 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__ 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return self.main(*args, **kwargs) 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] rv = self.invoke(ctx) 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return ctx.invoke(self.callback, **ctx.params) 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return __callback(*args, **kwargs) 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_trainer.py", line 76, in main 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] model.train(**kwargs) 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/src/trainer.py", line 207, in train 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] trainer = pl.Trainer( 2023-07-05 20:00:12.726 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/utilities/argparse.py", line 340, in insert_env_defaults 2023-07-05 20:00:12.727 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return fn(self, **kwargs) 2023-07-05 20:00:12.727 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 414, in __init__ 2023-07-05 20:00:12.727 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._accelerator_connector = AcceleratorConnector( 2023-07-05 20:00:12.727 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 224, in __init__ 2023-07-05 20:00:12.727 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._lazy_init_strategy() 2023-07-05 20:00:12.727 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 795, in _lazy_init_strategy 2023-07-05 20:00:12.727 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.strategy.set_world_ranks() 2023-07-05 20:00:12.727 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 212, in set_world_ranks 2023-07-05 20:00:12.727 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self.cluster_environment.set_global_rank(self.node_rank * self.num_processes + self.local_rank) 2023-07-05 20:00:12.727 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] TypeError: unsupported operand type(s) for +: 'int' and 'str' 2023-07-05 20:00:13.290 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] Internal error 2023-07-05 20:00:13.290 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] Traceback (most recent call last): 2023-07-05 20:00:13.290 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow/cli.py", line 1172, in main 2023-07-05 20:00:13.290 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] start(auto_envvar_prefix="METAFLOW", obj=state) 2023-07-05 20:00:13.290 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow/_vendor/click/core.py", line 829, in __call__ 2023-07-05 20:00:13.290 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return self.main(args, kwargs) 2023-07-05 20:00:13.290 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow/_vendor/click/core.py", line 782, in main 2023-07-05 20:00:13.290 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] rv = self.invoke(ctx) 2023-07-05 20:00:13.290 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow/_vendor/click/core.py", line 1259, in invoke 2023-07-05 20:00:13.290 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return _process_result(sub_ctx.command.invoke(sub_ctx)) 2023-07-05 20:00:13.290 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow/_vendor/click/core.py", line 1066, in invoke 2023-07-05 20:00:13.290 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return ctx.invoke(self.callback, ctx.params) 2023-07-05 20:00:13.290 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow/_vendor/click/core.py", line 610, in invoke 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return callback(args, kwargs) 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow/_vendor/click/decorators.py", line 21, in new_func 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return f(get_current_context(), args, kwargs) 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow/cli.py", line 581, in step 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] task.run_step( 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow/task.py", line 586, in run_step 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] self._exec_step_function(step_func) 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow/task.py", line 60, in _exec_step_function 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] step_function() 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow/plugins/parallel_decorator.py", line 34, in _step_func_with_setup 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] step_func() 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/gpu_profile.py", line 148, in func 2023-07-05 20:01:13.100 [281/train/control-281-start-1085 (pid 955)] AWS Batch error: 2023-07-05 20:01:13.101 [281/train/control-281-start-1085 (pid 955)] Essential container in task exited This could be a transient error. Use @retry to retry. 2023-07-05 20:01:13.305 [281/train/control-281-start-1085 (pid 955)] 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] f(s) 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/custom_decorators.py", line 55, in wrapper 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] return function(args, kwargs) 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/ddp_flow.py", line 157, in train 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] subprocess.run( 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/subprocess.py", line 524, in run 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] raise CalledProcessError(retcode, process.args, 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360] subprocess.CalledProcessError: Command '['python', 'ddp_trainer.py', '--output-dir', 'outputs', '--source-max-token-length', '256', '--target-max-token-length', '256', '--batch-size', '6', '--max-epochs', '1', '--learning-rate', '3e-4', '--weight-decay', '0.1', '--adam-epsilon', '9e-7', '--warmup-steps', '0', '--gradient-accumulation-steps', '16', '--n-gpu', '-1', '--num-nodes=4', '--early-stopping-patience-epochs', '0', '--precision', 'bf16', '--logger', 'default', '--dataloader-num-workers', '2', '--opt-level', '01', '--max-grad-norm', '0.5', '--seed', '42', '--use-gpu']' returned non-zero exit status 1. 2023-07-05 20:00:13.291 [281/train/control-281-start-1085 (pid 955)] [170763d6-2e77-463d-89c4-4a26a6bb6360]