2023-07-05 23:51:28.975 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b] Traceback (most recent call last):
2023-07-05 23:51:28.975 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/bin/torchrun", line 33, in <module>
2023-07-05 23:51:28.975 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     sys.exit(load_entry_point('torch==1.12.0', 'console_scripts', 'torchrun')())
2023-07-05 23:51:28.975 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
2023-07-05 23:51:28.975 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     return f(*args, **kwargs)
2023-07-05 23:51:28.975 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/run.py", line 761, in main
2023-07-05 23:51:28.976 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     run(args)
2023-07-05 23:51:28.976 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/run.py", line 752, in run
2023-07-05 23:51:28.976 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     elastic_launch(
2023-07-05 23:51:28.976 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
2023-07-05 23:51:28.976 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     return launch_agent(self._config, self._entrypoint, list(args))
2023-07-05 23:51:28.976 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 214, in launch_agent
2023-07-05 23:51:28.976 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     spec = WorkerSpec(
2023-07-05 23:51:28.976 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "<string>", line 15, in __init__
2023-07-05 23:51:28.976 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 87, in __post_init__
2023-07-05 23:51:28.976 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     assert self.local_world_size > 0
2023-07-05 23:51:28.976 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b] AssertionError
2023-07-05 23:51:29.217 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b] <flow T5DDPFlow step train[0] (input: 0)> failed:
2023-07-05 23:51:30.619 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     Internal error
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b] Traceback (most recent call last):
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow/cli.py", line 1172, in main
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     start(auto_envvar_prefix="METAFLOW", obj=state)
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow/_vendor/click/core.py", line 829, in __call__
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     return self.main(args, kwargs)
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow/_vendor/click/core.py", line 782, in main
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     rv = self.invoke(ctx)
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow/_vendor/click/core.py", line 1259, in invoke
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     return _process_result(sub_ctx.command.invoke(sub_ctx))
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow/_vendor/click/core.py", line 1066, in invoke
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     return ctx.invoke(self.callback, ctx.params)
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow/_vendor/click/core.py", line 610, in invoke
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     return callback(args, kwargs)
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow/_vendor/click/decorators.py", line 21, in new_func
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     return f(get_current_context(), args, kwargs)
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow/cli.py", line 581, in step
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     task.run_step(
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow/task.py", line 586, in run_step
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     self._exec_step_function(step_func)
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow/task.py", line 60, in _exec_step_function
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     step_function()
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow/plugins/parallel_decorator.py", line 34, in _step_func_with_setup
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     step_func()
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/gpu_profile.py", line 148, in func

2023-07-05 23:52:32.805 [287/train/control-287-start-1112 (pid 13237)] AWS Batch error:
2023-07-05 23:52:32.805 [287/train/control-287-start-1112 (pid 13237)] Essential container in task exited This could be a transient error. Use @retry to retry.
2023-07-05 23:52:33.022 [287/train/control-287-start-1112 (pid 13237)]
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     f(s)
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/custom_decorators.py", line 55, in wrapper
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     return function(args, kwargs)
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/ddp_flow.py", line 157, in train
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     subprocess.run(
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/subprocess.py", line 524, in run
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b]     raise CalledProcessError(retcode, process.args,
2023-07-05 23:51:30.620 [287/train/control-287-start-1112 (pid 13237)] [e2bdbef1-9b75-4055-8bb9-9084e9551e9b] subprocess.CalledProcessError: Command '['torchrun', '--nproc_per_node=-1', '--nnodes=4', '--rdzv_id=287', '--rdzv_backend=c10d', '--rdzv_endpoint=10.14.51.235:29400', 'ddp_trainer.py', '--output-dir', 'outputs', '--source-max-token-length', '256', '--target-max-token-length', '256', '--batch-size', '6', '--max-epochs', '1', '--learning-rate', '3e-4', '--weight-decay', '0.1', '--adam-epsilon', '9e-7', '--warmup-steps', '0', '--gradient-accumulation-steps', '16', '--n-gpu', '-1', '--num-nodes=4', '--early-stopping-patience-epochs', '0', '--precision', 'bf16', '--logger', 'default', '--dataloader-num-workers', '2', '--opt-level', '01', '--max-grad-norm', '0.5', '--seed', '42', '--use-gpu']' returned non-zero exit status 1.