2023-07-06 00:09:43.339 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517] Calling PTL process...
2023-07-06 00:09:43.339 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]
2023-07-06 00:09:43.339 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]
2023-07-06 00:09:44.825 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517] Traceback (most recent call last):
2023-07-06 00:09:44.825 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/bin/torchrun", line 33, in <module>
2023-07-06 00:09:44.825 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     sys.exit(load_entry_point('torch==1.12.0', 'console_scripts', 'torchrun')())
2023-07-06 00:09:44.825 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
2023-07-06 00:09:44.825 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     return f(*args, **kwargs)
2023-07-06 00:09:44.825 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/run.py", line 761, in main
2023-07-06 00:09:44.825 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     run(args)
2023-07-06 00:09:44.825 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/run.py", line 752, in run
2023-07-06 00:09:44.825 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     elastic_launch(
2023-07-06 00:09:44.825 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
2023-07-06 00:09:44.826 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     return launch_agent(self._config, self._entrypoint, list(args))
2023-07-06 00:09:44.826 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 214, in launch_agent
2023-07-06 00:09:44.826 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     spec = WorkerSpec(
2023-07-06 00:09:44.826 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "<string>", line 15, in __init__
2023-07-06 00:09:44.826 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 87, in __post_init__
2023-07-06 00:09:44.826 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     assert self.local_world_size > 0
2023-07-06 00:09:44.826 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517] AssertionError
2023-07-06 00:09:45.036 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517] <flow T5DDPFlow step train[0] (input: 0)> failed:
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     Internal error
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517] Traceback (most recent call last):
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow/cli.py", line 1172, in main
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     start(auto_envvar_prefix="METAFLOW", obj=state)
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow/_vendor/click/core.py", line 829, in __call__
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     return self.main(args, kwargs)
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow/_vendor/click/core.py", line 782, in main
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     rv = self.invoke(ctx)
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow/_vendor/click/core.py", line 1259, in invoke
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     return _process_result(sub_ctx.command.invoke(sub_ctx))
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow/_vendor/click/core.py", line 1066, in invoke
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     return ctx.invoke(self.callback, ctx.params)
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow/_vendor/click/core.py", line 610, in invoke
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     return callback(args, kwargs)
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow/_vendor/click/decorators.py", line 21, in new_func
2023-07-06 00:09:46.523 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     return f(get_current_context(), args, kwargs)
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow/cli.py", line 581, in step
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     task.run_step(
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow/task.py", line 586, in run_step
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     self._exec_step_function(step_func)
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow/task.py", line 60, in _exec_step_function
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     step_function()
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow/plugins/parallel_decorator.py", line 34, in _step_func_with_setup
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     step_func()
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/gpu_profile.py", line 148, in func

2023-07-06 00:10:31.013 [289/train/control-289-start-1121 (pid 14326)] AWS Batch error:
2023-07-06 00:10:31.014 [289/train/control-289-start-1121 (pid 14326)] Essential container in task exited This could be a transient error. Use @retry to retry.
2023-07-06 00:10:31.511 [289/train/control-289-start-1121 (pid 14326)]
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     f(s)
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/custom_decorators.py", line 55, in wrapper
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     return function(args, kwargs)
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/ddp_flow.py", line 157, in train
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     subprocess.run(
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]   File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/subprocess.py", line 524, in run
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517]     raise CalledProcessError(retcode, process.args,
2023-07-06 00:09:46.524 [289/train/control-289-start-1121 (pid 14326)] [245e6b1f-ec29-43df-89f2-396f0cde6517] subprocess.CalledProcessError: Command '['torchrun', '--nproc_per_node=-1', '--nnodes=4', '--rdzv_id=metaflow_289', '--rdzv_backend=c10d', '--rdzv_endpoint=10.14.52.57:29400', 'ddp_trainer.py', '--output-dir', 'outputs', '--source-max-token-length', '256', '--target-max-token-length', '256', '--batch-size', '6', '--max-epochs', '1', '--learning-rate', '3e-4', '--weight-decay', '0.1', '--adam-epsilon', '9e-7', '--warmup-steps', '0', '--gradient-accumulation-steps', '16', '--n-gpu', '-1', '--num-nodes=4', '--early-stopping-patience-epochs', '0', '--precision', 'bf16', '--logger', 'default', '--dataloader-num-workers', '2', '--opt-level', '01', '--max-grad-norm', '0.5', '--seed', '42', '--use-gpu']' returned non-zero exit status 1.