WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
[rank: 15] Global seed set to 84
[rank: 13] Global seed set to 84
[rank: 14] Global seed set to 84
[rank: 12] Global seed set to 84

Downloading (…)okenizer_config.json:   0%|          | 0.00/2.32k [00:00<?, ?B/s]
Downloading (…)okenizer_config.json: 100%|██████████| 2.32k/2.32k [00:00<00:00, 25.0MB/s]

Downloading (…)ve/main/spiece.model:   0%|          | 0.00/792k [00:00<?, ?B/s]
Downloading (…)ve/main/spiece.model: 100%|██████████| 792k/792k [00:00<00:00, 10.7MB/s]

Downloading (…)/main/tokenizer.json:   0%|          | 0.00/1.39M [00:00<?, ?B/s]
Downloading (…)/main/tokenizer.json: 100%|██████████| 1.39M/1.39M [00:00<00:00, 6.22MB/s]
Downloading (…)/main/tokenizer.json: 100%|██████████| 1.39M/1.39M [00:00<00:00, 6.20MB/s]

Downloading (…)lve/main/config.json:   0%|          | 0.00/1.21k [00:00<?, ?B/s]
Downloading (…)lve/main/config.json: 100%|██████████| 1.21k/1.21k [00:00<00:00, 14.8MB/s]

Downloading pytorch_model.bin:   0%|          | 0.00/242M [00:00<?, ?B/s]
Downloading pytorch_model.bin:  13%|█▎        | 31.5M/242M [00:00<00:00, 256MB/s]
Downloading pytorch_model.bin:  30%|███       | 73.4M/242M [00:00<00:00, 342MB/s]
Downloading pytorch_model.bin:  48%|████▊     | 115M/242M [00:00<00:00, 364MB/s] 
Downloading pytorch_model.bin:  65%|██████▍   | 157M/242M [00:00<00:00, 383MB/s]
Downloading pytorch_model.bin:  87%|████████▋ | 210M/242M [00:00<00:00, 400MB/s]
Downloading pytorch_model.bin: 100%|██████████| 242M/242M [00:00<00:00, 381MB/s]

0it [00:00, ?it/s][rank: 13] Global seed set to 84
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/16
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs

0it [00:00, ?it/s][rank: 12] Global seed set to 84
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/16

0it [00:00, ?it/s][rank: 15] Global seed set to 84
Initializing distributed: GLOBAL_RANK: 3, MEMBER: 4/16

                  
2023-07-06 00:28:41 INFO [torch.distributed.distributed_c10d] Added key: store_based_barrier_key:1 to store for rank: 3

0it [00:00, ?it/s]
0it [00:00, ?it/s][rank: 14] Global seed set to 84
Initializing distributed: GLOBAL_RANK: 2, MEMBER: 3/16

                  
2023-07-06 00:28:41 INFO [torch.distributed.distributed_c10d] Added key: store_based_barrier_key:1 to store for rank: 2

0it [00:00, ?it/s]
                  
2023-07-06 00:28:42 INFO [torch.distributed.distributed_c10d] Added key: store_based_barrier_key:1 to store for rank: 1

0it [00:01, ?it/s]
                  
2023-07-06 00:28:42 INFO [torch.distributed.distributed_c10d] Added key: store_based_barrier_key:1 to store for rank: 0

0it [00:00, ?it/s]
                  
2023-07-06 00:28:42 INFO [torch.distributed.distributed_c10d] Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 16 nodes.

0it [00:00, ?it/s]----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 16 processes
----------------------------------------------------------------------------------------------------


                  
2023-07-06 00:28:42 INFO [torch.distributed.distributed_c10d] Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 16 nodes.

0it [00:01, ?it/s]Missing logger folder: /metaflow/lightning_logs
Missing logger folder: /metaflow/lightning_logs

                  

                  
2023-07-06 00:28:42 INFO [torch.distributed.distributed_c10d] Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 16 nodes.
2023-07-06 00:28:42 INFO [torch.distributed.distributed_c10d] Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 16 nodes.

0it [00:00, ?it/s]
0it [00:00, ?it/s]Missing logger folder: /metaflow/lightning_logs
Missing logger folder: /metaflow/lightning_logs

0it [00:02, ?it/s]
2023-07-06 00:28:44 ERROR [__main__] main failed
Traceback (most recent call last):
  File "/metaflow/ddp_trainer.py", line 86, in <module>
    main()
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/metaflow/ddp_trainer.py", line 76, in main
    model.train(**kwargs)
  File "/metaflow/src/trainer.py", line 224, in train
    trainer.fit(self.t5_model, self.data_module)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
    call._call_and_handle_interrupt(
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 645, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1035, in _run
    self.__setup_profiler()
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1495, in __setup_profiler
    self.profiler.setup(stage=self.state.fn, local_rank=local_rank, log_dir=self.log_dir)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1828, in log_dir
    dirpath = self.strategy.broadcast(dirpath)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 315, in broadcast
    torch.distributed.broadcast_object_list(obj, src, group=_group.WORLD)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1877, in broadcast_object_list
    broadcast(object_sizes_tensor, src=src, group=group)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1193, in broadcast
    work = default_pg.broadcast([tensor], opts)
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1656352630480/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1191, unhandled system error, NCCL version 2.10.3
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer, you can check NCCL warnings for failure reason and see if there is connection closure by a peer.
Traceback (most recent call last):
  File "/metaflow/ddp_trainer.py", line 86, in <module>
    main()
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/metaflow/ddp_trainer.py", line 76, in main
    model.train(**kwargs)
  File "/metaflow/src/trainer.py", line 224, in train
    trainer.fit(self.t5_model, self.data_module)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
    call._call_and_handle_interrupt(
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 645, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1035, in _run
    self.__setup_profiler()
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1495, in __setup_profiler
    self.profiler.setup(stage=self.state.fn, local_rank=local_rank, log_dir=self.log_dir)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1828, in log_dir
    dirpath = self.strategy.broadcast(dirpath)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 315, in broadcast
    torch.distributed.broadcast_object_list(obj, src, group=_group.WORLD)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1877, in broadcast_object_list
    broadcast(object_sizes_tensor, src=src, group=group)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1193, in broadcast
    work = default_pg.broadcast([tensor], opts)
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1656352630480/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1191, unhandled system error, NCCL version 2.10.3
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer, you can check NCCL warnings for failure reason and see if there is connection closure by a peer.

0it [00:02, ?it/s]
2023-07-06 00:28:44 ERROR [__main__] main failed
Traceback (most recent call last):
  File "/metaflow/ddp_trainer.py", line 86, in <module>
    main()
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/metaflow/ddp_trainer.py", line 76, in main
    model.train(**kwargs)
  File "/metaflow/src/trainer.py", line 224, in train
    trainer.fit(self.t5_model, self.data_module)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
    call._call_and_handle_interrupt(
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 645, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1035, in _run
    self.__setup_profiler()
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1495, in __setup_profiler
    self.profiler.setup(stage=self.state.fn, local_rank=local_rank, log_dir=self.log_dir)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1828, in log_dir
    dirpath = self.strategy.broadcast(dirpath)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 315, in broadcast
    torch.distributed.broadcast_object_list(obj, src, group=_group.WORLD)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1877, in broadcast_object_list
    broadcast(object_sizes_tensor, src=src, group=group)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1193, in broadcast
    work = default_pg.broadcast([tensor], opts)
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1656352630480/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1191, unhandled system error, NCCL version 2.10.3
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer, you can check NCCL warnings for failure reason and see if there is connection closure by a peer.
Traceback (most recent call last):
  File "/metaflow/ddp_trainer.py", line 86, in <module>
    main()
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/metaflow/ddp_trainer.py", line 76, in main
    model.train(**kwargs)
  File "/metaflow/src/trainer.py", line 224, in train
    trainer.fit(self.t5_model, self.data_module)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
    call._call_and_handle_interrupt(
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 645, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1035, in _run
    self.__setup_profiler()
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1495, in __setup_profiler
    self.profiler.setup(stage=self.state.fn, local_rank=local_rank, log_dir=self.log_dir)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1828, in log_dir
    dirpath = self.strategy.broadcast(dirpath)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 315, in broadcast
    torch.distributed.broadcast_object_list(obj, src, group=_group.WORLD)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1877, in broadcast_object_list
    broadcast(object_sizes_tensor, src=src, group=group)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1193, in broadcast
    work = default_pg.broadcast([tensor], opts)
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1656352630480/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1191, unhandled system error, NCCL version 2.10.3
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer, you can check NCCL warnings for failure reason and see if there is connection closure by a peer.

0it [00:02, ?it/s]
2023-07-06 00:28:44 ERROR [__main__] main failed
Traceback (most recent call last):
  File "/metaflow/ddp_trainer.py", line 86, in <module>
    main()
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/metaflow/ddp_trainer.py", line 76, in main
    model.train(**kwargs)
  File "/metaflow/src/trainer.py", line 224, in train
    trainer.fit(self.t5_model, self.data_module)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
    call._call_and_handle_interrupt(
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 645, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1035, in _run
    self.__setup_profiler()
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1495, in __setup_profiler
    self.profiler.setup(stage=self.state.fn, local_rank=local_rank, log_dir=self.log_dir)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1828, in log_dir
    dirpath = self.strategy.broadcast(dirpath)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 315, in broadcast
    torch.distributed.broadcast_object_list(obj, src, group=_group.WORLD)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1877, in broadcast_object_list
    broadcast(object_sizes_tensor, src=src, group=group)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1193, in broadcast
    work = default_pg.broadcast([tensor], opts)
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1656352630480/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1191, unhandled system error, NCCL version 2.10.3
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer, you can check NCCL warnings for failure reason and see if there is connection closure by a peer.
Traceback (most recent call last):
  File "/metaflow/ddp_trainer.py", line 86, in <module>
    main()
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/metaflow/ddp_trainer.py", line 76, in main
    model.train(**kwargs)
  File "/metaflow/src/trainer.py", line 224, in train
    trainer.fit(self.t5_model, self.data_module)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
    call._call_and_handle_interrupt(
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 645, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1035, in _run
    self.__setup_profiler()
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1495, in __setup_profiler
    self.profiler.setup(stage=self.state.fn, local_rank=local_rank, log_dir=self.log_dir)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1828, in log_dir
    dirpath = self.strategy.broadcast(dirpath)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 315, in broadcast
    torch.distributed.broadcast_object_list(obj, src, group=_group.WORLD)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1877, in broadcast_object_list
    broadcast(object_sizes_tensor, src=src, group=group)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1193, in broadcast
    work = default_pg.broadcast([tensor], opts)
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1656352630480/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1191, unhandled system error, NCCL version 2.10.3
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer, you can check NCCL warnings for failure reason and see if there is connection closure by a peer.
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 862 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 859) of binary: /metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/bin/python
Traceback (most recent call last):
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==1.12.0', 'console_scripts', 'torchrun')())
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
    return f(*args, **kwargs)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/run.py", line 761, in main
    run(args)
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/run.py", line 752, in run
    elastic_launch(
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/launcher/jupyter/default/metaflow/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages/torch/distributed/launcher/jupyter/default/metaflow/api.py", line 245, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
ddp_trainer.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2023-07-06_00:28:46
  host      : ip-10-14-52-55.us-west-2.compute.internal
  rank      : 13 (local_rank: 1)
  exitcode  : 1 (pid: 860)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2023-07-06_00:28:46
  host      : ip-10-14-52-55.us-west-2.compute.internal
  rank      : 14 (local_rank: 2)
  exitcode  : 1 (pid: 861)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-07-06_00:28:46
  host      : ip-10-14-52-55.us-west-2.compute.internal
  rank      : 12 (local_rank: 0)
  exitcode  : 1 (pid: 859)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
<flow T5DDPFlow step train[0] (input: 0)> failed:
    Internal error
Traceback (most recent call last):
  File "/metaflow/metaflow/cli.py", line 1172, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/metaflow/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
  File "/metaflow/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/metaflow/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/metaflow/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
  File "/metaflow/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
  File "/metaflow/metaflow/_vendor/click/decorators.py", line 21, in new_func
    return f(get_current_context(), args, kwargs)
  File "/metaflow/metaflow/cli.py", line 581, in step
    task.run_step(
  File "/metaflow/metaflow/task.py", line 586, in run_step
    self._exec_step_function(step_func)
  File "/metaflow/metaflow/task.py", line 60, in _exec_step_function
    step_function()
  File "/metaflow/metaflow/plugins/parallel_decorator.py", line 34, in _step_func_with_setup
    step_func()
  File "/metaflow/gpu_profile.py", line 148, in func
    f(s)
  File "/metaflow/custom_decorators.py", line 55, in wrapper
    return function(args, kwargs)
  File "/metaflow/ddp_flow.py", line 157, in train
    subprocess.run(
  File "/metaflow/metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['torchrun', '--nproc_per_node=4', '--nnodes=4', '--rdzv_id=metaflow_290', '--rdzv_backend=c10d', '--rdzv_endpoint=10.14.52.55:29400', 'ddp_trainer.py', '--output-dir', 'outputs', '--source-max-token-length', '256', '--target-max-token-length', '256', '--batch-size', '6', '--max-epochs', '1', '--learning-rate', '3e-4', '--weight-decay', '0.1', '--adam-epsilon', '9e-7', '--warmup-steps', '0', '--gradient-accumulation-steps', '16', '--n-gpu', '4', '--num-nodes=4', '--early-stopping-patience-epochs', '0', '--precision', 'bf16', '--logger', 'default', '--dataloader-num-workers', '2', '--opt-level', '01', '--max-grad-norm', '0.5', '--seed', '42', '--use-gpu']' returned non-zero exit status 1.

    AWS Batch error:
    Essential container in task exited This could be a transient error. Use @retry to retry.

Task failed.