Setting up task environment. Downloading code package... Code package downloaded. Task is starting. Bootstrapping environment... Environment bootstrapped. No dot env! Pip Install: opencv_python_headless 4.5.5.62 Collecting opencv_python_headless==4.5.5.62 Downloading opencv_python_headless-4.5.5.62-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (47.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.7/47.7 MB 53.6 MB/s eta 0:00:00 Requirement already satisfied: numpy>=1.21.2 in ./metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages (from opencv_python_headless==4.5.5.62) (1.25.0) Installing collected packages: opencv_python_headless Successfully installed opencv_python_headless-4.5.5.62 Pip Install: transformers 4.25.1 Collecting transformers==4.25.1 Downloading transformers-4.25.1-py3-none-any.whl (5.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.8/5.8 MB 73.2 MB/s eta 0:00:00 Collecting filelock (from transformers==4.25.1) Downloading filelock-3.12.2-py3-none-any.whl (10 kB) Collecting huggingface-hub<1.0,>=0.10.0 (from transformers==4.25.1) Downloading huggingface_hub-0.16.2-py3-none-any.whl (268 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 268.5/268.5 kB 23.2 MB/s eta 0:00:00 Requirement already satisfied: numpy>=1.17 in ./metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages (from transformers==4.25.1) (1.25.0) Requirement already satisfied: packaging>=20.0 in ./metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages (from transformers==4.25.1) (23.1) Requirement already satisfied: pyyaml>=5.1 in ./metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages (from transformers==4.25.1) (6.0) Collecting regex!=2019.12.17 (from transformers==4.25.1) Downloading regex-2023.6.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (770 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 770.4/770.4 kB 81.1 MB/s eta 0:00:00 Requirement already satisfied: requests in ./metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages (from transformers==4.25.1) (2.31.0) Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.25.1) Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 135.2 MB/s eta 0:00:00 Requirement already satisfied: tqdm>=4.27 in ./metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages (from transformers==4.25.1) (4.65.0) Requirement already satisfied: fsspec in ./metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.10.0->transformers==4.25.1) (2023.6.0) Requirement already satisfied: typing-extensions>=3.7.4.3 in ./metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.10.0->transformers==4.25.1) (4.7.0) Requirement already satisfied: charset-normalizer<4,>=2 in ./metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages (from requests->transformers==4.25.1) (3.1.0) Requirement already satisfied: idna<4,>=2.5 in ./metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages (from requests->transformers==4.25.1) (3.4) Requirement already satisfied: urllib3<3,>=1.21.1 in ./metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages (from requests->transformers==4.25.1) (1.26.15) Requirement already satisfied: certifi>=2017.4.17 in ./metaflow_T5DDPFlow_linux-64_53554079b7dbe8c9b8991241feb21cc7cda7c922/lib/python3.10/site-packages (from requests->transformers==4.25.1) (2023.5.7) Installing collected packages: tokenizers, regex, filelock, huggingface-hub, transformers Successfully installed filelock-3.12.2 huggingface-hub-0.16.2 regex-2023.6.3 tokenizers-0.13.3 transformers-4.25.1 Pip Install: ninja 1.11.1 Collecting ninja==1.11.1 Downloading ninja-1.11.1-py2.py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (145 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 146.0/146.0 kB 7.0 MB/s eta 0:00:00 Installing collected packages: ninja Successfully installed ninja-1.11.1 Pip Install: click 8.0.4 Collecting click==8.0.4 Downloading click-8.0.4-py3-none-any.whl (97 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.5/97.5 kB 6.2 MB/s eta 0:00:00 Installing collected packages: click Successfully installed click-8.0.4 Torch CUDA available? True Reading in data and save to csv ... Calling PTL process... ip-10-14-52-44:531:531 [3] NCCL INFO Bootstrap : Using eth0:10.14.52.44<0> ip-10-14-52-44:531:531 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ip-10-14-52-44:531:531 [3] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1] ip-10-14-52-44:531:531 [3] NCCL INFO NET/Socket : Using [0]eth0:10.14.52.44<0> ip-10-14-52-44:531:531 [3] NCCL INFO Using network Socket ip-10-14-52-44:529:529 [1] NCCL INFO Bootstrap : Using eth0:10.14.52.44<0> ip-10-14-52-44:529:529 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ip-10-14-52-44:529:529 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1] ip-10-14-52-44:529:529 [1] NCCL INFO NET/Socket : Using [0]eth0:10.14.52.44<0> ip-10-14-52-44:529:529 [1] NCCL INFO Using network Socket ip-10-14-52-44:528:528 [0] NCCL INFO Bootstrap : Using eth0:10.14.52.44<0> ip-10-14-52-44:528:528 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ip-10-14-52-44:528:528 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1] ip-10-14-52-44:528:528 [0] NCCL INFO NET/Socket : Using [0]eth0:10.14.52.44<0> ip-10-14-52-44:528:528 [0] NCCL INFO Using network Socket ip-10-14-52-44:530:530 [2] NCCL INFO Bootstrap : Using eth0:10.14.52.44<0> ip-10-14-52-44:530:530 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ip-10-14-52-44:530:530 [2] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1] ip-10-14-52-44:530:530 [2] NCCL INFO NET/Socket : Using [0]eth0:10.14.52.44<0> ip-10-14-52-44:530:530 [2] NCCL INFO Using network Socket ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 0(=1b0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 0(=1b0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 0(=1b0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 0(=1b0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 0(=1b0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 0(=1b0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 1(=1c0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 1(=1c0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 1(=1c0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 1(=1c0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 1(=1c0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 1(=1c0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 2(=1d0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 2(=1d0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 2(=1d0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 2(=1d0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 2(=1d0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 2(=1d0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 3(=1e0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 3(=1e0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 3(=1e0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 3(=1e0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 3(=1e0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 3(=1e0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 0(=1b0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 0(=1b0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 0(=1b0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 0(=1b0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 1(=1c0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 0(=1b0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 1(=1c0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 0(=1b0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 1(=1c0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 1(=1c0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 2(=1d0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 1(=1c0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 2(=1d0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 1(=1c0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 2(=1d0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 2(=1d0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 3(=1e0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 2(=1d0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 3(=1e0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 2(=1d0) ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 3(=1e0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 3(=1e0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 3(=1e0) ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 3(=1e0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 0(=1b0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 0(=1b0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 0(=1b0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 1(=1c0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 1(=1c0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 1(=1c0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 2(=1d0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 2(=1d0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 2(=1d0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 3(=1e0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 3(=1e0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 0(=1b0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 3(=1e0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 0(=1b0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 0(=1b0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 1(=1c0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 1(=1c0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 1(=1c0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 2(=1d0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 0(=1b0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 2(=1d0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 0(=1b0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 2(=1d0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 0(=1b0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 3(=1e0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 1(=1c0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 3(=1e0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 1(=1c0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 3(=1e0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 1(=1c0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 2(=1d0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 2(=1d0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 2(=1d0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 3(=1e0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 3(=1e0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 0(=1b0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 3(=1e0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 0(=1b0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 0(=1b0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 1(=1c0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 1(=1c0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 1(=1c0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 2(=1d0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 2(=1d0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 2(=1d0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 3(=1e0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 3(=1e0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 3(=1e0) ip-10-14-52-44:530:658 [2] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 ip-10-14-52-44:529:656 [1] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/8/-1->5->4 ip-10-14-52-44:531:655 [3] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 ip-10-14-52-44:528:657 [0] NCCL INFO Trees [0] 5/-1/-1->4->9 [1] 5/0/-1->4->12 ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 1(=1c0) ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 0(=1b0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 2(=1d0) ip-10-14-52-44:528:657 [0] NCCL INFO Channel 00 : 3[1e0] -> 4[1b0] [receive] via NET/Socket/0 ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 1(=1c0) ip-10-14-52-44:528:657 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 0(=1b0) ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 2(=1d0) ip-10-14-52-44:528:657 [0] NCCL INFO Channel 01 : 3[1e0] -> 4[1b0] [receive] via NET/Socket/0 ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 3(=1e0) ip-10-14-52-44:528:657 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 1(=1c0) ip-10-14-52-44:528:657 [0] NCCL INFO Channel 00 : 4[1b0] -> 5[1c0] via direct shared memory ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 1(=1c0) ip-10-14-52-44:530:658 [2] NCCL INFO Channel 00 : 6[1d0] -> 7[1e0] via direct shared memory ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 3(=1e0) ip-10-14-52-44:528:657 [0] NCCL INFO Channel 01 : 4[1b0] -> 5[1c0] via direct shared memory ip-10-14-52-44:530:658 [2] NCCL INFO Channel 01 : 6[1d0] -> 7[1e0] via direct shared memory ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 2(=1d0) ip-10-14-52-44:529:656 [1] NCCL INFO Channel 00 : 5[1c0] -> 6[1d0] via direct shared memory ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 2(=1d0) ip-10-14-52-44:529:656 [1] NCCL INFO Channel 01 : 5[1c0] -> 6[1d0] via direct shared memory ip-10-14-52-44:531:655 [3] NCCL INFO Channel 00 : 7[1e0] -> 8[1b0] [send] via NET/Socket/0 ip-10-14-52-44:529:656 [1] NCCL INFO Connected all rings ip-10-14-52-44:531:655 [3] NCCL INFO Channel 01 : 7[1e0] -> 8[1b0] [send] via NET/Socket/0 ip-10-14-52-44:529:656 [1] NCCL INFO Channel 01 : 5[1c0] -> 8[1b0] [send] via NET/Socket/0 ip-10-14-52-44:530:658 [2] NCCL INFO Connected all rings ip-10-14-52-44:530:658 [2] NCCL INFO Could not enable P2P between dev 2(=1d0) and dev 3(=1e0) ip-10-14-52-44:530:658 [2] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device ip-10-14-52-44:530:658 [2] NCCL INFO include/shm.h:41 -> 2 ip-10-14-52-44:530:658 [2] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-a8d5333cdcb26a56-0-7-6 (size 9637888) ip-10-14-52-44:530:658 [2] NCCL INFO transport/shm.cc:100 -> 2 ip-10-14-52-44:530:658 [2] NCCL INFO transport.cc:34 -> 2 ip-10-14-52-44:530:658 [2] NCCL INFO transport.cc:84 -> 2 ip-10-14-52-44:530:658 [2] NCCL INFO init.cc:789 -> 2 ip-10-14-52-44:530:658 [2] NCCL INFO init.cc:904 -> 2 ip-10-14-52-44:530:658 [2] NCCL INFO group.cc:72 -> 2 [Async thread] ip-10-14-52-44:528:657 [0] NCCL INFO Connected all rings ip-10-14-52-44:528:657 [0] NCCL INFO Channel 01 : 0[1b0] -> 4[1b0] [receive] via NET/Socket/0 ip-10-14-52-44:528:657 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread ip-10-14-52-44:531:655 [3] NCCL INFO Connected all rings ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 2(=1d0) ip-10-14-52-44:531:655 [3] NCCL INFO Channel 00 : 7[1e0] -> 6[1d0] via direct shared memory ip-10-14-52-44:531:655 [3] NCCL INFO Could not enable P2P between dev 3(=1e0) and dev 2(=1d0) ip-10-14-52-44:531:655 [3] NCCL INFO Channel 01 : 7[1e0] -> 6[1d0] via direct shared memory ip-10-14-52-44:528:657 [0] NCCL INFO Channel 00 : 4[1b0] -> 9[1c0] [send] via NET/Socket/0 ip-10-14-52-44:528:657 [0] NCCL INFO Channel 01 : 12[1b0] -> 4[1b0] [receive] via NET/Socket/0 ip-10-14-52-44:528:657 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread ip-10-14-52-44:528:657 [0] NCCL INFO Channel 01 : 4[1b0] -> 12[1b0] [send] via NET/Socket/0 ip-10-14-52-44:529:656 [1] NCCL INFO Channel 01 : 8[1b0] -> 5[1c0] [receive] via NET/Socket/0 ip-10-14-52-44:529:656 [1] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread ip-10-14-52-44:528:657 [0] NCCL INFO Channel 00 : 9[1c0] -> 4[1b0] [receive] via NET/Socket/0 ip-10-14-52-44:528:657 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread ip-10-14-52-44:528:657 [0] NCCL INFO Channel 01 : 4[1b0] -> 0[1b0] [send] via NET/Socket/0 ip-10-14-52-44:528:657 [0] NCCL INFO Could not enable P2P between dev 0(=1b0) and dev 1(=1c0) ip-10-14-52-44:528:657 [0] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device ip-10-14-52-44:528:657 [0] NCCL INFO include/shm.h:41 -> 2 ip-10-14-52-44:528:657 [0] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-2aef21165edcff7f-0-5-4 (size 9637888) ip-10-14-52-44:528:657 [0] NCCL INFO transport/shm.cc:100 -> 2 ip-10-14-52-44:528:657 [0] NCCL INFO transport.cc:34 -> 2 ip-10-14-52-44:528:657 [0] NCCL INFO transport.cc:84 -> 2 ip-10-14-52-44:528:657 [0] NCCL INFO init.cc:789 -> 2 ip-10-14-52-44:528:657 [0] NCCL INFO init.cc:904 -> 2 ip-10-14-52-44:528:657 [0] NCCL INFO group.cc:72 -> 2 [Async thread] ip-10-14-52-44:529:656 [1] NCCL INFO Could not enable P2P between dev 1(=1c0) and dev 2(=1d0) ip-10-14-52-44:529:656 [1] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device ip-10-14-52-44:529:656 [1] NCCL INFO include/shm.h:41 -> 2 ip-10-14-52-44:529:656 [1] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-1a3e2abbc73f77e-0-6-5 (size 9637888) ip-10-14-52-44:529:656 [1] NCCL INFO transport/shm.cc:100 -> 2 ip-10-14-52-44:529:656 [1] NCCL INFO transport.cc:34 -> 2 ip-10-14-52-44:529:656 [1] NCCL INFO transport.cc:84 -> 2 ip-10-14-52-44:529:656 [1] NCCL INFO init.cc:789 -> 2 ip-10-14-52-44:529:656 [1] NCCL INFO init.cc:904 -> 2 ip-10-14-52-44:529:656 [1] NCCL INFO group.cc:72 -> 2 [Async thread] ip-10-14-52-44:531:655 [3] NCCL INFO Call to connect returned Connection refused, retrying ip-10-14-52-44:531:655 [3] NCCL INFO Call to connect returned Connection refused, retrying