WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv 2023-07-22 05:42:56,994 WARNING services.py:1832 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67108864 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=7.40gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM. 2023-07-22 05:43:01,610 INFO worker.py:1431 -- Connecting to existing Ray cluster at address: 10.14.51.232:6379... 2023-07-22 05:43:01,616 INFO worker.py:1612 -- Connected to Ray cluster. View the dashboard at 127.0.0.1:8265  2023-07-22 05:43:02,640 INFO tune.py:666 -- [output] This will use the new output engine with verbosity 1. To disable the new output and use the legacy output engine, set the environment variable RAY_AIR_NEW_OUTPUT=0. For more information, please see https://github.com/ray-project/ray/issues/36949 (LightningTrainer pid=481, ip=10.14.51.201) Starting distributed worker processes: ['534 (10.14.51.201)', '530 (10.14.51.246)', '518 (10.14.51.218)', '1040 (10.14.51.232)'] (LightningTrainer pid=481, ip=10.14.51.201) The actor died unexpectedly before finishing this task. (LightningTrainer pid=481, ip=10.14.51.201) Traceback (most recent call last): (LightningTrainer pid=481, ip=10.14.51.201) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/train/_internal/backend_executor.py", line 154, in start (LightningTrainer pid=481, ip=10.14.51.201) self._backend.on_start(self.worker_group, self._backend_config) (LightningTrainer pid=481, ip=10.14.51.201) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/train/torch/config.py", line 170, in on_start (LightningTrainer pid=481, ip=10.14.51.201) worker_group.execute(_set_nccl_network_interface) (LightningTrainer pid=481, ip=10.14.51.201) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/train/_internal/worker_group.py", line 272, in execute (LightningTrainer pid=481, ip=10.14.51.201) return ray.get(self.execute_async(func, *args, **kwargs)) (LightningTrainer pid=481, ip=10.14.51.201) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 24, in auto_init_wrapper (LightningTrainer pid=481, ip=10.14.51.201) return fn(*args, **kwargs) (LightningTrainer pid=481, ip=10.14.51.201) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper (LightningTrainer pid=481, ip=10.14.51.201) return func(*args, **kwargs) (LightningTrainer pid=481, ip=10.14.51.201) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/_private/worker.py", line 2495, in get (LightningTrainer pid=481, ip=10.14.51.201) raise value (LightningTrainer pid=481, ip=10.14.51.201) ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task. (LightningTrainer pid=481, ip=10.14.51.201) Failure occurred during startup. Restarting all workers and attempting to startup again. 2023-07-22 05:43:41,545 ERROR tune_controller.py:911 -- Trial task failed for trial LightningTrainer_9fad8_00000 Traceback (most recent call last): File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/air/execution/_internal/event_manager.py", line 110, in resolve_future result = ray.get(future) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 24, in auto_init_wrapper return fn(*args, **kwargs) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper return func(*args, **kwargs) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/_private/worker.py", line 2493, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(TypeError): ray::_Inner.train() (pid=481, ip=10.14.51.201, actor_id=5c3f29916a63dfb3a0b9a9e001000000, repr=LightningTrainer) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/tune/trainable/trainable.py", line 375, in train raise skipped from exception_cause(skipped) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/tune/trainable/function_trainable.py", line 349, in entrypoint return self._trainable_func( File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/train/base_trainer.py", line 809, in _trainable_func super()._trainable_func(self._merged_config, reporter, checkpoint_dir) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/tune/trainable/function_trainable.py", line 666, in _trainable_func output = fn() File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/train/base_trainer.py", line 719, in train_func trainer.training_loop() File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/train/data_parallel_trainer.py", line 491, in training_loop backend_executor.start(initialization_hook=clear_lazy_checkpoint_marker) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/train/_internal/backend_executor.py", line 161, in start self._increment_failures() File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/train/_internal/backend_executor.py", line 647, in _increment_failures raise failure TypeError: exceptions must derive from BaseException 2023-07-22 05:43:41,551 ERROR tune.py:1144 -- Trials did not complete: [LightningTrainer_9fad8_00000] 2023-07-22 05:43:41,554 WARNING experiment_analysis.py:916 -- Failed to read the results for 1 trials: - /tmp/ray_results/ptl-mnist-example/LightningTrainer_9fad8_00000_0_2023-07-22_05-43-02 failed: Internal error ray.exceptions.RayTaskError(TypeError): ray::_Inner.train() (pid=481, ip=10.14.51.201, actor_id=5c3f29916a63dfb3a0b9a9e001000000, repr=LightningTrainer) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/tune/trainable/trainable.py", line 375, in train raise skipped from exception_cause(skipped) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/tune/trainable/function_trainable.py", line 349, in entrypoint return self._trainable_func( File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/train/base_trainer.py", line 809, in _trainable_func super()._trainable_func(self._merged_config, reporter, checkpoint_dir) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/tune/trainable/function_trainable.py", line 666, in _trainable_func output = fn() File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/train/base_trainer.py", line 719, in train_func trainer.training_loop() File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/train/data_parallel_trainer.py", line 491, in training_loop backend_executor.start(initialization_hook=clear_lazy_checkpoint_marker) File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/train/_internal/backend_executor.py", line 161, in start self._increment_failures() File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/train/_internal/backend_executor.py", line 647, in _increment_failures raise failure TypeError: exceptions must derive from BaseException The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/metaflow/metaflow/cli.py", line 1172, in main start(auto_envvar_prefix="METAFLOW", obj=state) File "/metaflow/metaflow/_vendor/click/core.py", line 829, in __call__ return self.main(args, kwargs) File "/metaflow/metaflow/_vendor/click/core.py", line 782, in main rv = self.invoke(ctx) File "/metaflow/metaflow/_vendor/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/metaflow/metaflow/_vendor/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/metaflow/metaflow/_vendor/click/core.py", line 610, in invoke return callback(args, kwargs) File "/metaflow/metaflow/_vendor/click/decorators.py", line 21, in new_func return f(get_current_context(), args, kwargs) File "/metaflow/metaflow/cli.py", line 581, in step task.run_step( File "/metaflow/metaflow/task.py", line 586, in run_step self._exec_step_function(step_func) File "/metaflow/metaflow/task.py", line 60, in _exec_step_function step_function() File "/metaflow/metaflow/plugins/parallel_decorator.py", line 34, in _step_func_with_setup step_func() File "/metaflow/gpu_profile.py", line 148, in func f(s) File "/metaflow/ray_flow.py", line 121, in train result = trainer.fit() File "/metaflow/metaflow_RayFlow_linux-64_08124c2385d70038b03812e97537183f74f6b22c/lib/python3.10/site-packages/ray/train/base_trainer.py", line 630, in fit raise TrainingFailedError( ray.train.base_trainer.TrainingFailedError: The Ray Train run failed. Please inspect the previous error messages for a cause. After fixing the issue (assuming that the error is not caused by your own application logic, but rather an error such as OOM), you can restart the run from scratch or continue this run. To continue this run, you can use: `trainer = LightningTrainer.restore("/tmp/ray_results/ptl-mnist-example")`. To start a new run that will retry on training failures, set `air.RunConfig(failure_config=air.FailureConfig(max_failures))` in the Trainer's `run_config` with `max_failures > 0`, or `max_failures = -1` for unlimited retries. AWS Batch error: Essential container in task exited This could be a transient error. Use @retry to retry. Task failed.