2024-08-06 09:09:22.324 All packages already cached in s3. 2024-08-06 09:09:22.325 Caching 2 environments and aliases to s3 ...2024-08-06 09:09:22.504 done in 0 seconds. 2024-08-06 09:09:23.035 Workflow starting (run-id 1857): 2024-08-06 09:09:25.007 Creating Conda environment d0a3e13ca27571df018ab89ed0b6dd649ffca3a8 (4f335769efbc1faf587e200c9eedb0926730dd43)... 2024-08-06 09:09:25.406 Extracting and linking Conda environment ... (conda packages) ... (pypi packages) ... done in 11 seconds. 2024-08-06 09:09:36.445 [1857/start/23755 (pid 41785)] Task is starting. 2024-08-06 09:09:43.839 [1857/start/23755 (pid 41785)] /Users/mikebentleymills/projects/eru/production_flow.py:10: DeprecationWarning: 2024-08-06 09:09:47.049 [1857/start/23755 (pid 41785)] Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0), 2024-08-06 09:09:47.049 [1857/start/23755 (pid 41785)] (to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries) 2024-08-06 09:09:47.049 [1857/start/23755 (pid 41785)] but was not found to be installed on your system. 2024-08-06 09:09:47.049 [1857/start/23755 (pid 41785)] If this would cause problems for you, 2024-08-06 09:09:47.049 [1857/start/23755 (pid 41785)] please provide us feedback at 2024-08-06 09:09:47.049 [1857/start/23755 (pid 41785)] 2024-08-06 09:09:47.050 [1857/start/23755 (pid 41785)] import pandas as pd 2024-08-06 09:09:47.344 [1857/start/23755 (pid 41785)] Foreach yields 1 child steps. 2024-08-06 09:09:47.344 [1857/start/23755 (pid 41785)] Task finished successfully. 2024-08-06 09:09:48.477 Using existing Conda environment d0a3e13ca27571df018ab89ed0b6dd649ffca3a8 (4f335769efbc1faf587e200c9eedb0926730dd43) 2024-08-06 09:09:48.483 [1857/process_orders/23756 (pid 42056)] Task is starting. 2024-08-06 09:09:48.741 [1857/process_orders/23756 (pid 42056)] /Users/mikebentleymills/projects/eru/production_flow.py:10: DeprecationWarning: 2024-08-06 09:09:49.923 [1857/process_orders/23756 (pid 42056)] Order Artifacts Storage: s3://mike-test-slides/processed/0001-00000054/artifacts 2024-08-06 09:09:51.315 [1857/process_orders/23756 (pid 42056)] Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0), 2024-08-06 09:09:51.315 [1857/process_orders/23756 (pid 42056)] (to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries) 2024-08-06 09:09:51.315 [1857/process_orders/23756 (pid 42056)] but was not found to be installed on your system. 2024-08-06 09:09:51.315 [1857/process_orders/23756 (pid 42056)] If this would cause problems for you, 2024-08-06 09:09:51.315 [1857/process_orders/23756 (pid 42056)] please provide us feedback at 2024-08-06 09:09:51.315 [1857/process_orders/23756 (pid 42056)] 2024-08-06 09:09:51.315 [1857/process_orders/23756 (pid 42056)] import pandas as pd 2024-08-06 09:09:51.578 [1857/process_orders/23756 (pid 42056)] Task finished successfully. 2024-08-06 09:09:52.258 Using existing Conda environment d0a3e13ca27571df018ab89ed0b6dd649ffca3a8 (4f335769efbc1faf587e200c9eedb0926730dd43) 2024-08-06 09:09:52.272 [1857/slide_conversion/23757 (pid 42148)] Task is starting. 2024-08-06 09:09:52.576 [1857/slide_conversion/23757 (pid 42148)] /Users/mikebentleymills/projects/eru/production_flow.py:10: DeprecationWarning: 2024-08-06 09:09:54.077 [1857/slide_conversion/23757 (pid 42148)] Metaflow 2.12.10+netflix-ext(1.2.1) executing SlideToTIFF for user:mikebentleymills 2024-08-06 09:09:54.102 [1857/slide_conversion/23757 (pid 42148)] Validating your flow... 2024-08-06 09:09:54.102 [1857/slide_conversion/23757 (pid 42148)] The graph looks good! 2024-08-06 09:09:54.102 [1857/slide_conversion/23757 (pid 42148)] Running pylint... 2024-08-06 09:09:54.102 [1857/slide_conversion/23757 (pid 42148)] Pylint not found, so extra checks are disabled. 2024-08-06 09:09:54.102 [1857/slide_conversion/23757 (pid 42148)] Bootstrapping Conda environment... (this could take a few minutes) 2024-08-06 09:09:55.044 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:09:55.044 All packages already cached in s3. 2024-08-06 09:09:55.598 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:09:55.044 All environments already cached in s3. 2024-08-06 09:09:55.598 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:09:55.598 Workflow starting (run-id 1858): 2024-08-06 09:09:58.003 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:09:58.002 Creating Conda environment bac099b8ef09ba99f32812140d85680ca66ecc56 (a2271166301c8d4332566033673762770b78f643)... 2024-08-06 09:10:04.837 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:09:58.410 Extracting and linking Conda environment ... done in 6 seconds. 2024-08-06 09:10:04.840 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:04.840 [1858/start/23759 (pid 42697)] Task is starting. 2024-08-06 09:10:10.355 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:10.355 [1858/start/23759 (pid 42697)] Ignoring previous processed slides. 2024-08-06 09:10:10.420 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:10.420 [1858/start/23759 (pid 42697)] This is going to reprocess everything! in ['s3://valar-portal-prod/0001/drafts/D-0001-00000067/raw/D-0001-00000067_1b945591-1e2d-4fe2-aab1-d1889af92955_S24-13531_A1-1_H&E_Slide(s)_20240718_093020.isyntax'] ... 2024-08-06 09:10:10.471 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:10.471 [1858/start/23759 (pid 42697)] Found 1 files to process. 2024-08-06 09:10:11.948 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:11.948 [1858/start/23759 (pid 42697)] Found 0 general files. 2024-08-06 09:10:12.262 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:11.948 [1858/start/23759 (pid 42697)] Found 1 isyntax files. 2024-08-06 09:10:12.263 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:12.262 [1858/start/23759 (pid 42697)] Task finished successfully. 2024-08-06 09:10:13.340 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:13.340 Using existing Conda environment bac099b8ef09ba99f32812140d85680ca66ecc56 (a2271166301c8d4332566033673762770b78f643) 2024-08-06 09:10:13.343 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:13.343 [1858/isyntax_formats/23760 (pid 42928)] Task is starting. 2024-08-06 09:10:15.768 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:15.768 [1858/isyntax_formats/23760 (pid 42928)] Foreach yields 2 child steps. 2024-08-06 09:10:16.435 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:15.768 [1858/isyntax_formats/23760 (pid 42928)] Task finished successfully. 2024-08-06 09:10:16.435 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:16.434 Using existing Conda environment bac099b8ef09ba99f32812140d85680ca66ecc56 (a2271166301c8d4332566033673762770b78f643) 2024-08-06 09:10:16.442 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:16.442 [1858/general_formats/23761 (pid 43022)] Task is starting. 2024-08-06 09:10:18.811 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:18.811 [1858/general_formats/23761 (pid 43022)] Foreach yields 1 child steps. 2024-08-06 09:10:19.723 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:18.811 [1858/general_formats/23761 (pid 43022)] Task finished successfully. 2024-08-06 09:10:19.723 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:19.722 [1858/process_isyntax/23762 (pid 43141)] Task is starting. 2024-08-06 09:10:21.272 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:10:21.271 [1858/process_isyntax/23762 (pid 43141)] [pod t-bbff360d-8srz8-jpxtt] Task is starting (Pod is pending)... 2024-08-06 09:12:12.123 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:11:54.983 [1858/process_isyntax/23762 (pid 43141)] [pod t-bbff360d-8srz8-jpxtt] Setting up task environment. 2024-08-06 09:12:12.123 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:06.046 [1858/process_isyntax/23762 (pid 43141)] [pod t-bbff360d-8srz8-jpxtt] Downloading code package... 2024-08-06 09:12:12.123 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:06.765 [1858/process_isyntax/23762 (pid 43141)] [pod t-bbff360d-8srz8-jpxtt] Code package downloaded. 2024-08-06 09:12:12.164 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:06.822 [1858/process_isyntax/23762 (pid 43141)] [pod t-bbff360d-8srz8-jpxtt] Task is starting. 2024-08-06 09:12:12.164 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:07.208 [1858/process_isyntax/23762 (pid 43141)] [pod t-bbff360d-8srz8-jpxtt] Bootstrapping environment ... 2024-08-06 09:12:12.206 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:10.365 [1858/process_isyntax/23762 (pid 43141)] [pod t-bbff360d-8srz8-jpxtt] Setting up Conda ... done in 3 seconds. 2024-08-06 09:12:12.207 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:10.530 [1858/process_isyntax/23762 (pid 43141)] [pod t-bbff360d-8srz8-jpxtt] Traceback (most recent call last): 2024-08-06 09:12:12.207 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.206 [1858/process_isyntax/23762 (pid 43141)] Kubernetes error: 2024-08-06 09:12:12.207 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.206 [1858/process_isyntax/23762 (pid 43141)] Error: Setting up task environment. 2024-08-06 09:12:12.207 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.206 [1858/process_isyntax/23762 (pid 43141)] Downloading code package... 2024-08-06 09:12:12.207 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.206 [1858/process_isyntax/23762 (pid 43141)] Code package downloaded. 2024-08-06 09:12:12.207 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.207 [1858/process_isyntax/23762 (pid 43141)] Task is starting. 2024-08-06 09:12:12.207 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.207 [1858/process_isyntax/23762 (pid 43141)] Bootstrapping environment ... 2024-08-06 09:12:12.207 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.207 [1858/process_isyntax/23762 (pid 43141)] Setting up Conda ... done in 3 seconds. 2024-08-06 09:12:12.207 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.207 [1858/process_isyntax/23762 (pid 43141)] Traceback (most recent call last): 2024-08-06 09:12:12.207 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.207 [1858/process_isyntax/23762 (pid 43141)] File "/usr/lib/python3.8/runpy.py", line 194, in_run_module_as_main 2024-08-06 09:12:12.207 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.207 [1858/process_isyntax/23762 (pid 43141)] return_run_code(code, main_globals, None, 2024-08-06 09:12:12.207 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.207 [1858/process_isyntax/23762 (pid 43141)] File "/usr/lib/python3.8/runpy.py", line 87, in _run_code 2024-08-06 09:12:12.207 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.207 [1858/process_isyntax/23762 (pid 43141)] exec(code, run_globals) 2024-08-06 09:12:12.208 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.207 [1858/process_isyntax/23762 (pid 43141)] File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/remote_bootstrap.py", line 97, in 2024-08-06 09:12:12.330 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.207 [1858/process_isyntax/23762 (pid 43141)] bootstrap_environment(sys.argv[1:]) 2024-08-06 09:12:12.330 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.330 [1858/process_isyntax/23762 (pid 43141)] File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/remote_bootstrap.py", line 55, in bootstrap_environment 2024-08-06 09:12:12.330 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.330 [1858/process_isyntax/23762 (pid 43141)] raise RuntimeError( 2024-08-06 09:12:12.330 [1857/slide_conversion/23757 (pid 42148)] 2024-08-06 09:12:12.330 [1858/process_isyntax/23762 (pid 43141)] RuntimeError: Cannot find cached environment for hash d0a3e13ca27571df018ab89ed0b6dd649ffca3a8:4f335769efbc1faf587e200c9eedb0926730dd43