Metaflow 2.12.10+netflix-ext(1.2.1) executing ProductionPipelineFlow for user:mikebentleymills
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Bootstrapping Conda environment... (this could take a few minutes)
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step start, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step start, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', '--version']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:275]: Conda call: ['/Users/mikebentleymills/miniforge3/condabin/mamba', 'info', '--json']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step start, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:204]: Added environment to resolve {'id': EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64'), 'steps': ['start'], 'user_deps': {'pypi': ['pandas==2.2', 'requests==>=2.21.0', 'boto3==>=1.14.0', 'pydantic==2.6.1', 'kubernetes==29', 'pyjson5==1.6.6', 'metaflow-netflixext==1.2.1', 'metaflow==2.12.8'], 'conda': ['python==3.11.7'], 'sys': []}, 'deps': {'pypi': ['pandas==2.2', 'requests==>=2.21.0', 'boto3==>=1.14.0', 'pydantic==2.6.1', 'kubernetes==29', 'pyjson5==1.6.6', 'metaflow-netflixext==1.2.1', 'metaflow==2.12.8'], 'conda': ['python==3.11.7'], 'sys': []}, 'sources': {'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}, 'extras': {}, 'conda_format': ['_any'], 'base': None, 'base_accurate': None, 'resolved': <metaflow_extensions.netflix_ext.plugins.conda.env_descr.ResolvedEnvironment object at 0x1466e6810>, 'already_resolved': True, 'env_type': <EnvType.PYPI_ONLY: 'pypi-only'>, 'force': False}
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step process_orders, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step process_orders, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step process_orders, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'process_orders'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step slide_conversion, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step slide_conversion, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step slide_conversion, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'slide_conversion'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step rename_slides, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step rename_slides, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step rename_slides, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'rename_slides'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step generate_geometry, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step generate_geometry, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step generate_geometry, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'generate_geometry'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step ssd_inference, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step ssd_inference, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step ssd_inference, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'ssd_inference'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step segmentation_inference, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step segmentation_inference, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step segmentation_inference, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'segmentation_inference'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step tissue_inference, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step tissue_inference, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step tissue_inference, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'tissue_inference'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step tissue_geojson, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step tissue_geojson, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step tissue_geojson, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'tissue_geojson'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step inference_join, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step inference_join, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step inference_join, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'inference_join'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step component_rasterization, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step component_rasterization, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step component_rasterization, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'component_rasterization'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step component_feature_computation, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step component_feature_computation, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step component_feature_computation, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'component_feature_computation'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step component_slide_features, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step component_slide_features, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step component_slide_features, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'component_slide_features'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step component_stats, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step component_stats, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step component_stats, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'component_stats'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step order_process_join, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step order_process_join, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step order_process_join, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'order_process_join'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step end, got flow req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]; step req: StepReq[]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step end, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}}; sources={'pypi': ['https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/micromamba/envs/valar/bin/pip', 'config', 'list']
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step end, final req: StepReq[disabled=False; python=3.11.7; packages={'pypi': {'pandas': '2.2', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'pydantic': '2.6.1', 'kubernetes': '29', 'pyjson5': '1.6.6', 'metaflow-netflixext': '1.2.1', 'metaflow': '2.12.8'}, 'conda': {'python': '3.11.7'}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults'], 'pypi': ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cu118']}]
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64')' is also needed by 'end'
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:229]: Resolving environments:
2024-08-06 13:13:43.335     All packages already cached in s3.
2024-08-06 13:13:43.335     All environments already cached in s3.
2024-08-06 13:13:43.680 Workflow starting (run-id 1892):
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2031]: Locating MF environment metaflow_d0a3e13ca27571df018ab89ed0b6dd649ffca3a8_4f335769efbc1faf587e200c9eedb0926730dd43
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2022]: Found EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='4f335769efbc1faf587e200c9eedb0926730dd43', arch='osx-arm64') at dir_name /Users/mikebentleymills/miniforge3/envs/metaflow_d0a3e13ca27571df018ab89ed0b6dd649ffca3a8_4f335769efbc1faf587e200c9eedb0926730dd43
2024-08-06 13:13:45.459 Using existing Conda environment d0a3e13ca27571df018ab89ed0b6dd649ffca3a8 (4f335769efbc1faf587e200c9eedb0926730dd43)
2024-08-06 13:13:45.474 [1892/start/24171 (pid 52692)] Task is starting.
2024-08-06 13:13:45.725 [1892/start/24171 (pid 52692)] /Users/mikebentleymills/projects/eru/production_flow.py:10: DeprecationWarning:
2024-08-06 13:13:45.966 [1892/start/24171 (pid 52692)] Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),
2024-08-06 13:13:45.966 [1892/start/24171 (pid 52692)] (to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)
2024-08-06 13:13:45.966 [1892/start/24171 (pid 52692)] but was not found to be installed on your system.
2024-08-06 13:13:45.966 [1892/start/24171 (pid 52692)] If this would cause problems for you,
2024-08-06 13:13:45.966 [1892/start/24171 (pid 52692)] please provide us feedback at https://github.com/pandas-dev/pandas/issues/54466
2024-08-06 13:13:45.966 [1892/start/24171 (pid 52692)]
2024-08-06 13:13:45.966 [1892/start/24171 (pid 52692)] import pandas as pd
2024-08-06 13:13:45.966 [1892/start/24171 (pid 52692)] debug[conda /tmp/tmpbpcemsii/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:13:48.219 [1892/start/24171 (pid 52692)] Foreach yields 1 child steps.
2024-08-06 13:13:48.219 [1892/start/24171 (pid 52692)] Task finished successfully.
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2031]: Locating MF environment metaflow_d0a3e13ca27571df018ab89ed0b6dd649ffca3a8_4f335769efbc1faf587e200c9eedb0926730dd43
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2022]: Found EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='4f335769efbc1faf587e200c9eedb0926730dd43', arch='osx-arm64') at dir_name /Users/mikebentleymills/miniforge3/envs/metaflow_d0a3e13ca27571df018ab89ed0b6dd649ffca3a8_4f335769efbc1faf587e200c9eedb0926730dd43
2024-08-06 13:13:48.938 Using existing Conda environment d0a3e13ca27571df018ab89ed0b6dd649ffca3a8 (4f335769efbc1faf587e200c9eedb0926730dd43)
2024-08-06 13:13:48.950 [1892/process_orders/24172 (pid 52783)] Task is starting.
2024-08-06 13:13:49.218 [1892/process_orders/24172 (pid 52783)] /Users/mikebentleymills/projects/eru/production_flow.py:10: DeprecationWarning:
2024-08-06 13:13:49.457 [1892/process_orders/24172 (pid 52783)] Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),
2024-08-06 13:13:49.457 [1892/process_orders/24172 (pid 52783)] (to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)
2024-08-06 13:13:49.457 [1892/process_orders/24172 (pid 52783)] but was not found to be installed on your system.
2024-08-06 13:13:49.457 [1892/process_orders/24172 (pid 52783)] If this would cause problems for you,
2024-08-06 13:13:49.457 [1892/process_orders/24172 (pid 52783)] please provide us feedback at https://github.com/pandas-dev/pandas/issues/54466
2024-08-06 13:13:49.457 [1892/process_orders/24172 (pid 52783)]
2024-08-06 13:13:49.457 [1892/process_orders/24172 (pid 52783)] import pandas as pd
2024-08-06 13:13:49.457 [1892/process_orders/24172 (pid 52783)] debug[conda /tmp/tmpawu727w7/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:13:50.367 [1892/process_orders/24172 (pid 52783)] Order Storage:  s3://mike-test-slides/processed/0001-00000054-1892/artifacts
2024-08-06 13:13:52.172 [1892/process_orders/24172 (pid 52783)] Task finished successfully.
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2031]: Locating MF environment metaflow_d0a3e13ca27571df018ab89ed0b6dd649ffca3a8_4f335769efbc1faf587e200c9eedb0926730dd43
debug[conda /Users/mikebentleymills/micromamba/envs/valar/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2022]: Found EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='4f335769efbc1faf587e200c9eedb0926730dd43', arch='osx-arm64') at dir_name /Users/mikebentleymills/miniforge3/envs/metaflow_d0a3e13ca27571df018ab89ed0b6dd649ffca3a8_4f335769efbc1faf587e200c9eedb0926730dd43
2024-08-06 13:13:52.819 Using existing Conda environment d0a3e13ca27571df018ab89ed0b6dd649ffca3a8 (4f335769efbc1faf587e200c9eedb0926730dd43)
2024-08-06 13:13:52.829 [1892/slide_conversion/24173 (pid 52846)] Task is starting.
2024-08-06 13:13:53.121 [1892/slide_conversion/24173 (pid 52846)] /Users/mikebentleymills/projects/eru/production_flow.py:10: DeprecationWarning:
2024-08-06 13:13:53.357 [1892/slide_conversion/24173 (pid 52846)] Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),
2024-08-06 13:13:53.357 [1892/slide_conversion/24173 (pid 52846)] (to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)
2024-08-06 13:13:53.357 [1892/slide_conversion/24173 (pid 52846)] but was not found to be installed on your system.
2024-08-06 13:13:53.357 [1892/slide_conversion/24173 (pid 52846)] If this would cause problems for you,
2024-08-06 13:13:53.357 [1892/slide_conversion/24173 (pid 52846)] please provide us feedback at https://github.com/pandas-dev/pandas/issues/54466
2024-08-06 13:13:53.358 [1892/slide_conversion/24173 (pid 52846)]
2024-08-06 13:13:53.358 [1892/slide_conversion/24173 (pid 52846)] import pandas as pd
2024-08-06 13:13:53.358 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:13:54.653 [1892/slide_conversion/24173 (pid 52846)] Metaflow 2.12.10+netflix-ext(1.2.1) executing SlideToTIFF for user:mikebentleymills
2024-08-06 13:13:54.695 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:13:54.715 [1892/slide_conversion/24173 (pid 52846)] Validating your flow...
2024-08-06 13:13:54.716 [1892/slide_conversion/24173 (pid 52846)] The graph looks good!
2024-08-06 13:13:54.716 [1892/slide_conversion/24173 (pid 52846)] Running pylint...
2024-08-06 13:13:54.716 [1892/slide_conversion/24173 (pid 52846)] Pylint not found, so extra checks are disabled.
2024-08-06 13:13:54.716 [1892/slide_conversion/24173 (pid 52846)] Bootstrapping Conda environment... (this could take a few minutes)
2024-08-06 13:13:54.717 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step start, got flow req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]; step req: StepReq[]
2024-08-06 13:13:54.717 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step start, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]
2024-08-06 13:13:54.718 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/mikebentleymills/miniforge3/envs/metaflow_d0a3e13ca27571df018ab89ed0b6dd649ffca3a8_4f335769efbc1faf587e200c9eedb0926730dd43/bin/pip', '--version']
2024-08-06 13:13:54.851 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:275]: Conda call: ['/Users/mikebentleymills/miniforge3/condabin/mamba', 'info', '--json']
2024-08-06 13:13:55.200 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step start, final req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'python': '3.11.7', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'conda-forge::s3fs': ''}, 'pypi': {}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults']}]
2024-08-06 13:13:55.201 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64') found locally
2024-08-06 13:13:55.201 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:204]: Added environment to resolve {'id': EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64'), 'steps': ['start'], 'user_deps': {'conda': ['python==3.11.7', 'conda-forge::pydantic==2.6.1', 'conda-forge::numpy==1.26.3', 'requests==>=2.21.0', 'boto3==>=1.14.0', 'conda-forge::s3fs'], 'pypi': [], 'sys': []}, 'deps': {'conda': ['python==3.11.7', 'conda-forge::pydantic==2.6.1', 'conda-forge::numpy==1.26.3', 'requests==>=2.21.0', 'boto3==>=1.14.0', 'conda-forge::s3fs'], 'pypi': [], 'sys': []}, 'sources': {'conda': ['conda-forge', 'nodefaults']}, 'extras': {}, 'conda_format': ['_any'], 'base': None, 'base_accurate': None, 'resolved': <metaflow_extensions.netflix_ext.plugins.conda.env_descr.ResolvedEnvironment object at 0x125bd8290>, 'already_resolved': True, 'env_type': <EnvType.CONDA_ONLY: 'conda-only'>, 'force': False}
2024-08-06 13:13:55.201 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step isyntax_formats, got flow req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]; step req: StepReq[]
2024-08-06 13:13:55.202 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step isyntax_formats, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]
2024-08-06 13:13:55.202 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step isyntax_formats, final req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'python': '3.11.7', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'conda-forge::s3fs': ''}, 'pypi': {}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults']}]
2024-08-06 13:13:55.203 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64') found locally
2024-08-06 13:13:55.203 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64')' is also needed by 'isyntax_formats'
2024-08-06 13:13:55.204 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step general_formats, got flow req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]; step req: StepReq[]
2024-08-06 13:13:55.204 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step general_formats, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]
2024-08-06 13:13:55.204 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step general_formats, final req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'python': '3.11.7', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'conda-forge::s3fs': ''}, 'pypi': {}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults']}]
2024-08-06 13:13:55.205 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64') found locally
2024-08-06 13:13:55.205 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64')' is also needed by 'general_formats'
2024-08-06 13:13:55.206 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step process_general, got flow req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]; step req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3path': '0.5.2', 'conda-forge::pydantic': '2.6.1', 'conda-forge::tiffslide': '2.3.1', 'fastai::opencv-python-headless': '4.9.0.80', 'conda-forge::numpy': '1.26.3', 'conda-forge::pyvips': '2.2.1'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]
2024-08-06 13:13:55.206 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step process_general, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3', 'conda-forge::s3path': '0.5.2', 'conda-forge::tiffslide': '2.3.1', 'fastai::opencv-python-headless': '4.9.0.80', 'conda-forge::pyvips': '2.2.1'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]
2024-08-06 13:13:55.207 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step process_general, final req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3path': '0.5.2', 'boto3': '>=1.14.0', 'conda-forge::tiffslide': '2.3.1', 'conda-forge::pydantic': '2.6.1', 'python': '3.11.7', 'fastai::opencv-python-headless': '4.9.0.80', 'conda-forge::numpy': '1.26.3', 'requests': '>=2.21.0', 'conda-forge::pyvips': '2.2.1', 'conda-forge::s3fs': ''}, 'pypi': {}, 'sys': {'__glibc': '2.27'}}; sources={'conda': ['conda-forge', 'nodefaults']}]
2024-08-06 13:13:55.207 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='2185cdd1ae7f63a786d367cfb1cff0fe487dd97b', full_id='_default', arch='linux-64') found locally
2024-08-06 13:13:55.207 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:204]: Added environment to resolve {'id': EnvID(req_id='2185cdd1ae7f63a786d367cfb1cff0fe487dd97b', full_id='_default', arch='linux-64'), 'steps': ['process_general'], 'user_deps': {'conda': ['conda-forge::s3path==0.5.2', 'boto3==>=1.14.0', 'conda-forge::tiffslide==2.3.1', 'conda-forge::pydantic==2.6.1', 'python==3.11.7', 'fastai::opencv-python-headless==4.9.0.80', 'conda-forge::numpy==1.26.3', 'requests==>=2.21.0', 'conda-forge::pyvips==2.2.1', 'conda-forge::s3fs'], 'pypi': [], 'sys': ['__glibc==2.27']}, 'deps': {'conda': ['conda-forge::s3path==0.5.2', 'boto3==>=1.14.0', 'conda-forge::tiffslide==2.3.1', 'conda-forge::pydantic==2.6.1', 'python==3.11.7', 'fastai::opencv-python-headless==4.9.0.80', 'conda-forge::numpy==1.26.3', 'requests==>=2.21.0', 'conda-forge::pyvips==2.2.1', 'conda-forge::s3fs'], 'pypi': [], 'sys': ['__glibc==2.27']}, 'sources': {'conda': ['conda-forge', 'nodefaults']}, 'extras': {}, 'conda_format': ['_any'], 'base': None, 'base_accurate': None, 'resolved': <metaflow_extensions.netflix_ext.plugins.conda.env_descr.ResolvedEnvironment object at 0x125c37150>, 'already_resolved': True, 'env_type': <EnvType.CONDA_ONLY: 'conda-only'>, 'force': False}
2024-08-06 13:13:55.208 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step general_formats_join, got flow req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]; step req: StepReq[]
2024-08-06 13:13:55.208 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step general_formats_join, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]
2024-08-06 13:13:55.209 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step general_formats_join, final req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'python': '3.11.7', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'conda-forge::s3fs': ''}, 'pypi': {}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults']}]
2024-08-06 13:13:55.209 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64') found locally
2024-08-06 13:13:55.210 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64')' is also needed by 'general_formats_join'
2024-08-06 13:13:55.210 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step process_isyntax, got flow req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]; step req: StepReq[disabled=True]
2024-08-06 13:13:55.210 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step process_isyntax, merged requirement: StepReq[disabled=True]
2024-08-06 13:13:55.211 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step isyntax_ome_processing, got flow req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]; step req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3path': '0.5.2', 'conda-forge::pydantic': '2.6.1', 'conda-forge::tiffslide': '2.3.1', 'fastai::opencv-python-headless': '4.9.0.80', 'conda-forge::numpy': '1.26.3', 'conda-forge::pyvips': '2.2.1'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]
2024-08-06 13:13:55.211 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step isyntax_ome_processing, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3', 'conda-forge::s3path': '0.5.2', 'conda-forge::tiffslide': '2.3.1', 'fastai::opencv-python-headless': '4.9.0.80', 'conda-forge::pyvips': '2.2.1'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]
2024-08-06 13:13:55.212 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step isyntax_ome_processing, final req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3path': '0.5.2', 'boto3': '>=1.14.0', 'conda-forge::tiffslide': '2.3.1', 'conda-forge::pydantic': '2.6.1', 'python': '3.11.7', 'fastai::opencv-python-headless': '4.9.0.80', 'conda-forge::numpy': '1.26.3', 'requests': '>=2.21.0', 'conda-forge::pyvips': '2.2.1', 'conda-forge::s3fs': ''}, 'pypi': {}, 'sys': {'__glibc': '2.27'}}; sources={'conda': ['conda-forge', 'nodefaults']}]
2024-08-06 13:13:55.212 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='2185cdd1ae7f63a786d367cfb1cff0fe487dd97b', full_id='_default', arch='linux-64') found locally
2024-08-06 13:13:55.213 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='2185cdd1ae7f63a786d367cfb1cff0fe487dd97b', full_id='_default', arch='linux-64')' is also needed by 'isyntax_ome_processing'
2024-08-06 13:13:55.213 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step process_isyntax_join, got flow req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]; step req: StepReq[]
2024-08-06 13:13:55.213 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step process_isyntax_join, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]
2024-08-06 13:13:55.214 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step process_isyntax_join, final req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'python': '3.11.7', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'conda-forge::s3fs': ''}, 'pypi': {}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults']}]
2024-08-06 13:13:55.214 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64') found locally
2024-08-06 13:13:55.215 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64')' is also needed by 'process_isyntax_join'
2024-08-06 13:13:55.215 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step process_join, got flow req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]; step req: StepReq[]
2024-08-06 13:13:55.215 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step process_join, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]
2024-08-06 13:13:55.216 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step process_join, final req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'python': '3.11.7', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'conda-forge::s3fs': ''}, 'pypi': {}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults']}]
2024-08-06 13:13:55.216 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64') found locally
2024-08-06 13:13:55.217 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64')' is also needed by 'process_join'
2024-08-06 13:13:55.217 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step end, got flow req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]; step req: StepReq[]
2024-08-06 13:13:55.218 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step end, merged requirement: StepReq[disabled=False; python=3.11.7; packages={'conda': {'conda-forge::s3fs': '', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3'}, 'pypi': {}}; sources={'conda': [], 'pypi': []}]
2024-08-06 13:13:55.218 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step end, final req: StepReq[disabled=False; python=3.11.7; packages={'conda': {'python': '3.11.7', 'conda-forge::pydantic': '2.6.1', 'conda-forge::numpy': '1.26.3', 'requests': '>=2.21.0', 'boto3': '>=1.14.0', 'conda-forge::s3fs': ''}, 'pypi': {}, 'sys': {}}; sources={'conda': ['conda-forge', 'nodefaults']}]
2024-08-06 13:13:55.218 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64') found locally
2024-08-06 13:13:55.219 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64')' is also needed by 'end'
2024-08-06 13:13:55.219 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:229]: Resolving environments:
2024-08-06 13:13:55.584 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:13:55.584     All packages already cached in s3.
2024-08-06 13:13:56.085 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:13:55.584     All environments already cached in s3.
2024-08-06 13:13:56.085 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:13:56.084 Workflow starting (run-id 1893):
2024-08-06 13:13:58.407 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64') found locally
2024-08-06 13:13:58.409 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2031]: Locating MF environment metaflow_bac099b8ef09ba99f32812140d85680ca66ecc56_a2271166301c8d4332566033673762770b78f643
2024-08-06 13:13:58.415 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2022]: Found EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='a2271166301c8d4332566033673762770b78f643', arch='osx-arm64') at dir_name /Users/mikebentleymills/miniforge3/envs/metaflow_bac099b8ef09ba99f32812140d85680ca66ecc56_a2271166301c8d4332566033673762770b78f643
2024-08-06 13:13:58.416 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:13:58.415 Using existing Conda environment bac099b8ef09ba99f32812140d85680ca66ecc56 (a2271166301c8d4332566033673762770b78f643)
2024-08-06 13:13:58.428 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:13:58.428 [1893/start/24175 (pid 52981)] Task is starting.
2024-08-06 13:13:59.132 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:13:59.132 [1893/start/24175 (pid 52981)] debug[conda /tmp/tmp58k3od4w/tmpwk9cc28d/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:13:59.884 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:13:59.884 [1893/start/24175 (pid 52981)] Ignoring previous processed slides.
2024-08-06 13:13:59.923 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:13:59.923 [1893/start/24175 (pid 52981)] This is going to reprocess everything! in ['s3://valar-portal-prod/0001/drafts/D-0001-00000067/raw/D-0001-00000067_1b945591-1e2d-4fe2-aab1-d1889af92955_S24-13531_A1-1_H&E_Slide(s)_20240718_093020.isyntax'] ...
2024-08-06 13:13:59.946 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:13:59.945 [1893/start/24175 (pid 52981)] Found 1 files to process.
2024-08-06 13:14:01.349 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:01.349 [1893/start/24175 (pid 52981)] Found 0 general files.
2024-08-06 13:14:01.598 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:01.349 [1893/start/24175 (pid 52981)] Found 1 isyntax files.
2024-08-06 13:14:01.599 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:01.598 [1893/start/24175 (pid 52981)] Task finished successfully.
2024-08-06 13:14:02.257 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64') found locally
2024-08-06 13:14:02.260 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2031]: Locating MF environment metaflow_bac099b8ef09ba99f32812140d85680ca66ecc56_a2271166301c8d4332566033673762770b78f643
2024-08-06 13:14:02.265 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2022]: Found EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='a2271166301c8d4332566033673762770b78f643', arch='osx-arm64') at dir_name /Users/mikebentleymills/miniforge3/envs/metaflow_bac099b8ef09ba99f32812140d85680ca66ecc56_a2271166301c8d4332566033673762770b78f643
2024-08-06 13:14:02.266 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:02.265 Using existing Conda environment bac099b8ef09ba99f32812140d85680ca66ecc56 (a2271166301c8d4332566033673762770b78f643)
2024-08-06 13:14:02.275 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:02.275 [1893/isyntax_formats/24176 (pid 53072)] Task is starting.
2024-08-06 13:14:02.547 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='_default', arch='osx-arm64') found locally
2024-08-06 13:14:02.548 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2031]: Locating MF environment metaflow_bac099b8ef09ba99f32812140d85680ca66ecc56_a2271166301c8d4332566033673762770b78f643
2024-08-06 13:14:02.549 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2022]: Found EnvID(req_id='bac099b8ef09ba99f32812140d85680ca66ecc56', full_id='a2271166301c8d4332566033673762770b78f643', arch='osx-arm64') at dir_name /Users/mikebentleymills/miniforge3/envs/metaflow_bac099b8ef09ba99f32812140d85680ca66ecc56_a2271166301c8d4332566033673762770b78f643
2024-08-06 13:14:02.549 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:02.549 Using existing Conda environment bac099b8ef09ba99f32812140d85680ca66ecc56 (a2271166301c8d4332566033673762770b78f643)
2024-08-06 13:14:02.552 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:02.552 [1893/general_formats/24177 (pid 53073)] Task is starting.
2024-08-06 13:14:02.621 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:02.621 [1893/isyntax_formats/24176 (pid 53072)] debug[conda /tmp/tmpm7s40tg_/tmp3sd5kz9m/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:14:02.889 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:02.889 [1893/general_formats/24177 (pid 53073)] debug[conda /tmp/tmpa3ls4pds/tmpoill0rd5/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:14:04.280 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:04.280 [1893/isyntax_formats/24176 (pid 53072)] Foreach yields 2 child steps.
2024-08-06 13:14:05.013 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:04.280 [1893/isyntax_formats/24176 (pid 53072)] Task finished successfully.
2024-08-06 13:14:05.013 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:05.012 [1893/process_isyntax/24178 (pid 53134)] Task is starting.
2024-08-06 13:14:05.261 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:05.257 [1893/process_isyntax/24179 (pid 53135)] Task is starting.
2024-08-06 13:14:05.527 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:05.527 [1893/general_formats/24177 (pid 53073)] Foreach yields 1 child steps.
2024-08-06 13:14:06.035 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:05.527 [1893/general_formats/24177 (pid 53073)] Task finished successfully.
2024-08-06 13:14:06.035 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='2185cdd1ae7f63a786d367cfb1cff0fe487dd97b', full_id='_default', arch='linux-64') found locally
2024-08-06 13:14:06.038 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:06.038 [1893/process_general/24180 (pid 53172)] Task is starting.
2024-08-06 13:14:06.039 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:06.038 [1893/process_isyntax/24178 (pid 53134)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:14:06.039 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:06.039 [1893/process_isyntax/24179 (pid 53135)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:14:06.574 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:06.574 [1893/process_general/24180 (pid 53172)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:14:06.614 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:06.614 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm] Task is starting (Pod is pending)...
2024-08-06 13:14:06.834 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:06.834 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v] Task is starting (Pod is pending)...
2024-08-06 13:14:07.366 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:14:07.366 [1893/process_general/24180 (pid 53172)] [pod t-4c757fd9-rbsvx-bnz9x] Task is starting (Pod is pending)...
2024-08-06 13:16:09.430 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:15:52.173 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm] Setting up task environment.
2024-08-06 13:16:09.430 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:03.449 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm] Downloading code package...
2024-08-06 13:16:09.430 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:04.123 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm] Code package downloaded.
2024-08-06 13:16:09.459 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:04.182 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm] Task is starting.
2024-08-06 13:16:09.460 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:04.586 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm] Bootstrapping environment ...
2024-08-06 13:16:09.505 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:07.596 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm]     Setting up Conda ... done in 2 seconds.
2024-08-06 13:16:09.505 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:04.685 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:16:09.505 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.505 [1893/process_isyntax/24178 (pid 53134)] Kubernetes error:
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.505 [1893/process_isyntax/24178 (pid 53134)] Error: Setting up task environment.
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.617 [1893/process_isyntax/24178 (pid 53134)] Downloading code package...
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] Code package downloaded.
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] Task is starting.
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] Bootstrapping environment ...
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/conda.py:275]: Conda call: ['/root/.local/bin/micromamba', 'info', '--json']
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] Setting up Conda ... done in 2 seconds.
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='4f335769efbc1faf587e200c9eedb0926730dd43', arch='linux-64') not found locally
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2104]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='4f335769efbc1faf587e200c9eedb0926730dd43', arch='linux-64') not found remotely at envs/linux-64/d0a3e13ca27571df018ab89ed0b6dd649ffca3a8/4f335769efbc1faf587e200c9eedb0926730dd43/env
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] Traceback (most recent call last):
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] return _run_code(code, main_globals, None,
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] exec(code, run_globals)
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/remote_bootstrap.py", line 97, in <module>
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] bootstrap_environment(sys.argv[1:])
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/remote_bootstrap.py", line 55, in bootstrap_environment
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] raise RuntimeError(
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] RuntimeError: Cannot find cached environment for hash d0a3e13ca27571df018ab89ed0b6dd649ffca3a8:4f335769efbc1faf587e200c9eedb0926730dd43
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)] (exit code 1). This could be a transient error. Use @retry to retry.
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.618 [1893/process_isyntax/24178 (pid 53134)]
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:07.107 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/conda.py:275]: Conda call: ['/root/.local/bin/micromamba', 'info', '--json']
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:07.597 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='4f335769efbc1faf587e200c9eedb0926730dd43', arch='linux-64') not found locally
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:07.771 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2104]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='4f335769efbc1faf587e200c9eedb0926730dd43', arch='linux-64') not found remotely at envs/linux-64/d0a3e13ca27571df018ab89ed0b6dd649ffca3a8/4f335769efbc1faf587e200c9eedb0926730dd43/env
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:07.771 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm] Traceback (most recent call last):
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:07.771 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm]   File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:07.771 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm]     return _run_code(code, main_globals, None,
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:07.771 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm]   File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:07.771 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm]     exec(code, run_globals)
2024-08-06 13:16:09.618 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:07.771 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm]   File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/remote_bootstrap.py", line 97, in <module>
2024-08-06 13:16:09.619 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:07.771 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm]     bootstrap_environment(*sys.argv[1:])
2024-08-06 13:16:09.619 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:07.771 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm]   File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/remote_bootstrap.py", line 55, in bootstrap_environment
2024-08-06 13:16:09.619 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:07.771 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm]     raise RuntimeError(
2024-08-06 13:16:09.619 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:07.771 [1893/process_isyntax/24178 (pid 53134)] [pod t-26e9b69d-s7ggf-q57bm] RuntimeError: Cannot find cached environment for hash d0a3e13ca27571df018ab89ed0b6dd649ffca3a8:4f335769efbc1faf587e200c9eedb0926730dd43
2024-08-06 13:16:09.697 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.697 [1893/process_isyntax/24178 (pid 53134)] Task failed.
2024-08-06 13:16:09.842 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.841 [1893/process_isyntax/24178 (pid 55674)] Task fallback is starting to handle the failure.
2024-08-06 13:16:10.333 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:10.333 [1893/process_isyntax/24178 (pid 55674)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:16:11.209 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:15:52.174 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v] Setting up task environment.
2024-08-06 13:16:11.209 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:03.639 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v] Downloading code package...
2024-08-06 13:16:11.237 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:04.334 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v] Code package downloaded.
2024-08-06 13:16:11.237 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:04.394 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v] Task is starting.
2024-08-06 13:16:11.275 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:04.804 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v] Bootstrapping environment ...
2024-08-06 13:16:11.275 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.776 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v]     Setting up Conda ... done in 4 seconds.
2024-08-06 13:16:11.275 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:04.902 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:16:11.275 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.275 [1893/process_isyntax/24179 (pid 53135)] Kubernetes error:
2024-08-06 13:16:11.339 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.275 [1893/process_isyntax/24179 (pid 53135)] Error: Setting up task environment.
2024-08-06 13:16:11.339 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.339 [1893/process_isyntax/24178 (pid 55674)] @catch caught an exception from <flow SlideToTIFF step process_isyntax[0] (input: s3://valar-portal-pr...)>
2024-08-06 13:16:11.339 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.339 [1893/process_isyntax/24178 (pid 55674)] >  Traceback (most recent call last):
2024-08-06 13:16:11.339 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.339 [1893/process_isyntax/24178 (pid 55674)] >    File "/tmp/tmpbbl8towv/metaflow/task.py", line 650, in run_step
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.339 [1893/process_isyntax/24178 (pid 55674)] >      self._exec_step_function(step_func)
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] Downloading code package...
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] Code package downloaded.
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] Task is starting.
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] Bootstrapping environment ...
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/conda.py:275]: Conda call: ['/root/.local/bin/micromamba', 'info', '--json']
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] Setting up Conda ... done in 4 seconds.
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='4f335769efbc1faf587e200c9eedb0926730dd43', arch='linux-64') not found locally
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2104]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='4f335769efbc1faf587e200c9eedb0926730dd43', arch='linux-64') not found remotely at envs/linux-64/d0a3e13ca27571df018ab89ed0b6dd649ffca3a8/4f335769efbc1faf587e200c9eedb0926730dd43/env
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] Traceback (most recent call last):
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] return _run_code(code, main_globals, None,
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] exec(code, run_globals)
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/remote_bootstrap.py", line 97, in <module>
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] bootstrap_environment(sys.argv[1:])
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/remote_bootstrap.py", line 55, in bootstrap_environment
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] raise RuntimeError(
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] RuntimeError: Cannot find cached environment for hash d0a3e13ca27571df018ab89ed0b6dd649ffca3a8:4f335769efbc1faf587e200c9eedb0926730dd43
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)] (exit code 1). This could be a transient error. Use @retry to retry.
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.378 [1893/process_isyntax/24179 (pid 53135)]
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.282 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/conda.py:275]: Conda call: ['/root/.local/bin/micromamba', 'info', '--json']
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.777 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='4f335769efbc1faf587e200c9eedb0926730dd43', arch='linux-64') not found locally
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.950 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v] debug[conda /metaflow/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2104]: EnvID(req_id='d0a3e13ca27571df018ab89ed0b6dd649ffca3a8', full_id='4f335769efbc1faf587e200c9eedb0926730dd43', arch='linux-64') not found remotely at envs/linux-64/d0a3e13ca27571df018ab89ed0b6dd649ffca3a8/4f335769efbc1faf587e200c9eedb0926730dd43/env
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.950 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v] Traceback (most recent call last):
2024-08-06 13:16:11.378 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.950 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v]   File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2024-08-06 13:16:11.379 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.950 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v]     return _run_code(code, main_globals, None,
2024-08-06 13:16:11.379 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.950 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v]   File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
2024-08-06 13:16:11.379 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.950 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v]     exec(code, run_globals)
2024-08-06 13:16:11.379 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.951 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v]   File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/remote_bootstrap.py", line 97, in <module>
2024-08-06 13:16:11.379 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.951 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v]     bootstrap_environment(*sys.argv[1:])
2024-08-06 13:16:11.379 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.951 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v]   File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/remote_bootstrap.py", line 55, in bootstrap_environment
2024-08-06 13:16:11.427 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.951 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v]     raise RuntimeError(
2024-08-06 13:16:11.427 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:09.951 [1893/process_isyntax/24179 (pid 53135)] [pod t-95eea0b3-q7ghs-6q44v] RuntimeError: Cannot find cached environment for hash d0a3e13ca27571df018ab89ed0b6dd649ffca3a8:4f335769efbc1faf587e200c9eedb0926730dd43
2024-08-06 13:16:11.427 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.427 [1893/process_isyntax/24179 (pid 53135)] Task failed.
2024-08-06 13:16:11.598 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:11.598 [1893/process_isyntax/24179 (pid 55720)] Task fallback is starting to handle the failure.
2024-08-06 13:16:12.107 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:12.107 [1893/process_isyntax/24179 (pid 55720)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:16:12.972 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:12.972 [1893/process_isyntax/24179 (pid 55720)] @catch caught an exception from <flow SlideToTIFF step process_isyntax[1]>
2024-08-06 13:16:12.972 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:12.972 [1893/process_isyntax/24179 (pid 55720)] >  Traceback (most recent call last):
2024-08-06 13:16:12.972 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:12.972 [1893/process_isyntax/24179 (pid 55720)] >    File "/tmp/tmpbbl8towv/metaflow/task.py", line 650, in run_step
2024-08-06 13:16:12.972 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:12.972 [1893/process_isyntax/24179 (pid 55720)] >      self._exec_step_function(step_func)
2024-08-06 13:16:12.972 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:12.972 [1893/process_isyntax/24179 (pid 55720)] >    File "/tmp/tmpbbl8towv/metaflow/task.py", line 62, in _exec_step_function
2024-08-06 13:16:13.756 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:13.755 [1893/process_isyntax/24178 (pid 55674)] >    File "/tmp/tmpbbl8towv/metaflow/task.py", line 62, in _exec_step_function
2024-08-06 13:16:13.756 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:13.755 [1893/process_isyntax/24178 (pid 55674)] >      step_function()
2024-08-06 13:16:13.756 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:13.755 [1893/process_isyntax/24178 (pid 55674)] >    File "/tmp/tmpbbl8towv/metaflow/plugins/catch_decorator.py", line 107, in fallback_step
2024-08-06 13:16:13.964 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:13.756 [1893/process_isyntax/24178 (pid 55674)] >      raise FailureHandledByCatch(retry_count)
2024-08-06 13:16:13.964 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:13.756 [1893/process_isyntax/24178 (pid 55674)] >  metaflow.plugins.catch_decorator.FailureHandledByCatch: Task execution kept failing over 1 attempts. Your code did not raise an exception. Something in the execution environment caused the failure.
2024-08-06 13:16:13.964 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:13.963 [1893/process_isyntax/24178 (pid 55674)] Task finished successfully.
2024-08-06 13:16:14.530 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='2185cdd1ae7f63a786d367cfb1cff0fe487dd97b', full_id='_default', arch='linux-64') found locally
2024-08-06 13:16:14.542 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:14.541 [1893/isyntax_ome_processing/24181 (pid 55814)] Task is starting.
2024-08-06 13:16:14.542 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:14.542 [1893/process_isyntax/24179 (pid 55720)] >      step_function()
2024-08-06 13:16:14.542 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:14.542 [1893/process_isyntax/24179 (pid 55720)] >    File "/tmp/tmpbbl8towv/metaflow/plugins/catch_decorator.py", line 107, in fallback_step
2024-08-06 13:16:14.542 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:14.542 [1893/process_isyntax/24179 (pid 55720)] >      raise FailureHandledByCatch(retry_count)
2024-08-06 13:16:14.543 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:14.542 [1893/process_isyntax/24179 (pid 55720)] >  metaflow.plugins.catch_decorator.FailureHandledByCatch: Task execution kept failing over 1 attempts. Your code did not raise an exception. Something in the execution environment caused the failure.
2024-08-06 13:16:14.750 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:14.747 [1893/process_isyntax/24179 (pid 55720)] Task finished successfully.
2024-08-06 13:16:15.226 [1892/slide_conversion/24173 (pid 52846)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='2185cdd1ae7f63a786d367cfb1cff0fe487dd97b', full_id='_default', arch='linux-64') found locally
2024-08-06 13:16:15.229 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:15.229 [1893/isyntax_ome_processing/24182 (pid 55818)] Task is starting.
2024-08-06 13:16:15.230 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:15.230 [1893/isyntax_ome_processing/24181 (pid 55814)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:16:15.729 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:15.729 [1893/isyntax_ome_processing/24182 (pid 55818)] debug[conda /tmp/tmpbbl8towv/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at s3://metaflows3qjr8flmt/metaflow/conda_env
2024-08-06 13:16:15.989 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:15.989 [1893/isyntax_ome_processing/24181 (pid 55814)] [pod t-a09f42f8-rwh9h-cvcsh] Task is starting (Pod is pending, Container is waiting - ContainerCreating)...
2024-08-06 13:16:16.509 [1892/slide_conversion/24173 (pid 52846)] 2024-08-06 13:16:16.509 [1893/isyntax_ome_processing/24182 (pid 55818)] [pod t-c0a415d6-f42v8-ghpfd] Task is starting (Pod is pending, Container is waiting - ContainerCreating)...