Ways to Implement ML Containers
The ML containers were developed to be used both at the command line and with each of the AF's jupyter interfaces.
Command Line ML Container Usage
The images are deployed onto the Docker hub and CVMFS and there are 4 ways to run the ML images at the command line.
Apptainer/Singularity
You can directly use Apptainer or Singularity to develop with the containers using the
apptainer run
or singularity run
command.
``` output
lxplus901% apptainer run /cvmfs/unpacked.cern.ch/registry.hub.docker.com/yesw2000/ml-base:alma9-python39
For the content in this container,
please read the file /list-of-pkgs-inside.txt
To create your own new env, run "source /create-newEnv-on-base.sh -h" for help
Singularity> python --version
Python 3.9.19
Singularity>
```
Virtual Environment
We have included a bash script with every container that will create a virtual environment based on that container.
Docker or podman
Since the containers are also deployed to Docker hub, you can use docker
or podman
to develop with them.
Script for laptops
A script is available to run the ML container on laptops.
lxplus901% wget https://raw.githubusercontent.com/usatlas/ML-Containers/main/run-ml_container.sh
lxplus901% source run-ml_container.sh ml-base:alma9-python39
Found the image name= ml-base:alma9-python39 with the dockerPath= docker.io/yesw2000/ml-base:alma9-python39
Trying to pull docker.io/yesw2000/ml-base:alma9-python39...
[...]
To reuse the same container next time, just run
source runML-here.sh
or
source run-ml_container.sh
podman exec -it yesw_ml-base_alma9-python39 /bin/bash
root@3c90a02d92d6:[1]%
Jupyter Interface at AFs
Brookhaven
Use the following steps to implement the ML container through the BNL jupyter interface:
- Open the BNL Jupyter Interface and create a new notebook
- For Select JupyterLab Environment select Default
- Select Custom in the Singularity Container question and input the
/cvmfs
address for ML container - Press Start!
After pressing start, the server for the notebook will begin the setup process. After a minute or so, your jupyter notebook will appear and you will have a number of kernels available to use. The kernel associated with the container is called ML-Python3
.
UChicago
Currently the UChicago JupyterLab servers don't allow custom containers when creating a new notebook. This is planning on being updated soon. In the meantime, the ml-platform:latest
container shown in the Image
dropdown box does have all packages necessary for running the ML Tutorial material.
SLAC
Steps for using ML containers using SLAC Jupyter interface:
- Connect to SLAC's S3DF Jupyter Portal
- After you login, you'll come to the menu where you can define your jupyter server. The first dropdown is the Jupyter Instance. Choose the "Custom Singularity Container" option.
- Now the Commands to initiate Jupyter box will be editable. Replace the content of the box by the following:
- You may need to check the Use Jupyter Lab instead of Jupyter Notebook box (in case only Jupyter Lab or only Jupyter Notebook is available in the container)
- Make sure the two pull down menu under Run on cluster type choose
Batch
ands3df
. - In Account box, type in
atlas:usatlas
- In Partition choose
ampere
if you want GPUs, or chooserome
if you don't need a GPU. - Fill the Number of GPUs box if you are asking for GPUs.
- Your server will be submitted to the queue and will take a minute or so to generate. You will see the Connect to Jupyter button when your server has initialized successfully.