For those doing AI/ML on the European Weather Cloud, we have published a number of templates to help you set up the necessary software stacks to run your favourite workloads. You may apply those onto an existing instance, and combine them as needed.

The stacks currently part of this collection are:

ML Basic: provides a conda environment with the basic AI/ML tools in python such as torch, tensorflow, keras, scikit-learn, and others.
AI Models: sets up a conda environment with the AI-models package, which allows you to run popular data-driven weather forecasting models such as panguweather or graphcast.
Anemoi: installs a conda environment featuring all the Anemoi components. It includes the basic packages such as datasets, training, graphs, models or inference.
AIFS Single MSE: installs the ECMWF AIFS Data-Driven Forecasting system and supporting dependencies.

These stacks are based on Ansible playbooks and can be found in https://github.com/ewcloud/ewc-ecmwf-ai-stacks.

AI/ML software stacks usually require a significant amount of disk space. Make sure your instances have enough free space before applying them. Although it is possible that less space is required based on the specific setup and combinations, we recommend at least 15 GB for each of those you may want to apply. You may extend your instance volume volume or add a new one where to install those if needed. If a new volume is added, make sure

How to apply an AI stack to an existing instance via Morpheus

If using ECMWF's Morpheus portal, you may find those are offered as Morpheus Workflows by default:

European Weather Cloud Knowledge Base > EWC ECMWF AI Stacks > ecmwf-ai-stacks_workflows.png

If you cannot see them, you may also add them to your tenancy by creating a new Git integration of the https://github.com/ewcloud/ewc-ecmwf-ai-stacks as described in the Morpheus documentation, and then creating the Automation Tasks and Workflows required using that git integration, which is described here.

Go to Provisioning - Instances, and select the desired instance to apply the stack. If you don't have one, create one first.
Click Actions - Run Workflow and type the name of the desired one. Some stacks may offer the possibility of customising certain parameters, such as the location of the conda installation. You may tune them if needed, for example if you are using an additional volume with more capacity for your software stack. You can also configure the ansible command options (for example, passing a -v for a more verbose output).
Click Execute
You may follow the progress of the workflow execution under History.

How to apply an AI stack to an existing instance via Ansible

You can also install those directly with Ansible, for example as part of your existing Infrastructure as Code (IaC) or CI/CD pipelines.

You may run Ansible on the same instance or anywhere else from where you can connect to your instance via SSH, such as another instance in the same private network, or from your own computer if your instance can be reached over SSH. We will refer to this as your seed platform. You will need at least git and python available to follow these steps.

On your seed platform, clone the https://github.com/ewcloud/ewc-ecmwf-ai-stacks:

git clone https://github.com/ewcloud/ewc-ecmwf-ai-stacks.git
cd ewc-ecmwf-ai-stacks

If you don't have Ansible installed, you may install it with pip:

pip install --user -r requirements.txt

or if you prefer to do it in a virtual environment:

python3 -m venv ansible-venv
source ansible-venv/bin/activate
pip install -r requirements.txt

Install the necessary Ansible roles that are going to be used by the playbooks:
ansible-galaxy role install -r requirements.yml roles/
If you don't have it already, define your Ansible inventory. The simplest approach would be to create a file called inventory, on the same directory where the plabooks are, containing the fully qualified domain name (FQDN) or IP address used to connect to the instance from your seed platform. If running on the same instance, you may use localhost.
Apply the desired playbook with ansible-playbook:
ansible-playbook -i inventory playbookname.yml
You may pass additional options to ansible-playbook, such as:
- -v for verbose output
- -K for asking sudo password, if your user does not have password-less sudo privileges on the target instance.
- -u yourremoteuser if Ansible needs to use a specific user account to connect to the target instance.
- -e var=value for add-hoc customisation of playbook variables to customise your installation.

Check each specific role documentation in the URLs found in requirements.yml to see all the variables you may customise when running the automation. For example, for the Anemoi role: https://github.com/ewcloud/ewc-ansible-role-anemoi

Advanced: How to use your custom Ansible playbook using provided Ansible roles