We have made available in Github a series of templates for automating a number of aspects of provisioning and configuring your instances to tailor them to your needs.
Ansible templates
Ansible is a very popular automation tool that can be used to configure and customise your instances and other operations. We have published a number of Ansible Playbooks and Roles to carry out specific operations and customisations on the EWC that you can mix and match.
Ansible playbook repositories
- ewc-flavours: A set of flavours and instance types such as the ECMWF Data flavour or the local Jupyterhub. Although these are integrated in the Morpheus Portal, these playbooks allow you to apply them yourself to standard instances either manually or through your IaC pipelines.
- ewc-ecmwf-ai-stacks: A collection of playbooks to for setting up ML/AI stacks. See EWC ECMWF AI Stacks for more details on how to use them.
Generic usage instructions
Each of those repositories will have specific instructions on how to use them, but here are the fundamental steps to get started if you want to use them directly on the command line or CI/CD pipeline:
You may run Ansible on the same instance or anywhere else from where you can connect to your instance via SSH, such as another instance in the same private network, or from your own computer if your instance can be reached over SSH. We will refer to this as your seed platform. You will need at least git and python available to follow these steps. - On your seed platform, clone the repository and cd into the directory:
git clone <repository-url>
cd <repository> |
- If you don't have Ansible installed, you may install it with pip:
pip install --user -r requirements.txt | or if you prefer to do it in a virtual environment:
python3 -m venv ansible-venv
source ansible-venv/bin/activate
pip install -r requirements.txt |
- Install the necessary Ansible roles that are going to be used by the playbooks:
ansible-galaxy role install -r requirements.yml roles/ |
- If you don't have it already, define your Ansible inventory. The simplest approach would be to create a file called
inventory , on the same directory where the plabooks are, containing the fully qualified domain name (FQDN) or IP address used to connect to the instance from your seed platform. If running on the same instance, you may use localhost. - Apply the desired playbook with
ansible-playbook:
ansible-playbook -i inventory playbookname.yml | You may pass additional options to ansible-playbook , such as:-v for verbose output-K for asking sudo password, if your user does not have password-less sudo privileges on the target instance.-u yourremoteuser if Ansible needs to use a specific user account to connect to the target instance.-e var=value for add-hoc customisation of playbook variables to customise your installation.
|
Ansible role repositories
These are the basic building blocks upon which the provided playbooks are built, so you can create your own Ansible playbooks combining them as you see fit. All these repositories are clearly named with the prefix "ewc-ansible-role-":
ewc-ansible-role-update-system: Make sure your instance is up to date.
ewc-ansible-role-mars-client: Install the MARS client.
ewc-ansible-role-conda: Install conda package manager to create your own enviroments.
ewc-ansible-role-ecmwf-toolbox: Create a conda environment with an ECMWF software stack including ecCodes, Metview, Aviso and earthkit.
ewc-ansible-role-jupyterhub-local: Install and run a Jupyterhub on your instance to offer a convenient web portal to your instance.
ewc-ansible-role-ml-basic: Create a conda environment with the basic AI/ML tools in python such as torch, tensorflow, keras, scikit-learn, and others.
ewc-ansible-role-anemoi: install a conda environment featuring all the Anemoi components. It includes the basic packages such as datasets , training , graphs , models or inference .
ewc-ansible-role-ai-models: Set up a conda environment with the AI-models package, which allows you to run popular da ta-driven weather forecasting models such as panguweather or graphcast.
ewc-ansible-role-aifs-single-mse: installs the ECMWF AIFS Data-Driven Forecasting system and supporting dependencies, so you can run it yourself.
Advanced: How to use your custom Ansible playbook using provided Ansible roles
For advanced users with existing Ansible playbooks and greater customisation needs, it is possible to just pick and choose roles you are interested and include them into your own playbooks, instead of running separate playbooks. You can find the URLs for each of the dependant roles in the requirements.yml file. - Open your playbook file and include the roles that you are interested in. For example, here is a playbook that includes 3 roles and customises some variable:
---
- hosts: all
become: yes
vars:
conda_prefix: /opt/conda
tasks:
- name: Mars client
ansible.builtin.include_role:
name: ewc-ansible-role-mars-client
- name: ML basic stack
ansible.builtin.include_role:
name: ewc-ansible-role-ml-basic
- name: Anemoi
ansible.builtin.include_role:
name: ewc-ansible-role-ecmwf-anemoi
|
See the official Ansible documentation for more information on roles and how to include them. - Make sure the necessary roles are available before you run the playbook. For that you may install them with
ansible-galaxy either individually or writing your own requirements.yml . For the example above, this could be a requirements.yaml file:
---
# Ansible Requirements
roles:
- name: ewc-ansible-role-mars-client
src: https://github.com/ewcloud/ewc-ansible-role-mars-client.git
version: main
- name: ewc-ansible-role-conda
src: https://github.com/ewcloud/ewc-ansible-role-conda.git
version: main
- name: ewc-ansible-role-ml-basic
src: https://github.com/ewcloud/ewc-ansible-role-ml-basic.git
version: main
- name: ewc-ansible-role-anemoi
src: https://github.com/ewcloud/ewc-ansible-role-anemoi.git
version: main |
See the official Ansible documentation for more details on how to create such file. - Install the roles with
ansible-galaxy :
ansible-galaxy role install -r requirements.yml roles/ |
- Run the playbook
$ ansible-playbook -i inventory myplaybook.yml |
Check each specific role documentation in the URLs above to see all the variables you may customise when running the automation. |
|