We have made available in Github a series of templates for automating a number of aspects of provisioning and configuring your instances to tailor them to your needs.

Ansible templates

Ansible is a very popular automation tool that can be used to configure and customise your instances and other operations. We have published a number of Ansible Playbooks and Roles to carry out specific operations and customisations on the EWC that you can mix and match.

Ansible playbook repositories

• ewc-flavours: A set of flavours and instance types such as the ECMWF Data flavour or the local Jupyterhub. Although these are integrated in the Morpheus Portal, these playbooks allow you to apply them yourself to standard instances either manually or through your IaC pipelines.
• ewc-ecmwf-ai-stacks: A collection of playbooks to for setting up ML/AI stacks. See EWC ECMWF AI Stacks for more details on how to use them.

Generic usage instructions

Each of those repositories will have specific instructions on how to use them, but here are the fundamental steps to get started if you want to use them directly on the command line or CI/CD pipeline:

• Install ansible and other dependencies. You may want to do it in its own virtual environment (pip install -r requirements.txt)

• Fetch the external requirements
  

  $ ansible-galaxy role install -r requirements.yml roles/

• If you don't have it already, define your Ansible inventory. The simplest approach would be to create a file called inventory, on the same directory where the plabooks are, containing the fully qualified domain name (FQDN) or IP address used to connect to the instance from your seed platform. If running this ansible playbook on the same instance, you may use localhost.

• Run the apropriate playbook:

  $ ansible-playbook -i inventory playbookname.yml

Ansible role repositories

These are the basic building blocks upon which the provided playbooks are built, so you can create your own Ansible playbooks combining them as you see fit. All these repositories are clearly named with the prefix "ewc-ansible-role-":

• ewc-ansible-role-update-system: Make sure your instance is up to date.

• ewc-ansible-role-mars-client: Install the MARS client.

• ewc-ansible-role-conda: Install conda package manager to create your own enviroments.

• ewc-ansible-role-ecmwf-toolbox: Create a conda environment with an ECMWF software stack including ecCodes, Metview, Aviso and earthkit.

• ewc-ansible-role-jupyterhub-local: Install and run a Jupyterhub on your instance to offer a convenient web portal to your instance.

• ewc-ansible-role-ml-basic: Create a conda environment with the basic AI/ML tools in python such as torch, tensorflow, keras, scikit-learn, and others.

• ewc-ansible-role-anemoi: install a conda environment featuring all the Anemoi components. It includes the basic packages such as datasets , training , graphs , models or inference .

• ewc-ansible-role-ai-models: Set up a conda environment with the AI-models package, which allows you to run popular da ta-driven weather forecasting models such as panguweather or graphcast.

• ewc-ansible-role-aifs-single-mse: installs the ECMWF AIFS Data-Driven Forecasting system and supporting dependencies, so you can run it yourself.

Advanced: How to use your custom Ansible playbook using provided Ansible roles

For advanced users with existing Ansible playbooks and greater customisation needs, it is possible to just pick and choose roles you are interested and include them into your own playbooks, instead of running separate playbooks. You can find the URLs for each of the dependant roles in the requirements.yml file. 

1. Open your playbook file and include the roles that you are interested in. For example, here is a playbook that includes 3 roles and customises some variable:
   
   myplaybook.yml

   ---
   - hosts: all
     become: yes
     vars:
       conda_prefix: /opt/conda
     tasks:
       - name: Mars client
         ansible.builtin.include_role:
           name: ewc-ansible-role-mars-client
    
       - name: ML basic stack
         ansible.builtin.include_role:
           name: ewc-ansible-role-ml-basic
    
       - name: Anemoi
         ansible.builtin.include_role:
           name: ewc-ansible-role-ecmwf-anemoi
   
   

   
   See the official Ansible documentation for more information on roles and how to include them. 
2. Make sure the necessary roles are available before you run the playbook. For that you may install them with ansible-galaxy either individually or writing your own requirements.yml. For the example above, this could be a requirements.yaml file:
   
   requirements.yml

   ---
   # Ansible Requirements
   roles:
     - name: ewc-ansible-role-mars-client
       src: https://github.com/ewcloud/ewc-ansible-role-mars-client.git
       version: main
   
     - name: ewc-ansible-role-conda
       src: https://github.com/ewcloud/ewc-ansible-role-conda.git
       version: main
   
     - name: ewc-ansible-role-ml-basic
       src: https://github.com/ewcloud/ewc-ansible-role-ml-basic.git
       version: main
   
     - name: ewc-ansible-role-anemoi
       src: https://github.com/ewcloud/ewc-ansible-role-anemoi.git
       version: main

   
   See the official Ansible documentation for more details on how to create such file.
3. Install the roles with ansible-galaxy:
   

   ansible-galaxy role install -r requirements.yml roles/

4. Run the playbook

   $ ansible-playbook -i inventory myplaybook.yml