Tykky is a set of tools developed at CSC which make software installations on HPC systems easier and more efficient using Apptainer containers. Its main use cases are Python-based conda or virtual environments, and any software or application with a great number of files that when installed on a shared or parallel filesystem may perform very poorly. Tykky will encapsulate those installations in containers, and generate the necessary wrappers so that installed software can be used (almost) as if it was not containerised, improving startup times and reducing the overall load on the filesystems. 

To start using Tykky you must first load its module:

module load tykky

Containerising a conda environment

You can create a containerised conda environment from an environment file. You can either create it manually or from an existing conda environment with:

conda env export -n <target_env_name> > env.yml.

An example env.yml file will look like:

channels:
    - conda-forge
dependencies:
    - python
    - ipykernel
    - cf-units
    - eccodes
    - pip
    - pip:
      - earthkit

The example would install Python and eccodes from the conda-forge channel, as well as earthkit through pip. You may also pin specific versions if desired.

Once your environment file is ready, you can use Tikky to perform the installation for you. You must decide where to install it, but use of HPCPERM is strongly recommended. Use of HOME, PERM or SCRATCH for such installations is discouraged.

conda-containerize new --mamba --prefix <install env.yml

When the installation is complete, you will find all the executables in your environment in <install_dir>/bin. You may call them directly using the full path, or add it to your PATH to make them available as if you were activating the environment:

export PATH="<install_dir>/bin:$PATH"

Updating an existing containerised conda environment

If you wish to modify an existing installation, you must create a bash script containing the commands you want to execute to update the installation. They will be executed inside the container, with the environment activated, when you run:

conda-containerize update <install_dir> --post-install update.sh  

Here is an example of an update script, which would install metview-batch on top of the existing environment

#!/bin/bash
mamba install -y metview-batch

Removing a containerised conda environment

If you no longer need an existing containerised environment, you may simply delete the top level installation directory

rm -rf <install_dir>

Integration with JupyterHub

You can expose your containerised conda environment as an additional kernel in ECMWF's JupyterHub service.  To do that, first make sure you have python and ipykernel installed, and then you can create the kernel with:

<install_dir>/bin/python3 -m ipykernel install --user --name=<my_containerised_env_name>