Page History

...

• s3 using Python libraries
• Object storage: How to use s3cmd and s3fs
• S3 from command line

for more help. 

Mounting the buckets using FUSE

Following describes how to mount EUMETSAT data buckets as FUSE mounts. The possibility to mount the buckets into as filesystem is provided to as convenience. Readers are advised to note that mounting buckets via FUSE will provide significantly worse performance than accessing the data directly from workloads, for example using python.  Whenever possible, users are advised to use the buckets directly from the workloads.

Tenants having Morpheus account at EUMETSAT Morpheus, can mount the data buckets using ready workflow. This article describes how to run the workflow and how to mount the data manually. One can naturally use the same steps to any other buckets as well by modifying bucket name, service endpoint, credentials, and potentially project id.

Using the Workflow

1. Go to instances and select the instance of interest
   Image Removed
2. Select Actions → Run Workflow
   Image Removed
   
3. Select Mount EUMETSAT databuckets from Workflow drop-down menu and click Execute. Command options are not needed.
   Image Removed
   

You can then follow the workflow from the output or by going to Instance History tab. After the workflow is executed, you can access the data at /eumetsatdata.

Manual mount

Following steps are using ASCAT data as an example. Please modify the commands to mount any other bucket. 

Make sure you have latest rclone installed

curl https://rclone.org/install.sh | sudo bash

Create mount directory, i.e. for ascat:

sudo mkdir -p /eumetsatdata/ascat && sudo chown -R $USER:$USER /eumetsatdata/ascat && chmod 755 /eumetsatdata/ascat

Put rclone like listed below config into a config file. For example, into /etc/eum_bucket_creds (keys without brackets) 

sudo sh -c "cat >>/etc/eum_bucket_creds <<-EOF
[eumetsat-ascat]
type = s3
provider = Ceph
endpoint = https://s3.waw3-1.cloudferro.com
EOF

...

Now you can already mount the data with command:

sudo rclone --config /etc/eum_bucket_creds mount --allow-other --daemon eumetsat-ascat:ascat.metop-abc.level-1b.native /eumetsatdata/ascat

Setting up a service

To have mount points available all the time and after reboots, one need to set up the service and add mount points into fstab.

Link rclone

sudo ln -s /usr/bin/rclone /sbin/mount.rclone

Create systemd service:, for example for ascat:

sudo sh -c "cat >>/etc/systemd/system/eum-data-ascat.service" <<-EOF
[Unit]
Description=Mount EUMETSAT data bucket ascat
After=network-online.target

[Mount]
Type=rclone
What=eumetsat-ascat:ascat.metop-abc.level-1b.native
Where=/eumetsatdata/ascat
Options=ro,allow_other,args2env,vfs-cache-mode=writes,config=/etc/eum_bucket_creds,cache-dir=/var/rclone,vfs-cache-max-size=512Mi,vfs-cache-poll-interval=30m'
EOF

Add corresponding line into /etc/fstab:

echo "eumetsat-ascat:ascat.metop-abc.level-1b.native /eumetsatdata/ascat rclone ro,noauto,nofail,x-systemd.automount,args2env,allow_other,vfs_cache_mode=writes,config=/etc/eum_bucket_creds,cache_dir=/var/cache/rclone,vfs-cache-max-size=512Mi,vfs-cache-poll-interval=30m 0 0" | sudo tee -a /etc/fstab

...

Mount the folder:

sudo mount /eumetsatdata/ascat

Limitations

...

While not recommended, you can also use FUSE mounts to access the buckets. For that, please consult Mounting EUMETSAT data buckets using FUSE

Related articles

...