How to download Copernicus Contributing Mission products to object storage
Copernicus Contributing Mission (CCM) products can be ordered in EO Finder https://finder.eo-lab.org. Most of the products can be found in a common repository, when the order is completed. CCM products are ordered differently and can be downloaded to Object Storage only. This manual describes how to order and download CCM products.
Object Storage
First, you need to create an Object Storage. Object Storage can be accessed via the OpenStack Dashboard (Horizon) and console. The Dashboard is a convenient way to manipulate your Object Storages. Go to https://cloud.fra1-1.cloudferro.com/auth/login/?next=/ and log in with SAML2 authentication. Choose a Project → Object Store -> Containers tab on the left.
Choose “+ Container” and select a name for your new bucket.
You should see newly added container in your Object Storage list.
Console Access
The following steps will require to have applications like openstack-client and s3fs. Most modern distributions will propose to install them from default distribution repositories.
Let’s try to set up console access to your bucket from the console. Access can be performed in your virtual machine or local Linux machine. In order to have access first, you need to download an RC file. Go to your Dashboard and navigate to Project → API Access. Select “Download OpenStack RC File” button and press “OpenStack RC File”.
Save the file in a directory of your choice and go to the same direction in a console. Run the following command to import RC file data. You will be asked to provide a password which is the same as your EO-Lab.org profile password.
source ./cloud_00XXX_1-openrc.sh
Please enter your OpenStack Password for project cloud_00XXX_1 as user
When the password is provided correctly you are authenticated. There will be no prompt about the wrong password in case of a mistake. If the next command results with “The request you have made requires authentication.” message, just run source command once again and provide the correct password.
Run an OpenStack command to obtain EC2 credentials:
openstack ec2 credentials list
You should see the following output similar to this:
+----------------------------------+----------------------------------+----------------------------------+----------------------------------+
| Access | Secret | Project ID | User ID |
+----------------------------------+----------------------------------+----------------------------------+----------------------------------+
| e82b581s08bk4doiu2344g19f79f112b | 88ba74d9h3alow5761dce904047652aa | 8b939edi8n3f0a7fk5890ef4cced1e0a | 55a0826d0fl6sm2v0td953vf989fc873 |
+----------------------------------+----------------------------------+----------------------------------+----------------------------------+
Note down Access and Secret keys. They will be needed later in Finder ordering. In the meantime, you can check them to browse your Object Storage bucket from a console. Create a credentials file which includes Access and Secret keys:
echo Access_key:Secret_key > ~/.passwd-s3fs
This command will create a file in your home directory. Now you can mount your bucket with the following command:
s3fs container /mnt/container -o passwd_file=~/.passwd-s3fs -o url=https://cloud.fra1-1.cloudferro.com/auth/login/?next=/:8080 -o use_path_request_style -o umask=0002 -o allow_other
“container” should be changed to your container name and /mnt/container to your favour mount point.
Access to Copernicus Contributing Mission Products
CCM products require to have working CDS-SSO credentials. More information about Copernicus SSO account can be found on the official Copernicus webpage: ssoidp.copernicus.eu. To register an account navigate to the Registration page and provide required data: https://ssoidp.copernicus.eu/idp/umsso20/registration. CDS-SSO ID and Password will be used during the ordering process in Finder. In order to have access to the chosen repository, you need to report it into Copernicus Portal in “Personal Area” and request for the access. Fill in “Account Details” and”User Category” and submit the request.
CCM Catalogue in EO-Lab Finder
Open https://finder.eo-lab.org and login with EO-Lab credentials. Select “Copernicus Contributing Mission” option from Data Catalogue.
There will a Collection also to choose from:
Select processor to CCM download.
And press search. Select products you want to add to order.
When products are added select Cart button in the top right corner:
Now it is time to use EC2 credentials which were generated with “openstack ec2 credentials list” command. s3 bucket is the name of created object storage in OpenStack Dashboard. CDS_SSO username and password were created during registration on Copernicus Portal.
Now you can finish with an Order button. When the order will be completed products will be available in your Object Storage.