EODATA - Earth Observation Data Repositories on EO-Lab
In this article you will find definitions of terms and practical hints when working with Earth Observation data on the EO-Lab cloud.
EO-Lab provides access to one of the largest European satellite data archives with a major focus on data and information products from Sentinel missions. The Sentinel satellites are the space component of the European Commission’s Copernicus Earth Observation program.
Note
In the Knowledge Base Articles on EO-Lab the term EODATA is used as a synonym for the Earth Observation Data Repository associated with the EO-Lab cloud. However, in some articles the term EODATA may refer to the CREODIAS EO Data Repository.
The EO-Lab team is busy resolving such ambiguities in the Knowledge Base Articles.
Earth Observation Data Repositories
On all CloudFerro clouds Earth Observation data is organized in individual collections (usually defined by the satellite mission name, e.g. Sentinel-1). These Satellite collections are further structured by their processing levels and thematic content. For example, the acronym “Sentinel-1 L2 BackScatter (BS)” stands for a Level 2 product providing terrain corrected backscatter data from the Sentinel-1 mission.
Earth Observation Data Repositories are composed of assemblies of several collections within one cloud.
On EO-Lab, the following Earth Observation Data Repositories can be accessed:
The dedicated (native) EO-Lab EO Data Repository hosted in the cloud FRA1-1 in Frankfurt.
The CODE-DE EO Data Repository providing online archives of EO data recorded over Germany also hosted in the cloud FRA1-1 in Frankfurt.
The CREODIAS EO Data Repository allows access to worldwide data in a rolling archive physically stored in the cloud WAW3-1.
Those EO Data Repositories can be interactively browsed using the EO-Lab Data Explorer.
The data repositories are updated on a daily basis and newly recorded data and data products are ingested in the respective collections. CloudFerro pulls data from two main sources:
the EUMETSAT data service and
Typically, products are updated with a latency of less than 30 minutes after publication on the source site.
Technical Setup of Earth Observation Data Repositories
From a technical point of view, all individual data collections (representing satellite missions such as e.g. Sentinel-1) within Earth Observation Data Repositories are stored in so-called buckets or containers using S3 object storage technology set up on HDD Ceph filesystems.
For all CloudFerro Earth Observation Clouds access to the native EO Data Repository is granted by default. Native in this context meaning the repository is physically located on the same cloud.
Access via a EO-Lab Virtual Machine
When starting a Virtual Machine based on a predefined VM image on EO-Lab, the respective EO-Lab EO data repositories will automatically be accessible as mounted directories. The buckets (representing collections/ satellite missions) appear as sub-folders in the mounted directories. In more detail, the following generic logic applies for the folder structure:
repository_mount_endpoint/satellite_mission/instrument/data_product/year_of_acquisition/month_of_acquisition/day_of_acquisition
For EO-Lab the listing for the repository_mount_endpoint /codede would look like this:
Access via S3 tools
Instead of mounting S3 buckets as directories, a faster and more elegant way of accessing EO Data Repositories is using dedicated S3 object storage tools such as s3cmd: How to access EODATA using s3cmd on EO-Lab. To configure access also use How to Create S3 Credentials on EO-Lab to get the necessary Access Credentials.
Important Note
Attention
For historical reasons, in some EO-Lab Knowledge Base Articles the term EODATA explicitly refers to the CREODIAS Earth Observation Data Repository. Thus, some adaptations might be necessary when following tutorials or articles:
In case you wish to use such an article to work with EO-Lab native EO Data Repositories make sure to replace all CREODIAS specific entries with the correct EO-Lab entries.
For example, on CREODIAS Virtual Machines the native EO Data Repository by default is mounted to a folder called /eodata whereas on CODE-DE it is named /codede.
The mount endpoint for the configuration of S3 tools on CREODIAS is https://s3.cloudferro.com, whilst on EO-Lab it is https://s3.fra1-1.cloudferro.com.
In case you wish to use such an article to work with the CREODIAS Earth Observation Data Repository, make sure you have requested the corresponding access credential from EO-Lab support. See also: How to access the CREODIAS repository from a VM?.
If in doubt, please contact the EO-Lab Support Team. We also highly appreciate your feedback on possible ambiguities or inconsistencies.