Dear TOAR-Community,
An update of the TOAR database infrastructure will take place between next Monday, May 26th, and Wednesday, May 28th. This involves:
- improvements to the dashboard,
- additional metrics in the analysis service
- the newly developed gridding service
- introduction of TOAR User management and registration.
Please read on to understand these changes, and in particular the new use constraints that are related to the user management.
Happy analyzing and best regards,
the TOAR data infrastructure team
TOAR user concept
Since accessing TOAR services can consume considerable resources, we are forced to introduce user registration and login to the TOAR data infrastructure. It will still be possible to explore data as an anonymous user, but there will be strict limitations on the number of data sets and/or length of timeseries that you can process. Registration is easy and can be done with a few clicks from the dashboard.
Explanation:
Access to the TOAR services will be controlled through roles. If you are not logged in, you will be regarded as an anonymous user with very limited access rights. Self-registration elevates you to a registered user, which enhances your download and processing abilities. As a scientist who is directly involved in a TOAR activity, you can request to become a TOAR user with even more capabilities. Finally, limited individuals who are in charge of producing global analyses for the TOAR assessment can be elevated to power user. The detailed rights associated with each user category are listed below.
Once registered, you can log in to access all TOAR services via the login button on the TOAR dashboard. The login makes use of the Helmholtz AAI. If your institution is not connected to this (it does not appear on the list of institutions shown) you can also use your ORCID or a Google account.
If you need help with the registration, please send us an email to toar-info@fz-juelich.de. To request elevation to the TOAR user status, please send the request via the button in your personal profile in the dashboard and provide us with a short introduction of yourself and your involvement in TOAR.
Gridding service
Through this service you can generate gridded data products at arbitrary regular grids, thereby applying various filters and statistical metrics. This is especially useful for the evaluation of atmospheric chemistry models. For details, see documentation.
New metrics in the analysis service
A new dma8avg metric has been added to the TOAR analysis service. This calculates the same daily statistics as dma8epax but provides simple arithmetic mean values for annual, summer, or xsummer sampling periods. It can also be applied to the custom sampling period. Note that the original dma8epax metric returns the 4th highest daily value for these sampling periods. This behaviour has not changed.
A new osdma8 metric has been added to the TOAR analysis service upon request of the Health working group. The OSDMA8 metric is an annual aggregate that is calculated for a given year x from all data for months 1-12 of year x and all data for months 1-3 of year x+1.
- the daterange passed to the “analysis” service denotes the data to be loaded from the database for the calculation.
- dma8-monthly averages are calculated for all loaded data.
- starting with the first monthly aggregate, 6-month-running-means are calculated (i.e. the mean value is always calculated from 6 consecutive monthly means, and then the time window is moved forward by one month).
- in the last step, the maximum is determined from these moving averages (since only data from x-01-01 to (x+1)-03-31 are to be used for this annual value, the maximum is determined from 10 values in this last step).
Improvements to the dashboard
The services “Download Data”, “Get statistics”, and “TOARgridding” have been fully integrated into the dashboard.
Logged-in users are provided with a profile where they can see their user status and set a few user options (this will be expanded in the future). Furthermore, the profile makes a history of important requests from the dashboard available so that it becomes easy to re-use requests that you found useful.
In addition to these enhancements, minor bug fixes and optical improvements were implemented.
Resource limitations and user roles
To handle the very different types of database requests, we established the new TOAR Billing Unit (TBU), which combines two types of request parameters: length of timeseries and number of timeseries.
A TBU is a (partial) result of the Analysis Service or the TOAR Gridding Service. It combines a daterange and the number of timeseries that are processed in one request. We count 5 years of statistical analysis of one timeseries (arbitrary number of metrics) or aggregation of data from max. 6,000 timeseries as 1 TBU. This means that, for example, a complete global map of 10,000 ozone datasets will count as 2 TBUs, or the calculation of 10 metrics on one 30-year timeseries will count as 6 TBUs. Requesting a 20-years-calculation for all ozone time series (approximately 18000 time series) will therefore result in 12 TBUs (4-times 5 years x 3-times 6,000 timeseries ids = 12 TBU). Note that API requests with no specified daterange automatically count as if the maximum available daterange (55 years = 12 TBUs per 6,000 stations) had been requested.
Anonymous user:
- users who are not logged in
- max download of 5 time series at a time
- no custom gridded products, but only download of a static example file
- access to the REST API only for 3 time series within the ID range of 1000 to 2000
registered user:
- users who self-registered but did not get the privileged TOAR user access
- max download of 10 time series at a time, max. 250 time series per month
- max 1 concurrent gridded product per request, max 12 gridded products per month
TOAR user:
- users who self-registered and were elevated to TOAR user after sending us a register request via the dashboard
- max download of 500 time series at a time, max. 10000 time series per month
- max 12 concurrent gridded products per request, max 120 gridded products per month
Power user:
- special users who have a mandate to perform extensive analyses for the TOAR assessment papers or other relevant studies. These users should demonstrate that they understand the TOAR data services and can work efficiently with them.
- “no limitations”, access until the hardware winces
Best regards,
The TOAR database team at the Jülich Supercomputing Center
Image from Conny Schneider on Unsplash
