We are looking for a Research Software Engineer (A2) to work on the development of data infrastructure services to support Machine Learning (ML) efforts within the European Union’s Destination Earth (DestinE) initiative.

ECMWF produces world-leading scientific datasets. Connecting them to the myriad of users around the world is a challenging but truly valuable mission. Patterns of data use have been changing more rapidly than ever before, driven significantly by the explosive growth in AI-model use and training, while the resolution and diversity of the data is ever increasing.

In this world, novel approaches are critical to make datasets available to a wider audience, and for a wider range of uses. ECMWF has developed a range of new technologies for accessing meteorological data. Recently we have developed Zarr-FDB – a prototype system to create on-demand, user-defined views of data stored within our indexed data stores (the FDB), built making use of other developments (including Polytope and Gribjump). This allows custom views to be created according to scientifically-meaningful descriptions, which can be directly used for data exploration, visualisation, data-science and ML training.

In this role, you will explore and innovate solutions to improve data and related services to support the scientific and technical development of ML models and processing for the DestinE digital twins. You will develop tools and workflows to assist with assembling and efficiently serving datasets for ML training, and resulting meteorological output. You will have opportunities and be expected to bring creative solutions for handling large datasets across multiple data centres.

This will involve taking the Zarr-FDB prototype functionality and bringing it to life as a user-facing service – working with our production team, user-facing service owners, infrastructure providers and researchers to build a robust, observable, performant and above all useful capability.

You will work in the Data Management Services Team in the Development Section of the ECMWF Forecast and Services Department. We build robust and performant software and services to handle and manipulate large volumes of meteorological data. This software is used heavily in ECMWF’s operations, research, projects and initiatives, through all stages of our workflows from acquisition of observations, through model output on the HPC to archival and dissemination, and across a wide range of contexts from HPC, through dedicated storage infrastructure to cloud systems.

We have a special focus on a semantic approach to data handling, such that the technical realities of data handling are abstracted from the scientific software. We expend significant effort to navigate scalability challenges posed by future model upgrades and to exploring new technology and approaches for the future. This supports ECMWF’s strategy of producing cutting‐edge science and world-leading weather forecasts and monitoring of the Earth system, and DestinE’s goal to operationalise digital twin technology.

In summary, you will be developing future-proofed data-access, supporting our ML efforts and our external users with tailored, high-performance, scalable, semantically driven data access. This role is funded by the Destination Earth Initiative and collaborates closely with other activities of our Member States and with our partners ESA and EUMETSAT.