Introducing Ed Bennett from Swansea University! Read about his background and fellowship plans here.

Ed Bennett – Supporting reproducibility in STFC science

Hi! I’m Ed Bennett, and unlike the authors of the other blog entries in this series, rather than EPSRC, I’ve just become the first RSE Fellow to be funded by STFC. STFC has a broad remit, covering national facilities, the UK’s contribution to international experiments, and also funding research in particle physics and cosmology within the UK.

While for the last few years I have co-led the RSE effort of the Supercomputing Wales programme at Swansea University, before that I was a researcher working in an area of mathematical physics called Lattice Field Theory. This takes quantum field theories that are intractable analytically, and makes use of Monte Carlo methods to calculate observables numerically, allowing calculations both in the Standard Model of particle physics (“Lattice QCD”), and of other theories that could provide explanations for observations that don’t match Standard Model predictions. The main focus of my Fellowship is going to be supporting reproducibility in these calculations.

RSEs aren’t a new concept in LFT. Since calculations require vast amounts of computing resource, a large number of High Performance Computing-specialist RSEs are employed to develop and optimise efficient software. This has been done for many years, since even before the term “RSE” was coined, by organisations such as EPCC at Edinburgh, and DiRAC, STFC’s national supercomputing service for particle physics and cosmology. While there has been a huge effort applied to making the HPC problems run quickly, there has been less attention paid to what happens to the data once they come out of the supercomputer.

Think of Monte Carlo methods are similar to a large experimental facility. A lot of computer time is spent generating a set of samples on which measurements can be made. A smaller, but still significant, amount of computer time is spent measuring observables from these samples. Then, researchers take the resulting values away and use their own workflows to extract meaning from the numbers. The first two steps are where the RSE effort has largely been applied to date – to make an analogy with experimental science, these steps roughly correspond to the “equipment construction” and “data collection” phases. Being able to do these in a reproducible way is incredibly valuable, and put the field on a very solid footing.

However, as reproducibility efforts in other fields have shown, being able to consistently and reproducibly get the same analysed results from the same input data is a difficult problem in itself. Fully automating the analysis, so that collected data are turned into plots and tables for publication at the push of a button, can get one much closer to this ideal. LFT isn’t quite there yet, however: many researchers (myself included) still rely in some cases on numbers copied and pasted from emails, and plots generated by hand using tools like Grace and gnuplot. This leaves huge amounts of room for inconsistencies to sneak in to an analysis!

The main piece of work I am going to lead during my Fellowship is to try and help this situation improve. The first step, which I’m just about to start work on, is to work with the community (or at least a representative slice of it) to identify a set of best (or at least “good enough”) practices to enable automation and publication of analysis workflows. Having attempted this myself, even with a clear idea of what the solution “should” look like, putting it into practice is not trivial, and requires a significant time investment. The second aim of this work, therefore, is to develop tooling to make it so that this becomes easier, and needs less work, than the current way of working. Finally, learning new tools takes time, and adopting tools that are unproven and have yet to develop a good reputation is a risk. The third thrust of this work will be to work with researchers to integrate these practices and tools into their own analysis workflows, to prove their utility.

My aim is to build on work already being done in this space, and to similarly build tools that are broadly applicable where this doesn’t compromise usability within the LFT domain. My hope is that as well as promoting and driving reproducibility in this community, the work will also contribute to a broader push in this direction.

Since STFC are only funding a single Fellowship in this round, I thought it was important to not solely focus on one community, given the breadth of science that STFC supports. I’m also going to be working with researchers at Cardiff University in the LIGO Scientific collaboration to improve the efficiency of their simulation workflows, and at Aberystwyth University working on numerical models of the Sun to remove the restrictions on parallelisation currently imposed by using a proprietary language with per-core licensing. I’ll also be working with the rest of the RSE team at the Swansea Academy of Advanced Computing (SA2C) to train researchers across Swansea University, including a renewed focus on techniques for reproducible analysis. Finally, I will be working to put in place a sustainable career structure for RSEs at Swansea; rather than serial short-term contracts with little opportunity for advancement, I aim to learn from the excellent work that has already been done elsewhere in the UK and apply it to Swansea, providing improved opportunities for both career progression and job security.

When I see the portfolio of work that’s been done and is being done by the EPSRC Fellows, I’m amazed by the breadth, depth, and sheer quantity of work they’re able to accomplish, and daunted by how I can measure up against them. What I’m trying to keep in mind is that nobody is an island. I don’t know whether it’s possible for one person to change the world, but I know that I can’t do it by myself. But if enough of us will push in the same direction, we can ultimately create a positive change; my aim for the next five years is to try and push in the right directions.