Before Christmas 2022 lots of work went into getting recordings from RSECon22 uploaded onto the Society YouTube Channel. These are now live and viewable via the RSECon22 playlist (see below). We’re grateful to Ed Bennett for sorting this out and also for meticulously adding speaker responses to questions they didn’t have a chance to answer in person to the videos. However, in a number of instances speakers provided more thorough answers to questions that can’t be included in YouTube video descriptions (due to character limits) so we’ve included them below.

RSCon22 Talk Playlist

Matt Machin: What is co-design and why should you use it?

Extended Q&A

• What proportion of the overall project time do you think should be allocated to co-design?

This will depend a lot on the particular project. A key thing that impacts this will be what stage of development you are in with the software / intervention. If this is something that has already been used by participants in previous work then the amount of co-design required will be less. If it is brand new then more co-design is likely to be required. To give an example, in a typical health research project that I’m involved with we might have, for instance, 12 months of development work followed by a 24 month trial. Within the 12 month development work I would expect to see extensive co-design throughout. During the trial this will be much more limited (as the software is not changing / not changing much) but there will probably still be some, focused on learning from the participants in the research.

• What would you say is the difference of this compared to a framework like “Design Thinking”? Instantly it looks more lightweight and easier to adopt and adapt

My answer to this needs to come with an important caveat: I have never personally used Design Thinking for a project so I don’t have the (very valuable) real world experience of using this approach. From my understanding of Design Thinking, there are certainly some similarities with co-design such as the different design phases. Both are also iterative approaches. However, from my understanding of Design Thinking you can re-order the different phases to meet your needs. In co-design you would not normally do this. Yes, you iterate through each of the phases and go back to the start (when appropriate) but you wouldn’t normally run them in a different order. Design Thinking seems to also be focused on innovative / ground-breaking solutions. Co-design can be used for this but it is also used to just improve existing solutions without the need for the changes to be particularly ground-breaking.

• In my experience, grant-funded projects put the onus on a small set of individuals to assess deliverables. Does this restrict the co-design approach?

Yes, this can be a limiting factor. To help with this it can be very valuable to include one or more research participants (“users”) as part of the research team. That can help to bring a wider perspective.

• How do you deal with overlapping groups? Eg. when the researchers are also users of the software. This is quite common outside healthcare

That certainly complicates things somewhat and I think would require a somewhat different approach to co-design. In this situation you really only have 2 key groups: researchers / users and RSEs rather than the 3 I described in my talk: researchers, users, RSEs. I think you can still apply the co-design approach but it would be important to ensure there is the right “balance of power” between the two groups so that the researchers / users don’t have too much control over the process.

• How do you/funders measure the impact of co-design?

An interesting question and one that is not easy to provide a complete answer, particularly given that co-design is still a relatively novel approach. In our use of co-design we apply (relatively) standard approaches with an existing research base (although still evolving) to support them. We are not therefore usually looking to measure the impact of our co-design approach directly. I also don’t think that funders really assess the impact of co-design in projects like ours. They usually look at more traditional research impact measures such as publications, spin outs or other translation etc. I think the impact / effectiveness of co-design approaches is more likely to be assessed in research projects that are developing and testing new co-design approaches directly, as opposed to just applying them to other areas of research.

• Do you find a significant difference in feedback between 1on1 vs group sessions?

In short, yes. Both approaches are very different and some people feel much more comfortable to provide feedback in a 1-2-1 session than they would in a group. In a group session there is the opportunity for ideas generated by one person to be further developed by others. Both are valuable.

• Re. disruptive participants: is there a point in your process where co-designers have to sign off on an agreed plan?

Yes there is but in most of our projects the “sign off” is provided by the PI leading the research rather than the team as a whole. I think we could consider having a wider “sign off” in future projects.

• Are there projects that are a good fit for co-design vs user testing before release? The latter might be better for cases like your dementia example?

The problem with only doing user testing before release (without co-design) is that you get feedback far too late in the project to be able to meaningfully change what is being developed. If you develop something, e.g. an app, and then trial it with users and they hate it / can’t use it after you have spent all of the money and time what do you do then? I would absolutely recommend user testing before release (and we do this in our projects) but that should be in addition to co-design rather than instead of. Even the somewhat restricted co-design that we undertook with our dementia project still resulted in important changes to what we developed (e.g. One of the cognitive tests was swapped for another because the original was not suited to the user group) that would not have happened without that approach.

Ginestra Ferraro: A troubled love story of integration

Extended Q&A

• Who are the target users for the software you work on? Researchers or “general population”?

In KDL we usually have two main groups we target: researchers and the general public.
The nature of our work is to create digital platforms that support some ongoing research, so what we produce has to be used by this user group. But also, the final product will serve as an entry point to the research findings for the general public. Of course, when we say the general public, we are still talking about a small slice of the population. It could be primary school pupils of a specific country, a well defined online community, or visitors of a particular museum. Defining our audience is as key as testing afterwards. Different sections of the platforms may be intended for different user groups.
“Designing for Anyone Is like Packing for a Trip to Anywhere” (Kathryn Whitenton, NN Group) (see:https://www.nngroup.com/articles/everyone-as-users/)

• Are there particular design considerations for research particularly the (perceived?) tension between ease-of-use vs reproducibility?

If we are talking about ease-of-use for the end users, I don’t see a conflict between the two necessary, except maybe the initial investment (time, resources, hence money/budget). Ease-of-use should benefit from reproducibility, meaning that if there is a decision to make a design component reproducible, it should be for something that has been tested with the intended audience and improved at every iteration (and every release). If something is not easy to use for the end users, maybe it shouldn’t be made reproducible.

Instead, if we are talking about ease-of-use for designers and developers building software together, then a balance needs to be struck: every role needs a space where they are comfortable working and doesn’t feel like their efficiency is undermined by the tools they have been asked to use. How the share of effort is distributed can vary depending on workflows and team set ups. I’m simplifying here again, but for instance, if within a team, we have a group that is more resourced, I would expect that sub-team to commit to a bigger effort to accommodate the other group’s workflow. Defining responsibilities early is important for communication as well, to avoid stepping on each other’s toes.

It does not mean taking on the responsibility of another role, although we know that can naturally happen at times, it means supporting a collaborator to optimise the resource available.

• What can developers in teams without UI/UX specialists do to improve their designs?

I am not going to list resources available online, there are plenty and are easy to find.

• The list below is based on my experience, and it can be easily mirrored in the other direction too.
• If you need to work based on assumptions (no user research, no marketing analysis, etc.): test your assumptions as early as possible, multiple times.
• Test with (real) users and low-fi tech (slow network, old devices, etc.).
• Keep it simple, stupid (it’s not an insult, I promise, it’s the KISS principle!).
• Research the pattern you are going to use, if there is none, and you’re building something new, you might want a designer, even just for a chat.
• More than anything: empathise.

• What happens when we don’t integrate UX/UI workflows?

Mostly, frustrations. My experience shows that when that happens, a series of events often presents itself:

• Extra iterations of development: the product doesn’t perform as expected, one or more iterations are needed to “fix it”.
• Blind development: true for both designers and developers. E.g., if a designer is not aware of a technical limitation, it might propose something that is impossible to implement in that context. That design will need a second iteration that could have been avoided. (There is a conversation to be had about allowing creativity to be expressed in full and constraints can limit such endeavour, but that’s another topic!).
• Staff feeling undervalued: a recommendation, a principle, or a good practice is ignored: the role from which that recommendation comes from feels their work is considered less important.

All that is true in an environment where designers and developers are available, but an effort to integrate the roles is not made.

The reason Agile methodologies are so popular in software development is to create pockets where these intersections between workflows are made. 

The Waterfall approach works well in situations where workflows are sequential (building/construction engineering), but imagine designing or developing (one OR the other) a full digital platform down to the minimal detail (again, detail of design OR development) only to find out it doesn’t perform as expected after the other role puts their hands on it?

It’s likely that different components of the platform are interdependent and a cascading effect is triggered as soon as a change (from whatever role) is applied. The natural consequence is that the two roles will have more work to do.

• If designing for non-technical users, how do you balance the need for easy use/access with complex systems and outputs?

Always privilege the user. That is not to say that we should avoid presenting difficult or cumbersome user journeys. If there are no better solutions, what we should strive for is education and support. We can learn from users, but users learn from digital interactions all the time and new conventions are established. But there need to be very good reasons why new patterns are created. Furthermore, for these to become resilient, they need to be tested, essentially building with users gives a better chance of creating a sustainable product.

Ghisain Vaillant: Successes, pitfalls and lessons learnt from developing a multi-year RSE project

Extended Q&A

• Did someone have to be convinced to fund the work?

The work is funded through project proposals submitted to the French national research institutions, such as CNRS or Inria for instance.

• Could you elaborate a little on ubiquitous language?

Ubiquitous language is one of the stepping stones of Domain-Driven Design. Both domain experts (typically researchers and PIs) and engineers should strive to define a common language to express their problems, needs and solutions and the terms for this language should be used verbatim in code. The immediate benefit of defining the UL early is reducing cognitive load due to unnecessary translations between technical terms found in code and their corresponding domain representation when discussing project advancement or reporting for instance.

• How do you cope with users pushing for scope creep when they’re the PI and *funding* you?

Feature creeping often arises from poor scope definition early on during the planning phase for the project. Having PIs participate in scope definition and project statement gives them shared ownership of the boundaries of the project. Now if they are willing to break them despite warning signs raised by the technical team, they own it.

• What is the most useful way for users to give their feedback? GitHub Issues? Project management software?

It depends a lot on your users, so getting to know them will help you enable the right feedback communication channels. For Clinica for instance, we have got technically savvy users who are comfortable reporting problems through GitHub issues, whilst others prefer posting on our forum or contacting us directly by email.

• Do you do systematic code reviews ? If yes what is the workflow ?

All changes to the code base require a pull-request which should be reviewed and accepted by at least one of member of the technical team depending on the nature and complexity of the changes.

• Great talk, how do you manage saying no for strict scoping as someone in the infancy of their career?

Define a clear project statement and explicit scope to the project. I’d go as far as to say, it should be the very first piece of content in the project’s README. It’s easier to say no when expectations are set early and communicated clearly.

• How do you deal with the time it takes to review community contributions? How is the payoff Vs time invested?

Clinica is quite a niche project so our user base is small and community contributions are infrequent enough to be handled on an ad-hoc basis. We have yet to reach the stage where we need to weigh payoff versus time spent.

• What are good ways to encourage community discussion with your user base?

Being responsive to support requests definitely helps, since interest fades away very quickly overtime. If your community becomes large enough, letting other community members participate in answering issues or engage in design decisions are also beneficial. In Clinica, we use GitHub discussions as a means for sharing design decisions and get early community inputs on future directions.

• Do you actively try to encourage outside code contributions to your project?

As much as we possibly can. Users are encouraged to submit code and / or documentation changes when applicable.

• When does user research come in? I hope not at the end 

I am assuming you are talking about user experience (UX) here. Clinica was initially  designed as an internal tool to the ARAMIS lab before release to the public. As such, refinement to the UX happened by internal dogfooding as development progressed internally. Now that Clinica is available publicly and its CLI has stabilized, we have become more reluctant to changes to the UX.

• What are the basic features of an agile project management system?

Any system that facilitates software development in accordance to the principles listed in the Agile manifesto (https://agilemanifesto.org/). We found that adopting a “Release early, release often” mindset (with a 6 to 8-week release cadence) and striving to reduce meetings to the minimum necessary to be effective facilitators.

• Why is there so little research on whether any specific practices are actually effective?

No idea. Perhaps no one cares actually.

• What are the main reasons that software projects fail in academia? (As distinct from private sector)

I can think of a few that I have experienced myself:

1. limited and volatile funding

2. staff turnover with little to no knowledge passing

3. poor scope definition

Neither is really specific to research, but research is particularly exposed to them.

• How did clinica moved from 1 postdoc to a team and is still maintained after 6 years? how is it funded?

See the first answer.

• “F# for fun and profit” is a *great* resource for domain-driven design 

Indeed. “Domain Modeling Made Functional” is also a great resource from the same author (Scott Wlaschin).