All Hail the RTB Champion

“Hear Ye, Hear Ye! Allow me to regale you with the tale of the RTB Champion. Our hero keeps the dreaded RTB at bay and selflessly sacrifices themselves for the betterment of the team. Gather around and listen to their story.” - Town Crier

Seriously, the run the business (RTB) champion role has been a key factor to the success of our engineering team. As a long running team, we’ve had to fight the age-old battle of balancing developing new features/functionality with keeping on top of bugs, tech debt, customer support and other RTB tasks. In a perfect world we wouldn’t generate bugs or leave behind any tech debt, but it’s simply not natural in a complex, evolving system. There will be bugs. There will be tech debt. There will be customer support. Our goal is to reduce the amount and manage it, learning from each piece and evolving our product and our processes for the betterment of our customer and humankind — well — the team.

We have sworn that we will dedicate 20% of our time to RTB with the rest allocated for new features and technical tasks. Previously we had tried to scope in RTB as part of our sprint planning with the execution of the RTB tasks being a free-for-all where any member of the team that was available would pick up and clear the tasks as they came in. Whilst this approach works — the team was able to tackle sprint work and RTB that came in — we found that there was a price: the cost of context-switching. Let’s imagine:

A team member is working on a task which is part of the sprint goal for feature X. They finish the task and pick up a ticket for an alert that we received. They investigate and resolve the issue and then they need to familiarize themselves once more with the sprint goal and pick up the next task.

There are two context switches that occur here, one from the sprint work to the RTB, and then another back from RTB to the sprint work. There’s a big difference between the two types of work. Sprint work requires more context and focus whereas the RTB is usually driven by following runbooks or doing investigations. They demand of the engineer two different mindsets entirely and we didn’t realize it at first but it was a time expense for us to have this context switching occurring. We began to see a trend that when one team member switched to focus on RTB and took a string of RTB tasks in a row that they would actually process the tasks quicker than someone who was constantly having to context switch. This surfaced itself in a retrospective, the team discussed and we decided to experiment: what if we put one person 100% focused on RTB?

We tried it and observed that it paid off. The team members that were focused on sprint work found their productivity boosted due to not having to switch contexts. The team member that was in RTB mode for the week was able to move through tasks as they came in, always in the right context and benefiting from working through similar processes. We were able to tackle RTB at a higher capacity. It also had some side benefits. The new RTB champion role led to some ‘discovery’ and ‘innovation’ as some tasks had been found to be repetitive, but we hadn’t realized before because different people were taking them. We automated some processes for gathering data, improved our reporting and have made significant leaps forward in our managing of RTB.

The definition of the new role also meant our team had a key point of contact for other teams when they brought RTB type work to our attention. It was still possible, as it had always been, that they could come to any member of the team and raise the issue. However, we observed that in having an RTB champion and making it clear who that champion was for any particular week (through setting it as the topic in our slack chat), that other teams would go directly to the RTB champion with their concerns. This prevented further context switching and has helped the team to remain focused and productive.

Having someone with the full view/context of all the RTB also helped in reporting of the RTB. We have a weekly operations review where we review our operational excellence and having someone in the team who knows everything that occurred during the week helps to facilitate the conversations and communicate effectively the goings-on of the week. It has helped to spot trends, find points of focus and shape our technical roadmap. We strive to reduce RTB as much as possible either through elimination or mitigation.

We’ve been practising and iterating over the role for a few months now. The RTB champion role has evolved with the mandate to experiment more on trickier issues or to take on technical debt tasks when the RTB volume was low.

And that is the tale of the RTB champion. A hero to our team and a change which has brought much success. If you have something similar in your workplace I would love to hear about it and perhaps learn from it. If you haven’t then I would urge you to consider experimenting with an RTB champion rotation. It may not work for you, but then again, it just might.