Sharpening the Axe

Software projects routinely take longer than expected. When teams look back to figure out what went wrong, one common culprit is that various “distractions” came up. What if you could eliminate these distractions? We believe you can.

What constitutes a distraction is unique to each engineering organization, but they tend to be some variant of the following:

• Different teams reach out for engineering help. Some examples are…
  • Customer support is running into an issue they need to escalate
  • The quality assurance team needs help nailing down a bug
  • Product management needs an estimate for a chunk of work
  • The marketing team needs a change to some copy on the website
• Other engineers need help. Maybe someone would like a review of their code or bounce an idea for a solution off of somebody.
• There is an existing pile of bugs that need tending to. Even if your team is diligent about fixing bugs related to ongoing projects within the project timeframe, some bugs inevitably surface whose projects have long since finished. There’s no obvious time to dedicate to these bugs, so they go on a backlog. At some point the pile is so large that the team feels pressure to work on it.
• There is a critical issue in live/production systems affecting your customers.
• The continuous integration and/or delivery (CI/CD) system is broken.
• A new version of a product is ready but its release process is not fully automated. Someone needs to carry out the actual release. For instance, maybe someone needs to upload a mobile app to the relevant store portals, fill in release details, etc.

At Poll Everywhere, we have a specific role to address these concerns: Support Engineer. This is not a single person’s title, but rather a role that rotates among all the engineers. We try to put a different person in the role with each project cycle such that the responsibility is evenly distributed. While a person is the support engineer they are not expected to work on any features. Their daily responsibilities become the following, in priority order:

• Look into any production system issues. Obviously speed is of the essence here, and sometimes the support engineer is not the person who can solve the problem the quickest. That’s fine. But they can at least help, and learn about the issue and its resolution in the process.
• Look into any issue with CI/CD systems.
• Help others that need a hand. This can be folks from other disciplines or other engineers.
• Carry out any releases that need manual intervention.
• Chip away at the backlog of bugs.
• Work on any technical chores that don’t otherwise fit into the product cycle.

So far, our experience has been that the support engineer works on bugs most of the time, while being open to the other responsibilities preempting bug work if they come up.

At this point I’d like to address the elephant in the room. We’re not the first company to come up with the concept of a flexible resource to address unpredictable issues. I believe the reason such a role is not touted more widely is because it’s not glamorous. It’s easy to brag about something highly visible: “I built the onboarding flow!” as opposed to “I just fixed random bugs here and there”. It’s also easy to communicate the value of visible things to the company. When performance evaluations come around, it’s easy to agree that the new onboarding flow made the product better. But, it’s very hard to quantify how much faster that flow could’ve been delivered if the team working on it didn’t get distracted with other stuff. Because of its nebulous nature most organizations treat such a role as distinct from regular product engineering — almost as a chore. “You there. It’s your turn to take out the trash. Go clean stuff up while others get to make shiny new things.”

It’s just like chores around the house. In fact, the trash metaphor isn’t too far off. If you never took out your trash it would pile up and you’d eventually have a kitchen with no usable space and a really stinky house. Your pile of bugs is exactly like that. If you never tend to them they grow and become an ever increasing burden. On the other hand, there are things like painting your house. It increases the curb appeal. Maybe you even feel proud when you walk up to your house. So fresh and so clean. That’s the same feeling of releasing a new feature.

Notice that everybody wants a house without trash, but nobody wants to take trash out. Nobody intrinsically enjoys grabbing a smelly bag, making sure stuff doesn’t fall out while tying a knot at the top, and walking it outside when it’s freezing cold and pouring rain. But the mountains of trash you don’t have in the house (because you kept taking it out) increases your quality of life at least as much as the new paint job.

What if you could make taking out the trash an enjoyable activity? Perhaps this is where the house chore analogy breaks down. But support engineering can be a place where you have an impact on the work your team delivers, get other teams what they need, while also learning a lot yourself. Some of these benefits are:

• You are the clear point of contact for any other team that needs attention from engineering. This eliminates the overhead of trying to figure out who to talk to about each issue as it comes up.
• You have no expectation of shipping a regular project, hence you don’t resent these requests as distracting you from another, larger goal. Serving those requests is your goal during this time.
• You valiantly shield other engineers from interruptions, hence they can focus on their projects and ship quicker.
• If you’re otherwise a specialist, your time as support engineer builds breadth for you. Since you’ll work on a variety of issues, you’ll read and change code that you don’t work with often.
• Even if technical breadth is not an explicit goal, you’ll at least become more knowledgeable about your own product. As silly as that sounds, once a product gets somewhat large, we tend to not know how to use all of its features. Investigating issues outside your comfort zone will lead to getting familiar with new places.
• You’ll interact with and get to know colleagues you don’t work with every day.

If the above isn’t enough to convince people of the benefits of support engineering, some acknowledgement by the leadership team that this is a valuable position will help. When you assign someone to this spot treat it as if they’re simply on another project. Say it’s just another piston in the engine that makes the company go. If you truly feel the need to distinguish between regular product development and this work, at least don’t imply they’re doing chores. Pick a metaphor that ennobles it. Say it’s the whetstone to sharpen the axe. Or the oil change for your engine to remove the cruft. The medic on the battlefield. The mortar between the bricks. The greenhouse over the plant. Whatever your metaphor is for your endeavors, it’s supported by something. Call it out and celebrate its contribution. And with that shift in mindset, go build amazing things, consistently and quickly.