Choose Boring Technology

I lifted the name of this post from another blog post that I think did an excellent job of addressing the subject of tool choice. Think of this as a cover song.

New needs constantly emerge during software development. Taking an app from being installed on a single server, to handling large amounts of traffic or data, to adding data mining functionality — every new business challenge brings technical changes.

Historically, different companies have had similar technology needs at the same time, and tools show up in order to meet these needs. Typically, one team encounters a problem and builds a tool, then releases it to the public where other teams begin using it.

Actually, this process tends to happen multiple times in parallel. Today, jQuery is a de-facto web framework, but Prototype, MooTools, Dojo, YUI, and many others were created around the same time to address the increasing usage of Javascript in web pages. Any new technical challenge is likely to have many tools that address it.

Choosing new tools is an inflection point when it comes to technical debt. When Javascript libraries first showed up, it would have been remarkably hard to know that jQuery would be the winner. Reliably picking the longest-lasting tool from a new set of options should be thought of as being so hard that we should assume it is impractical.

Picking the wrong tool has serious implications for the pace of work that an engineering team is able to complete. The more people using a piece of technology, the easier it is to learn about, work with, and hire people who have skills working with it. The harder it is to learn, the slower you can work with it, and you’ll have more failures, and need to make more compromises in hiring new engineers.

These are all serious challenges that need to be dealt with, yet I’ve just told you that it is effectively impossible to pick winners out of tools that exist to solve leading-edge problems. Perhaps a different mindset is needed.

Instead of attempting to pick new tools, another approach is to assume that some new tools will need to be brought into the technology stack over time, that some of the choices will not be optimal. In this case, the goal is to ensure that this is done in a manageable way.

An engineer by the name of Dan McKinley spent over 6 years at Etsy. Etsy is a company known for having a high-performing engineering department, and this type of performance comes not only from the code, but from the philosophies of the team.

Dan has talked about one philosophy that comes from this culture — the concept of “innovation tokens”. The idea is that as you choose new pieces of technology to incorporate into a project, some pieces of technology cost you an innovation token.

Every team has a natural number of innovation tokens. Dan estimates that you should start by getting three tokens, and only getting extra tokens when the tech stack is sufficiently stable.

The way a token gets spent is when a decision is made to use a new and interesting piece of technology, instead of something that is well-known.

An example of this is database choice. At the moment, there are many databases on the market: PostgreSQL, MongoDB, CouchDB, Redis, Riak, and many others. Many of them are interesting, but if you want a truly reliable piece of technology, you choose MySQL. MySQL was released in 1995 and has been battle-tested. Everyone knows how to use it. It rarely fails, and when it does, there are standard procedures for recovering from those failures. Configurations and performance are widely known and predictable.

MySQL is not perfect, however, and the other databases exist to solve certain problems better than they could be solved with MySQL. However, they lack the battle testing and broad knowledge base that MySQL has. They have risk, and that risk will eventually show up with poor performance requiring unexpected changes to the code, slowing down feature development.

To choose a non-MySQL database is to spend an innovation token. By spending that token, you acknowledge that risk exists, and that there is a natural amount of budget in the engineering plan to handle unexpected situations, and that you are putting a limit on that budget.

One interesting social aspect to this is that the more rapidly technology is changing, the more likely people will want to use new tools, and that’s when it requires holding strong and not chasing a new trend for technology’s sake. Making a budget is easy, sticking to it is hard, but that’s how you stay out of debt.