Early computers were described using mathematical notation and theoretical constructions, which were then translated by enterprising machinists into custom built calculation engines. The first computer programmers using these machines built applications directly in hardware by plugging together wires and sockets in varying configurations. Even with these short-comings, the value of general purpose computing machines was evident. As companies began to realize the utility of computing, they saw computing as a means for differentiating themselves from their competitors.
The widespread adoption of computing by companies helped to evolve early computers from custom built machines solving particular use cases to Bill Gate’s vision of “a PC in every home”. In 2003, Amazon evolved computing even further — from a product you can buy, to a service you can rent — and competition in cloud computing has pushed the state of the art further still. Today, cloud computing is a commodity. You can easily use cloud providers to rent virtual machines that are virtually indistinguishable from one another with just a few API calls.
This evolution of computing can be plotted on a timeline that begins with the notion of computing as an idea, and evolves into computing as a commodity running the cloud.
This evolution may seem unique to computers given the speed it happened at, but in fact, as components within a company’s value chain evolve, some competitors will adapt to use it. When a competitor sees the benefits of efficiency — and therefore faster creation of new sources of value — that a technology enables, it will create pressure on the competitor to adapt. As more and more companies adopt a technology and gain efficiency from it, the pressure on those who remain in the “old world” increases until it is overwhelming (which is why you see so many consultants today working on “digital transformation”).
This effect is known as Van Valen’s “Red Queen Hypothesis” and it is the reason why we don’t see your average company building its own generators from scratch to supply their own electricity.
In fact, if a technology provides enough value to companies that adopt it, those that don’t will be rendered extinct by it.
“The rabbit runs faster than the fox, because the rabbit is running for his life while the fox is only running for his dinner.”
— Aesop
We can take a similar evolutionary view of container management. When development of container management platforms began, container management and microservices were an idea conceived and developed at large technology companies like Google. The value in this idea was readily apparent to the broader industry, and companies sprang up trying to productize the technologies that enabled this style of development. At Workiva we leveraged some of these early technologies in the form of Docker, CoreOS, and Fleet, and in 2015 these technologies were used to build the foundation of what would become Harbour — our first container management platform. Unfortunately, these technologies were not fully formed and required us to apply our own custom glue code and development effort to patch over the rough spots and to build a working product. We ended up in a state of evolution with components somewhere between idea and product.
Fast forward a few years, and the industry began turning early container management technology demos into full-fledged products. This competition within the industry moved the evolution of container management from something that was custom built to serve a particular purpose, to a product you can rent or buy.
We chose to rent Amazon’s Elastic Container Service to handle container management, pushing our use of container management from something that was built on top of CoreOS and Fleet to a managed service with greater stability.
In the meantime, the competitive forces in industry have driven container management (and related concerns like service discovery and configuration management) further and further toward a commodity future.
At the time of writing, more than thirty cloud providers operate Kubernetes conformant clusters that developers can rent or buy — any company that wants to run a container management platform can do so, with little to no variation between them.
We can zoom out of this Infrastructure-centric view and see how it relates to the broader application ecosystem.
To make things concrete, let’s think about an email application. The user gets their value directly from sending an email and indirectly from using the email application to send it. Similarly, the email application gets value directly from an email history component, attachments, and search, and indirect value from those component’s backing data storage technologies. From the user’s perspective, as you move further down the value chain, components provide less and less direct value: the user doesn’t care that their draft email is stored in an SQL database like Aurora or a key-value store like Datastore — these components further down the value chain are invisible to the user.
Since development dollars are limited, as an organization we want to focus our efforts on things that are both highly visible and provide customer value. Typically, these are the custom built components of the value chain that serve as key differentiators for a company and that make the company valuable to users and the marketplace.
On the other end of this graph lies infrastructure, which is both invisible to the user and farther down the value chain. In general, commodities don’t provide a lot of value to users, and as a company we cannot use commodities to differentiate ourselves from the competition and win in the market. This is why companies don’t typically run their own electric grid.
It’s worth noting that the trends I’ve shown in this essay are driven by the industry, not by an individual or by a company. We cannot slow this evolution down. Nor can we prevent it. We can, however, make strategic decisions in response to it. Which brings me to the whole point of this pictoral history: strategy. By understanding the forces at play within the industry, we can begin to make choices on developing infrastructure for the next three to five years. In particular,
Given,
- Container management is a commodity
- Commodities do not add customer value
Then,
Replace infrastructure with a product you can rent or buy