HybridTime introduces a hybrid between physical and logical clocks that can be used to implement globally consistent database snapshots. HybridTime is one implementation of the general concept of Hybrid Logical Clocks that combine the logical clock semantics of Lamport and Vector clocks with a physical representation of time such as Spanner’s TrueTime. HybridTime follows similar update semantics as Lamport and Vector Clocks, where each node in the system updates their internal clock in response to events received. [Read More]


TrueTime is Google’s solution to providing to globally consistent timestamps to determine ordering of events. Originally developed to support the Spanner distributed database, TrueTime is a clock implementation that depends on two key factors: well engineered and accurate GPS and atomic clocks and the representation of time as an interval of uncertainty. Representing TrueTime TrueTime explicitly represents each timestamp an interval that includes bounded time uncertainty. TrueTime represents time as an interval of the type TTinterval, which includes two timestamps for the beginning and the end of the interval. [Read More]

Vector Clocks

Vector clocks are an extension of Lamport Clocks designed to determine all correct orderings of events in a distributed system. Whereas Lamport Clocks yield one of the many possible event orderings, this does not for certain classes of problems that require knowledge of global program state such as reversing the order of execution to recover from errors or rollback. In such cases, it is more useful to have access to the entire set of orderings that are causally consistent at any moment in time. [Read More]

Lamport Clocks

Ordering events in a distributed system

Computers generally track the current time using a quartz crystal that oscillates at a known frequency. This frequency is used to advance the local clock at pre-defined intervals. Quartz oscillates with enough stability that it can be used to maintain time within a few milliseconds of accuracy per day. Over time however, these imperfections can accumulate, making the clock inaccurate. To fix these inaccuracies, computers typically run a background process like NTP to synchronize the local clock with servers that are known to be accurate. [Read More]

How Does NTP Work?

The Network Time Protocol (NTP) is a system for synchronizing the clocks of hosts and clients across the Internet. NTP is a protocol intended to synchronize all computers participating in the network to within a few milliseconds of Coordinated Universal Time (UTC). The core of the protocol is NTP’s clock discipline algorithm that adjusts the local computer’s clock time and tick frequency in response to an external source — such as another trusted NTP server, a radio or satellite receiver, or a telephone modem. [Read More]

Local Kubernetes Development with microk8s

Still in search of a Kubernetes development environment I can run on my development machine that somewhat replicates, I was recently introduced to both the multipass and microk8s projects from Canonical. multipass is a prerequisite for microk8s, so let’s start there. With multipass, you can easily launch and run Ubuntu virtual machines with a single command. 1 2 3 ❯ multipass launch --name foo Launched: foo With a virtual machine in hand, you can run execute arbitrary commands on the machine using the exec subcommand. [Read More]

Why Foxes Make Better Decisions Than Hedgehogs

The ancient Greek poet Archilochus wrote that “the fox knows many things, but the hedgehog knows one big thing.” In social psychology, these terms have been used as a shorthand for describing two cognitive styles: foxes, who are comfortable with nuance, can live with contradictions, and have different strategies for solving different problems, and hedgehogs, who focus on the big picture, reduce every problem to one organizing principle, and solve problems through that singular lens. [Read More]

Connecting Technology to the Needs of the Business

All healthy technology discussions should begin with business goals and use those goals as a reasonable set of guidelines to focus technology investment decisions. These business goals are best articulated from a deep understanding of what the company, the product team, or the marketing team want to accomplish. In a business-first model, technology is forced to balance the desires for technical effectiveness and efficiency with the operational needs of the business. [Read More]

The False Dichotomy of Design-First and Code-First API Development

In creating a development workflow for releasing HTTP+JSON APIs, many teams settle on the API description and documentation standard OpenAPI. A natural conversation that takes place after settling on OpenAPI is whether or not you should develop your API using a “design-first” method or a “code-first” method. I believe that narrowing the discussion to these two extremes overly simplifies the discussion. Instead, I prefer to talk about API development as a continuum of development process that revolve around the prominence of the API description document in the process. [Read More]
rest  api  openapi 

The Most Important Criteria for Choosing a Programming Language

Use a Language Your Company Can Support

One of the recurring themes of any technology discussion is programming language. It doesn’t take much effort to find blog posts with dramatic headlines (and even more dramatic comments) about how shipping a new project with Haskell or Clojure or Elm improved someones job, marriage, and life. These success stories are posted by raving fans that have nothing but the best to say about their language of choice. A common thread running through these posts is that they are typically tied to building out new, greenfield projects. [Read More]