Why Systems Work So Well

In the book “Thinking in Systems”, Donella Meadows dedicates an entire chapter to explaining why functioning systems seem to work so well. In it, she recognizes three characteristics: resilience, self-organization, and hierarchy. Resilience We can use the standard definition from the Oxford English dictionary to describe resilience: re·sil·ience /rəˈzilyəns/ noun the capacity to recover quickly from difficulties; toughness. “the often remarkable resilience of so many British institutions” the ability of a substance or object to spring back into shape; elasticity. [Read More]

Java For The Experienced Beginner

Java was the first programming language I was taught at University, and the language I used for the first decade of my career. It continues to be a reliable companion throughout my software development career. Unfortunately, not having developed with Java professionally for several years, I’ve found there are many aspects of the modern Java language that I’m simply not familiar with. To rectify this, I’ve collected the major improvements to the language beginning with Java 8, combined with a short explanation of how they work and how to use them. [Read More]

Behaviour Parameterization

One of the core features of modern Java is lambda expressions. Introduced in Java 8, lambdas provide concise syntax allowing the deferred execution of a block of code. Put a different way, lambdas allow us to pass behaviour as a method parameter. When the method executes, the lambda expression is run. This capability is often referred to as behaviour parameterization. Behaviour parameterization can be achieved in a number of ways, of which lambda expressions are usually the most convenient, and they are definitely the most concise. [Read More]
java  design 

What complex systems can teach us about building software

As a software system scales it becomes sufficiently large that the number of working parts, coupled with the number of working programmers making changes on it, makes the behaviour of the system extremely difficult to reason about. This complexity is exacerbated by the transition of many organizations towards a microservice architecture, as exemplified by the so-called “death star” architecture, where each point in the circumference of the circle represents a microservice and the lines between services represent their interactions. [Read More]

Hybrid Logical Clocks

The theory of distributed systems promoted the use of logical clocks by introducing the idea of causality tracking as an abstraction for reasoning about concurrency between events in the system. In practice, a lot of systems continue to operate using physical time, which presents difficulties due to clock synchronization drift. In an effort to bridge the gap between physical and logical time, HybridTime combines both logical and physical times in one system. [Read More]
time  hlc 


HybridTime introduces a hybrid between physical and logical clocks that can be used to implement globally consistent database snapshots. HybridTime is one implementation of the general concept of Hybrid Logical Clocks that combine the logical clock semantics of Lamport and Vector clocks with a physical representation of time such as Spanner’s TrueTime. HybridTime follows similar update semantics as Lamport and Vector Clocks, where each node in the system updates their internal clock in response to events received. [Read More]


TrueTime is Google’s solution to providing to globally consistent timestamps to determine ordering of events. Originally developed to support the Spanner distributed database, TrueTime is a clock implementation that depends on two key factors: well engineered and accurate GPS and atomic clocks and the representation of time as an interval of uncertainty. Representing TrueTime TrueTime explicitly represents each timestamp an interval that includes bounded time uncertainty. TrueTime represents time as an interval of the type TTinterval, which includes two timestamps for the beginning and the end of the interval. [Read More]

Vector Clocks

Vector clocks are an extension of Lamport Clocks designed to determine all correct orderings of events in a distributed system. Whereas Lamport Clocks yield one of the many possible event orderings, this does not for certain classes of problems that require knowledge of global program state such as reversing the order of execution to recover from errors or rollback. In such cases, it is more useful to have access to the entire set of orderings that are causally consistent at any moment in time. [Read More]

Lamport Clocks

Ordering events in a distributed system

Computers generally track the current time using a quartz crystal that oscillates at a known frequency. This frequency is used to advance the local clock at pre-defined intervals. Quartz oscillates with enough stability that it can be used to maintain time within a few milliseconds of accuracy per day. Over time however, these imperfections can accumulate, making the clock inaccurate. To fix these inaccuracies, computers typically run a background process like NTP to synchronize the local clock with servers that are known to be accurate. [Read More]

How Does NTP Work?

The Network Time Protocol (NTP) is a system for synchronizing the clocks of hosts and clients across the Internet. NTP is a protocol intended to synchronize all computers participating in the network to within a few milliseconds of Coordinated Universal Time (UTC). The core of the protocol is NTP’s clock discipline algorithm that adjusts the local computer’s clock time and tick frequency in response to an external source — such as another trusted NTP server, a radio or satellite receiver, or a telephone modem. [Read More]