Hi there 👋

Welcome to the personal blog of Kevin Sookocheff. I love to teach, speak, and write about distributed systems, cloud computing, architecture, and systems engineering.

Routing Users to a Login Page Based on Geolocation

A common pattern when scaling a SaaS application is to deploy your stack to multiple regions across the globe. Deploying to multiple regions helps distribute the application and data closer to the user, and isolates infrastructure in case of an outage in a single region. If your application is deployed in this configuration, you have the new problem of routing users to the correct region based on their tenancy or geography....

December 12, 2023 · 5 min · Kevin Sookocheff

What is eBPF and Why is it Important?

A short tagline from the book Learning eBPF describes eBPF as a revolutionary kernel technology that allows developers to write custom code that can be loaded into the kernel dynamically, changing the way the kernel behaves. The key word in this phrase is dynamically. eBPF allows you to write custom code that changes the way the kernel behaves without having to implement a kernel module or integrate your code directly into the kernel....

December 4, 2023 · 5 min · Kevin Sookocheff

Disaster Recovery with DynamoDB Global Tables

DynamoDB provides built-in support for cross regional data replication using a solution AWS calls global tables. This article shows how to build and run an application in Kubernetes that uses global tables to replicate data between regions. In the event of a regional disaster, a secondary Kubernetes cluster in a secondary region has all the data from DynamoDB replicated locally to continue operation. How global tables work A DynamoDB global table is a set of multiple replica tables....

November 20, 2023 · 11 min · Kevin Sookocheff

Supporting disaster recovery with a multi-region EKS deployment

Disaster events are one of the biggest challenges that a software organization can face. Natural disasters like earthquakes or floods, technical failures such as power or network loss, and human actions such as unauthorized attacks can disable an entire fleet of systems, leading to complete failure for a business. To deal with disaster scenarios requires a proactive approach to prepare and recover from failure. One of the key benefits of running in the cloud is how easy it is to run workloads in multiple regions....

November 9, 2023 · 11 min · Kevin Sookocheff

FSBench: A filesystem benchmarking utility

Check out the project on Github. I recently published a new project for repeatable filesystem benchmarking. This code is based on the benchmarks available in the AWS Mountpoint-s3 project and the Flexible I/O Tester, made into a more general purpose utility for benchmarking arbitrary filesystems. The benchmark consists of a mix of read and write workloads, each run for ten iterations. The average result of the ten iterations is reported as the final result of the benchmark....

September 22, 2023 · 2 min · Kevin Sookocheff

Benchmarking AWS CSI Drivers

AWS provides four different storage options for your Kubernetes cluster: EBS, EFS, FSx for Lustre, and Amazon File Cache. Each of these CSI drivers has different performance characteristics, depending on your workload. This post quantifies those performance differences using the flexible I/O tester FIO. Note: For an overview of the different CSI options available on AWS, see Picking the right AWS CSI driver for your Kubernetes application. Before we start, please note that these results come with multiple caveats, and you should absolutely test performance on your own workloads before making any final decisions....

September 19, 2023 · 7 min · Kevin Sookocheff

Picking the right AWS CSI driver for your Kubernetes application

Dealing with storage is a core challenge when running complex applications on Kubernetes. While many applications operate just fine using a cloud database or blob storage, some applications have performance or design requirements requiring local storage. Note: For performance benchmarks see [Benchmarking AWS CSI Drivers](<{{ relref “/post/kubernetes/benchmarking-aws-csi-drivers” }}). When this is the case, developers and cluster operators rely on Container Storage Interface (CSI) implementations to provide local storage for Pods. When running on the AWS cloud, no less than four CSI providers are available for us to use: Elastic Block Storage, Elastic File System, FSx for Lustre, and AWS File Cache....

September 12, 2023 · 12 min · Kevin Sookocheff

Leveraging eBPF in the Kubernetes Networking Model

eBPF is a relatively new extension of the Linux kernel that can run sandboxed programs in a privileged context. It is used to safely and efficiently extend the capabilities of the kernel at runtime without requiring changes to kernel source code or the loading of kernel modules. Because of eBPFs tight integration with the networking stack at the kernel level, it is seeing adoption in networking applications. This includes Kubernetes networking through eBPF implementations of the Kubnernetes networking stack like Cilium....

August 21, 2023 · 4 min · Kevin Sookocheff

Trust your users — they’re usually right

I was recently introduced to a simple article on system design: Users are almost always right. In it, a simple rule is stated: When the users keep doing it wrong, the users are right and your system is wrong. This rule reminded me of the concept of desire paths: unplanned trails created by human or animal traffic that usually represent the shortest or easiest route between two places. Desire paths in an urban setting often run counter to what a planner might suggest, leading to a natural conflict between the pedestrian and the planner....

May 29, 2023 · 3 min · Kevin Sookocheff

Generating Large Test Files

I was recently testing file upload performance, and needed several large files of different sizes to test with. To make the math easier, it was helpful if I had files with round numbered sizes like 10MB, 20MB, or 100MB. After searching around for files of the right size, it turns out the easiest solution is to generate one yourself using the Linux command line. Depending on your needs, you can use two different methods of generating files using some simple commands....

May 24, 2023 · 2 min · Kevin Sookocheff