Database

Ch-ch-changes!

• At TigerBeetle, we approach performance engineering by first getting the high-level performance architecture right, as that's what can enable multiple orders of magnitude improvement. Then, we employ low-level techniques like static memory allocation, amortization, zero-deserialization, memory locality, etc. to extract additional performance. Since these optimizations compound, we believe in investing in all, big or small. Using this methodology, last year, we tamed TigerBeetle’s tail latency and broke the 600K TPS record!
  • Sorting the memory tables with LSD-Radix – a stable, linear-time sorting algorithm, delivered a ~23% throughput increase over the standard library sort, as it uses fewer CPU cycles, executes fewer instructions, and incurs fewer branch misses, resulting in higher instruction-level parallelism (IPC).
  • The k-way merge iterator, responsible for joining streams during compaction and queries, also received performance improvements. It now uses a tournament tree (aka tree of losers) instead of a heap, skips unnecessary sorts, and avoids binary search where possible by leveraging known key ranges.
  • To keep P100 latency predictably low, TigerBeetle employs an incremental sort strategy, amortizing the cost of sorting the entire mutable table by doing only small portions at a time and performing the final full sort at the end. We removed the full sort at the end, and now use the optimized k-way merge iterator to join all sorted runs together, improving latency and query times.
• TigerBeetle’s consensus protocol consists of two recovery sub-protocols that help lagging replicas catch up: Write-Ahead Log (WAL) Repair and State Synchronization. WAL repair entails repairing missing or corrupted prepares over the network. However, the WAL is a finite, on-disk ring buffer wherein old prepares are overwritten after they’re checkpointed. Consequently, replicas may fall so far behind that WAL repair isn’t possible at all. In that case, replicas must perform State Synchronization, which entails fetching the latest checkpoint and using it to repair missing data blocks over the network. Last year, we made various performance and reliability improvements to our recovery and replication protocols!
  • We made the repair, replication, and request protocols more robust to changing network topology and latencies (addressing two fallacies of distributed computing) by implementing adaptive repair, adaptive replication routing, and request hedging. Replicas now maintain an exponentially weighted moving average of the repair latency they observe from each remote replica, and prioritize remote replicas that have previously responded with low repair latency. Additionally, replicas track commit latencies to select an optimal replication topology. Finally, the TigerBeetle client now sends a request to two replicas – the primary and a randomly selected backup, to route around network partitions!
  • State sync now requests blocks more aggressively, driven by the rate of incoming requests rather than an infrequent timeout. Progress is ratcheted across multiple checkpoint targets, avoiding the earlier behavior of restarting from the beginning on each checkpoint change, which caused duplicate work.
  • We made TigerBeetle more resilient to metastability failures! Typically, a TigerBeetle cluster makes progress at the pace of the fastest quorum of replicas. The other replicas may lag a bit, but eventually catch up with the rest of the cluster during quieter periods. Before this enhancement, if the cluster were under high load, lagging replicas would lag more and more, fall into state sync, and ultimately enter a potentially perpetual state of lag. Now, the primary applies backpressure to the client, which in turn backpressures the application, allowing lagging replicas to catch up with the rest of the cluster.
• As a financial transactions database, TigerBeetle is designed to serve as the system of record and source of truth for transactional data, since it excels at handling OLTP workloads, which necessarily have high “power law” contention. However, for the entire application to function, other databases must coexist, sometimes handling complementary data (e.g., OLGP databases, document and object storage) or mirroring different views of the same transactions committed to TigerBeetle (e.g., OLAP databases, data warehouses, legacy systems, and event-driven workflows).
  
  One pattern for interconnecting different systems is Change Data Capture (CDC). A technique where the source database tracks and streams only the changes (such as inserts, updates, and deletes) as they happen. Since financial transactions are immutable and changes are append-only (with no updates or deletions), CDC is particularly efficient for TigerBeetle to process. Last year, we landed CDC connectors targeting RabbitMQ and RedPanda!
• In the past, we dealt with small batches by automatically aggregating events from concurrent calls at the client side and submitting them as one larger batch. Although this worked, it required our client to be aware of the content being submitted and to do extra work to make it appear to the application as if it were from different requests. Worse, it didn’t apply to all TigerBeetle operations.
  
  We took TigerBeetle’s automatic batching one step further and introduced the concept of multi-batching: an encoding on top of VSR messages that allows clients to submit arrays of batches, encoding the payloads of concurrent calls into a single physical VSR request. The most significant benefit is that this now works for any kind of operation, including queries and lookups, optimizing the TigerBeetle client for highly concurrent environments.

Community!

The relationships we’ve built with many of you together—over years, across time zones, often involving gelato—mean the world to us. From 6 a.m. walks to beat jet lag, a call across three continents, and pickup hoops to thirteen city stops on our 1000x World Tour just last week—we’re grateful for on- and offline memories made.

We’re also grateful for your year-round contributions—to PRs and docs, by catching a broken link, answering #questions on Slack. An outstanding selection from this year’s contributions:

• Guilherme spotted a bug in the Go client’s result size; Alex Sharov found and reported issues in the Go client using the Go linter.
• Raghunandan enhanced the REPL with shortcuts!
• Nikita spotted that our Node.js client samples use equal, which has been deprecated since node v9, and corrected all occurrences to use strictEqual.
• Kirill changed the REPL to use StaticAllocator.
• Ivo spotted and fixed a bug in our Python client’s sample code, while Tamas added type annotations, making it compliant with mypy in "strict" mode.
• rbino and alvinyan-bond took CDC for a spin and reported issues. rbino also contributed a fix to our build.zig script, that would allow him to compile his TigerBeetle Elixir client using Zig 0.14.1, spotted a subtle performance bug in the client's random ID generator algorithm, and delivered a talk in Milan on the 1000x tour!
• Mohamed fixed the command to build TigerBeetle clients in HACKING.md.

"Database television" (DTV)

DTV cooked steadily this year: 40 new episodes of IronBeetle with matklad, our Rust client live-coded by Brian, and from Systems Distributed ‘25—13 talks from some fun-loving speakers!

We also enjoyed speaking in various shapes and places on everything from assertions, bug-bashing and taming TigerBeetle’s tail to what makes ACID, consensus, and TigerStyle:

• Rocket Science of Simulation Testing! - matklad at HYTRADBOI, Feb 28
• How to Write your own Deterministic Simulator - Joran and Dominik, Mar 5
• ThePrimeagen x Lex Fridman, on Asserts - Mar 22
• Two-Part Interview (in Russian) - matklad on Staff Podcast, Mar 26 and Apr 23
• The 1000x Faster Financial Transactions Database - Joran on Changelog Interviews, Apr 2
• Deterministic Upgrades with Multiversion Binaries - Federico Massimiliano Lorenzi at JOTB25, Jun 26
• Zig and Rust in Production (ft. matklad) - matklad on The Primeagen, Jul 2
• Redpanda and TigerBeetle: Real-time Financial Transaction Streaming - Joran and Alex Gallego, Sep 24
• TigerBeetle at the New York Stock Exchange - Joran with John Furrier, Sep 25
• Fixing Five "two-year" Bugs per Day - Joran on the Bug Bash Podcast, Oct 1
• The Tale of Taming TigerBeetle's Tale - Tobi at P99COnf, Oct 23
• A Million Transactions per second: Building TigerBeetle with Joran Greef - Joran on The Database School Podcast, Oct 23
• What is TigerStyle? Principles behind TigerBeetle ft. Joran & TigerStyle with matklad Vol. 2 (Systems Engineering) - The Geek Narrator, twice in November
• Mathematics of Consensus — Accidental Lecture - matklad with Tobi, Nov 22

Looking back

In case you missed any of these posts by TigerBeetle and friends, some holiday (re)reading:

A Descent Into the Vörtex, Oskar Wickström, Feb 13
We’re huge fans of Deterministic Simulation Testing (DST), but earlier this year we added an intentionally non-deterministic testing harness: the Vörtex! But, why?!

Why We Designed TigerBeetle's Docs from Scratch. Fabio, Fabian, matklad, Feb 27
In February, we rebuilt TigerBeetle’s docs site from scratch—to apply TigerStyle and give users the fastest possible experience.

Understanding TigerBeetle (Part 1), Swana Simran, Mar 6
We enjoyed Swana’s overview of why the world needs TigerBeetle!

Swarm Testing Data Structures, matklad, Apr 23
Back in April, we discovered a cute little pattern when refactoring TigerBeetle’s intrusive queue, using Zig’s comptime reflection for exhaustively testing a data structure’s public API.

Asserting Implications, matklad, May 26
When using assertions heavily, a common pattern is asserting an implication. A short post, to suggest: if (a) assert(b); instead of: assert(!a or b);.

How to add rate limiting to your API using TigerBeetle, May 30
A detailed solution from Mirceau for adding rate limiting to a Spring Boot API application, using TigerBeetle for the bookkeeping portion.

Boredom Over Beauty: Why Code Quality is Code Security, Jun 3
Some of the most devastating vulnerabilities stem from complexity. John Saigle’s post, referencing TigerStyle, explains why predictable, well-formed code is the foundation of security. Little did John know that Joran’s work before TigerBeetle was in fact in security, directly informing TigerStyle.

TigerBeetle was Jepsen'd! Jun 6
Jepsen found one correctness bug in what appeared to be a well-fuzzed component of the TigerBeetle code: the query engine. Read the report, our companion post, and one written by Dominik Tornow—or listen to Kyle talk about it. This was altogether fun, and a milestone.

Push Ifs Up and Fors Down - A Broader Perspective, Jul 19
One of TigerStyle’s recommendations is to centralize control flow. Debasish expounded on this, surmising that “early projections and deferred joins can be interpreted as reordering morphisms in a category to optimize composition.”

Code Review Can Be Better, matklad, Aug 4
A lot of people are unsatisfied with GitHub’s code review process! Over a period of a few months, we tried (and failed, yet learned) to create our own code reviewing tool/process.

Funding news for Profs. Ram Alagappan and Aishwarya Ganesan, Aug 15
In 2018, “Protocol-Aware Recovery” set the standard for Durability in ACID, making Profs. Ram Alagappan and Aishwarya Ganesan giants in the field. We’re grateful to stand on their shoulders and support their ongoing work.

Why TigerBeetle is the most interesting database in the world, Sep 30
“By many measures it’s safe to say that TigerBeetle is the most interesting database in the world. Like Costanza in Seinfeld, they seem to do the opposite of everyone else.” - wrote Justin Gage of Amplify Partners, an investor in TigerBeetle.

Tracking Time Without Clock, matklad, Oct 21
Time is at the heart of essential complexity. While virtualizing the entire clock is a universal approach, oftentimes a now or tick pattern is all you need.

Synadia and TigerBeetle Pledge $512,000 to the Zig Software Foundation, Joran, Oct 25
Together with Synadia, we made a pledge to the Zig Software Foundation, over the next two years, in support of the language, leadership, and communities building the future of simpler systems software.

The Write Last, Read First Rule, Nov 6
How to maintain consistency in the absence of transactions, how to reason about correctness when intermediate states are externalized, and how to recover from partial failures. A special guest post by Dominik.

Building a financial accounting workflow with TigerBeetle, Nov 18
Vivek Dehariya explored how TigerBeetle fits into billing/accounting systems by taking a simple example of an accounting workflow for a SaaS in multi-tenant setup across currencies.

Mathematics of Consensus — Accidental Lecture, matklad & Tobi, Nov 22
An impromptu ~2 hour lecture explaining the mathematics behind consensus algorithms, using a Go Pro fisheye, by matklad (principled, passionate, prolific!).

A Tale of Four Fuzzers, matklad, Nov 28
Some time ago we overhauled TigerBeetle’s routing algorithm to better handle varying network topologies in a cluster and ended up adding four very different new fuzzers to the system!

io_uring for High-Performance DBMSs: When and How to Use It, Dec 4
How modern database systems can leverage the Linux io_uring interface for efficient, low-overhead I/O, co-authored by Matthias Jasny, Viktor Leis, Tobi Ziegler and Carsten Binning

The TigerVerse

Thank you!