Exploring TigerBeetle, part 2

Viewstamped Replication (VSR) Consensus Protocol

Part of the ongoing Designing Ultra Large Scale Systems study group

Who am I?

Craig Rodrigues
Software Engineer in Silicon Valley

  • Interested in distributed systems
  • Interested in studying interesting topics in this space
  • Building a community of like-minded people where we can study together and learn

Why This Study Group?

I was inspired to start this study group after taking Chiradip Mandal's course:

Data Algorithms

Mastering this topic requires constant study and review of:

  • Papers
  • Algorithms
  • Cutting-edge implementations

Guiding Principles

We choose presentations that:

  1. No product pitches — substance over sales
  2. Adhere to computer science fundamentals — grounded in first principles
  3. Interesting claims must be verifiable — papers, code, benchmarks, not vibes
  4. Solve a specific problem that can help the industry at large — transferable lessons, not navel-gazing

Study Group — and Tech Evaluation & Critique

We are a study group: we learn together from papers, systems, and implementations.

We also act as a tech evaluation and critique group:

  • Examine claims carefully — architecture, performance, correctness
  • Ask hard questions: what works, what doesn’t, under what assumptions?
  • Separate marketing from engineering substance
  • Leave with a clearer judgment of when a technology is (or isn’t) a fit

Study Group Structure

  • Before the meeting: participants are encouraged to study the prerequisite material
  • During the event: first 30 minutes presentation, next ~20 minutes open discussion
  • Follow-up discussion on the Designing Ultra Large Scale Systems Discord

Pre-work Materials

  1. Part 1 slides: Exploring TigerBeetle – Debit/Credit Transactions
  2. QCon London '23 — A New Era for Database Design with TigerBeetle
  3. The Primeagen interview: TigerBeetle
  4. Jepsen analysis of TigerBeetle by Kyle Kingsbury
  5. Original Viewstamped Replication paper (1988)
  6. Viewstamped Replication Revisited (2012) — Barbara Liskov & James Cowling

TigerBeetle

  • A financial transactions database purpose-built for high-performance OLTP
  • Reimagines debit/credit as a first-class primitive (not just SQL queries and locks)
  • Aims for massive gains in correctness, safety, and speed at scale

TigerBeetle

  • Strict consistency with double-entry bookkeeping built-in
  • Batch thousands of transfers per query; eliminates traditional contention bottlenecks
  • Designed from first principles: durability, multi-cloud availability, extreme throughput

What is a Debit / Credit?

Debit − Account A $100 → $99  →  Credit + Account B $50 → $51

What We'll Cover

  • Deeper dive into TigerBeetle’s fault-tolerance design
  • Deeper dive into the Viewstamped Replication (VSR) consensus protocol

Recap: Part 1 Takeaways

Last session: debit/credit as a first-class primitive.

Slides: Presentations index

Main takeaways from Part 1:

  • No DDL — TigerBeetle has no schema language
  • Fixed schema — you cannot change the schema
  • No SQL / query language — not a general-purpose query engine
  • API only — access is via the client API (accounts, transfers, …)
  • Result: highly specialized and highly performant for debit/credit workloads

Prerequisite Readings — What I Took Away

TigerBeetle Workload Characteristics

From the TigerBeetle Architecture — Problem Statement:

TigerBeetle is a database for workloads that:

  1. Have contended access and parallelize/shard poorly (Amdahl’s Law)
  2. Consist mostly of writes
  3. Need very high throughput and moderately low latency
  4. Require strong consistency guarantees
  5. Demand very high levels of durability

Durability and the Art of Consensus

Watch: Joran Dirk Greef

A deeper take on consensus through the lens of durability

The Thesis

Availability is a function of durability —
and consensus is that function.

  • You don’t get availability “for free”
  • You get availability by preserving durable history across replicas
  • Consensus is the algorithm that turns local durability into cluster availability

From Physical → Logical

The talk reframes how we think about safety:

Physical Logical
Durability Bytes survive on disk / machine History is not lost for the cluster
Availability A node can serve The system can keep making progress
  • Consensus maps physical durabilitylogical durability
  • Then logical durability → availability under faults

Why Durability Comes First

  • Without D in ACID, A/C/I have nothing durable to guarantee
  • Disks fail; machines fail; fsync success ≠ forever-safe data
  • If you treat storage as perfect, consensus designs underestimate real faults
  • TigerBeetle’s view: design consensus assuming storage faults, then use replica redundancy to repair

Consensus as the Utility Function

  • Consensus converts durability into availability
  • Replicate durable prepares so a quorum can survive crashes / partitions
  • View change / primary failover: recover without losing committed durability
  • Optimal consensus ≈ maximizing availability given the durability you can actually keep

Viewstamped Replication — Two Papers

From the pre-work (slide 7):

  1. Viewstamped Replication (1988) — Oki & Liskov
    Original primary-copy method for highly available distributed systems

  2. Viewstamped Replication Revisited (2012) — Liskov & Cowling
    Same Oki protocol in spirit; clearer write-up + improved view-change subprotocol

TigerBeetle implements this VSR lineage (the revisited presentation).

What is a Viewstamp?

A viewstamp is still a real identifier: view + op

  • View — which primary/epoch of leadership you’re in
  • Op — position in that view’s log
  • Together they name a specific log entry under a specific primary
  • Operational vocabulary: views, primaries, ops / log, quorums, view change

VSR 1988 — Building Blocks

From Oki & Liskov (original paper):

  • Module — data + code on one node; unit of replication; talks via RPC
  • Module group / cohort — several replicas that behave as one module
  • Configuration — the set of cohorts; each has a mid / groupId
  • Primary — the active replica that handles client requests and coordinates the backups
  • Backups — passive; receive state from the primary

VSR 1988 — Views

  • View — cohorts that can communicate + which one is primary
    • Subset of the configuration; must contain a majority
    • Identified by a unique viewId
  • View change — group switches to a new view / new viewId (totally ordered)
    • Triggered when communication with a module fails
    • New view becomes active if a majority accepts; else views stay inactive
  • Active view — only here are transactions processed

VSR 1988 — Events & Viewstamps

  • Event — primary needs to tell backups something (e.g. prepare / commit)
  • Timestamp — unique id for an event (usually a counter)
  • Event record — FIFO info about the event (incl. timestamps), sent in order
  • Viewstamp = viewIdtimestamp — the original compound id
viewstamp { viewId, timestamp }
  • pset — per-transaction set of <groupId, viewstamp> entries

VSR Revisited (2012) — Clearer Vocabulary

Same protocol spirit; the write-up drops heavier 1988 module framing:

1988 2012
module / cohort replica in a replica group
event / event log on disk log (can be in memory + replicated)
timestamp + view → viewstamp view + op-number (still a viewstamp)
heavyweight module RPC framing clearer primary + backups service model

Fault Threshold f

  • f = maximum number of faulty nodes the system is designed to tolerate
  • Replica group size: at least 2f + 1
  • If f replicas are down, a quorum is still possible with f + 1 (a majority)
  • That majority quorum is how VSR keeps reliability and availability under faults

Example: f = 22f + 1 = 5 replicas

Even with f = 2 faulty nodes, a majority quorum of f + 1 = 3 can still make progress.

f = 2 faults tolerated 2f + 1 = 5 replicas f + 1 = 3 quorum (majority)
PrimaryR0
BackupR1
BackupR2
FaultyR3
FaultyR4
Primary (in quorum) Live backup (in quorum) Faulty / down (≤ f)

Why the Revisited Paper Matters

  • Same Oki protocol spirit: primary-copy replicated state machine
  • Clearer presentation for implementers
  • Cowling’s real delta: improved view-change subprotocol
  • Raft is in the same family — but missed that view-change improvement (more on this later)
  • Next: how TigerBeetle maps VSR onto prepares, WAL, and quorums

What VSR Revisited Gives You

Core idea (VSR revisited): a replicated state machine with a single primary

  • Primary orders client requests into a log (ops)
  • Backups accept/replicate prepares and ack
  • Commit when a replication quorum has durable prepares
  • On primary failure: view change elects a new primary and rebuilds a safe log suffix
  • Goal: no loss of committed history across failover

TigerBeetle’s VSR (Architecture)

From ARCHITECTURE.md + vsr.md:

  • Cluster of replicas; consensus keeps data files identical
  • Ground truth: hash-chained append-only log of prepares (batches of transfers)
  • Primary: accept → order → WAL append → replicate → wait for prepare_ok quorum → commit
  • Backups execute committed prepares in order (deterministic state machine)

Consensus Write Path (from vsr.md)

Normal protocol: client request, prepare chain, prepare_ok, reply

From Protocol: Normal — request → prepare → prepare_ok → reply

TigerBeetle VSR — Notable Choices

  • Flexible quorums (e.g. 6 replicas): replicate with 3, view-change with 4 (Flexible Paxos)
  • Protocol-aware recovery / NACKs — handle corrupt WAL entries safely during view change
  • Storage faults assumed — repair prepares/blocks from peers using checksums
  • Persist VSR state in the superblock — so TigerBeetle’s VSR runs on stable storage (no classic revisited-paper “Recovery Protocol”)
  • Pipelined replication: concurrent WAL write + replicate; commit needn’t wait on primary’s own write

Where is the superblock?

On each replica’s local disk, in that replica’s data file — not “in the network.”

  • Fixed location in the file (easy to find on startup)
  • Stored as 4 copies on disk for integrity
  • Data file zones include: WAL, grid (LSM), superblock

What the superblock does

  • Root pointer of durable state → reaches the rest of the grid via address + checksum
  • Holds local VSR state that must survive crash → VSR on stable storage
  • Runtime: current superblock often in memory; periodically flushed → checkpoint
  • After crash: read superblock from disk, then replay the WAL suffix after that checkpoint

Putting It Together

Layer Role
VSR Revisited Spec TigerBeetle implements (primary / views / quorums)
ARCHITECTURE.md How that maps to prepares, WAL, RSM, durability
Durability talk Why: availability = f(durability); consensus = that f

Why VSR? (Joran’s take)

From Durability and the Art of Consensus (~28:18):

  • VSR Revisited is one of the clearest papers for learning consensus
  • Raft (2 years later) is remarkably similar — same Oki-family protocol
  • Raft largely missed the improved view-change (deterministic next primary / Cowling’s nuance)

Advantage: Deterministic Next Primary

When the primary fails, VSR already knows almost certainly who is next
(Joran ~28:18):

primary(v)  =  vmodN\mathrm{primary}(v) \;=\; v \bmod N

  • (v) = view-number (increment each time you need a new primary)
  • (N) = number of replicas
  • No open-ended “election scramble” — failover stays elegant and fast

Advantage: Disk or Memory (+ recovery)

Other VSR ideas Joran highlights:

  • Can run on disk (what a real backup / DB system needs)
  • Or can run in memory (lighter deployments / teaching / testing shapes)
  • Extra recovery protocol when machines crash a lot (revisited paper’s recovery path)
    • TigerBeetle instead persists VSR state in the superblock — so TigerBeetle’s VSR runs on stable storage

Takeaway

  • Study VSR Revisited for the shared Oki/VSR protocol shape
  • The subtle part is view change — where Revisited improved, and Raft lagged
  • TigerBeetle’s VSR is that lineage, engineered for durability → availability

View-Change Protocol

VSR Revisited §4.2 — in depth

Paper talk walkthrough (~18:07)

When Does View Change Happen?

  • Backups monitor the primary (expect regular traffic)
  • Busy: prepare messages; idle: commit keep-alives
  • If a timeout expires without a commit → broadcast exit_view
  • Goal: switch to a new primary without losing committed ops
  • Quorum intersection still matters: enough replicas’ journals so committed ops are known

Deterministic New Primary

Same rule as before — everyone can compute the next primary locally:

primary(v)  =  vmodN\mathrm{primary}(v) \;=\; v \bmod N

  • Group membership is fixed within an epoch; (v) only goes up
  • No Raft-style election scramble — that is the “big idea” Raft largely missed

Paper names ↔ TigerBeetle names

TigerBeetle follows VSR (revisited) ideas but uses different command names
(vsr.md):

VSR Revisited paper TigerBeetle (vsr.md)
StartViewChange exit_view
DoViewChange join_view
StartView view (often with get_view)
PrepareOK prepare_ok
status view-change status view_change

Who sends exit_view?

Backups, when the primary goes quiet
(vsr.md).

  • No timely commit → broadcast exit_view
  • Means: “this view looks unhealthy — change views”

Who sends join_view?

Replicas that have entered the next view’s change.

  • After a view-change quorum of exit_view, send join_view (with log/header state)
  • The new primary is who collects that quorum

Who sends view?

Only the new primary.

  • Once it has a view-change quorum of join_view and a safe suffix → status normal
  • Broadcasts view: “view v is installed — follow me”

Put Together

  1. Detect — timeouts / missing commit
  2. Many replicas send exit_view
  3. Replicas send join_view — new primary gathers a view-change quorum
  4. New primary alone publishes view
  5. Repair / catch up as needed; cluster resumes normal writes

View-Change Flow (illustration)

TigerBeetle default: N = 6, view-change quorum 4. Primary R0 fails → new primary R1.

1
DetectR1–R5 timeout on R0
2
exit_viewbroadcast
3
join_view→ new primary R1
+ headers / suffix
4
viewR1 installs view;
others catch up
Was primaryR0
Primary · in quorumR1
In quorumR2
In quorumR3
In quorumR4
Live (extra)R5

Highlighted R1–R4 = view-change quorum of 4. R5 can be live but is not required for that minimum.

After view

  • Backups install suffix/checkpoint from the primary’s view
  • Repair missing headers/prepares (get_headers / get_prepare) as needed
  • Send prepare_ok for uncommitted ops still in the log
  • Execute committed ops not yet applied; status back to normal

Three Different Questions

TigerBeetle doesn’t use one “majority” for everything
(vsr.md — Quorums):

Quorum Question it answers
Replication How many replicas must confirm they stored a prepare before the cluster commits it?
View-change How many replicas must participate to finish a view change?
Nack How many “I never accepted that op” signals to safely drop it?

Different decisions → different thresholds.

Nack — a tiny story

  1. Primary starts prepare C; only one replica has anything about it; client never got a reply
  2. Primary dies; disks are messy
  3. New primary asks during view change: who has C?
  4. If a nack quorum says “never accepted C,” truncate it
  5. If C might have committed, you must not truncate — repair instead

This is the idea behind
Protocol-Aware Recovery (FAST ’18).

How to Read the Quorum Table

Defaults from vsr.md — pick a cluster size (N), then read down the column:

Replica count (N) 1 2 3 4 5 6
Replication 1 2 2 2 3 3
View-change 1 2 2 3 3 4
Nack 1 1 2 3 3 4

Example (N = 6) (recommended): commit with 3, change views with 4.

What That Means for (N = 6)

Size Meaning
Replication 3 Any 3 of 6 durable prepare_oks → commit
View-change 4 Need 4 of 6 for exit_view / join_view quorums
Nack 4 Need 4 nacks to truncate a maybe-uncommitted op

You can commit more easily (only 3) than you can change leaders (need 4). That’s intentional.

Why Not One Number ((f + 1))?

Classic VSR/Paxos: (N = 2f + 1), one majority (f + 1) for everything.

TigerBeetle uses flexible quorums (Flexible Paxos):

  • Smaller replication quorum → better availability for commits
  • Larger view-change quorum → safer when choosing a primary / log suffix
  • Safety: any replication-sized set and any view-change-sized set must intersect
    • So a new primary still learns every prepare that could have committed

Where View-Change Quorum Shows Up

In the protocol (vsr.md):

  1. View-change quorum of exit_view → enter view_change, bump view, send join_view
  2. View-change quorum of join_view → new primary installs suffix, repairs, broadcasts view
  3. Separately: nack quorum can truncate an op that never committed

This table is about how many must participate — not a simple “how many can fail” chart.

Very Cool Demo

https://sim.tigerbeetle.com/

Acknowledgments

Thank you:

Acknowledgments (AI)

These slides were prepared with the help of Grok AI.

Upcoming Meetings

  • September: Antithesis: First-Principles Approaches to Testing Ultra-Large State Spaces
  • October: ChronoQuorum: A Consensus Protocol for Unmanned Mission Vehicles

Stay in touch!