SPICEDB AT SCALE

Google-Scale Authorization: 1 Million Requests per Second on SpiceDB

Can a dedicated authorization system keep up when every request in every product asks it a question? A series of load tests set out to answer that, ending at one million requests per second against one hundred billion relationships.

Google's Zanzibar paper describes an authorization system that stores trillions of access control lists and serves millions of permission checks per second for products such as Calendar, Drive, Maps, Photos and YouTube. SpiceDB is built on the same ideas, and a natural question from teams evaluating it is whether that scale is real or just inherited reputation. A prospective customer asked exactly that, and challenged the team to prove it.

1,000,000requests per second, including 1% writes
100 billionrelationships stored
5.76 msP95 latency on CheckPermission at peak
99.9%+success rate required in every test

The challenge

The goal was simple to state and hard to meet: show that SpiceDB delivers consistent latency as both traffic and data grow by orders of magnitude. The team planned thirteen tests, starting at ten million relationships and one thousand requests per second, and working up to one hundred billion relationships and one million requests per second.

Every test ran for ten minutes and had to meet the same service-level objectives:

Two secondary goals mattered just as much to teams planning a rollout: understanding how compute grows with traffic and data, and learning how long it takes to load existing permission data into a new system.

A realistic workload, not a synthetic one

Load tests are only as useful as their resemblance to production. Rather than an artificial schema, the tests used a permission model provided by the prospective customer, describing a social networking product:

That model exercises most of what makes authorization hard: unions, intersections and exclusions, three levels of nesting, and wildcards that open content to everyone. It is a meaningfully demanding schema, not a single role check.

Shaping the data like real life

Real social graphs are lopsided. A small share of accounts produces most of the content and attracts most of the followers. To reflect that, the test data followed a Pareto distribution, where roughly 20% of users generate 80% of the activity. That matters for a graph-based system: very wide relationships and deep nesting are where permission checks get expensive, so a realistic shape is a much harder test than evenly spread data.

Shaping the traffic

The request mix mirrored the customer's expectations: 60% of checks asked whether someone could see a piece of content, 30% whether they could see another person's profile, and 10% whether they could see a conversation. On top of that, 1% of all requests were writes. That is several times the peak write share the Zanzibar paper reports, which makes the test more conservative, not less. Checks used SpiceDB's default low-latency consistency mode, and requests were drawn from a sampled pool of simulated daily active users so that caching behaved as it would in production.

How it was deployed

The tests ran on AuthZed Dedicated, the fully managed, single-tenant way to run SpiceDB, on AWS, with CockroachDB as the datastore. A few deployment choices made a noticeable difference:

Infrastructure grew with each test. At the very top, serving one million requests per second against one hundred billion relationships took 21 SpiceDB nodes with 32 virtual CPUs each, alongside a six-node CockroachDB cluster.

The results

Every test met its targets, and latency stayed remarkably steady as traffic and data grew. Here is how the largest dataset, one hundred billion relationships, performed at each traffic level:

Requests per secondCheck P95Check P50Write P95Cache hit rate
1,00017.1 ms8.05 ms17.9 ms0%
10,00010.0 ms3.44 ms37.0 ms65.1%
100,0005.86 ms3.08 ms49.3 ms89.2%
1,000,0005.76 ms3.03 ms48.3 ms95.9%

Results from the final round of tests on one hundred billion relationships. Write latency stayed well inside its 100 ms target throughout.

Checks got faster as traffic grew. SpiceDB breaks every permission check into smaller sub-problems and caches their answers. The more traffic it serves, the more those answers are reused: the cache hit rate climbed from zero at one thousand requests per second to almost 96% at one million, and latency fell with it.

The same pattern held at one billion relationships, where one million requests per second ran at a 5.89 ms P95. Data size, in other words, was not the limiting factor; the shape of the data and the traffic were.

What drives authorization performance

Raw numbers are less useful than understanding what moves them. The tests made the main levers clear, and they are worth keeping in mind for any capacity plan:

These factors interact. A higher cache hit rate lowers SpiceDB's CPU use and the load on the database, which frees capacity for more requests. Planning around that interplay is more reliable than extrapolating from a single benchmark.

Getting existing data in

Most teams adopting a permissions database already have years of access data to migrate, so import time is part of any rollout plan. In these tests, one billion relationships loaded in about two hours and forty minutes, and one hundred billion in about five days. Once indexed, the largest dataset used about 8 TB including three-way replication.

That work led directly to native bulk import and export in the SpiceDB API, which speeds up initial loads considerably. A rerun of the largest test on ARM-based AWS Graviton instances also delivered about 20% more throughput.

Improvements that came out of the tests

Pushing a system this hard surfaces bottlenecks that ordinary traffic never reaches. Several fixes found during the tests became part of SpiceDB itself, including better balancing of database connections, spreading cache refreshes over time to avoid sudden bursts of database traffic, steadier routing of work between SpiceDB nodes, and more even distribution of that work across the cluster.

Lessons for running your own scale test

What this means for your application

Authorization sits on the critical path of almost every request, so it has to be fast and it has to stay fast as you grow. These tests show that a dedicated, Zanzibar-style authorization system can serve one million permission checks per second against one hundred billion relationships with single-digit-millisecond latency, and that capacity grows predictably with traffic and data. The practical takeaway is that centralizing authorization does not have to become your bottleneck, provided you plan around your own schema and traffic.

These tests were run in 2023 on SpiceDB 1.21 and a CockroachDB release from that time. SpiceDB has continued to improve since.

Explore SpiceDB

See how relationship-based authorization works in practice, from the ideas behind Zanzibar to interactive examples you can run in your browser.

Explore SpiceDB · The Zanzibar paper, explained · SpiceDB vs. Zanzibar

Based on AuthZed's engineering report by Victor Roldán Betancort, originally published in July 2023. Read the full report for every test, infrastructure table and configuration detail.