SPICEDB AT SCALE
Google-Scale Authorization: 1 Million Requests per Second on SpiceDB
Can a dedicated authorization system keep up when every request in every product asks it a question? A series of load tests set out to answer that, ending at one million requests per second against one hundred billion relationships.
Google's Zanzibar paper describes an authorization system that stores trillions of access control lists and serves millions of permission checks per second for products such as Calendar, Drive, Maps, Photos and YouTube. SpiceDB is built on the same ideas, and a natural question from teams evaluating it is whether that scale is real or just inherited reputation. A prospective customer asked exactly that, and challenged the team to prove it.
The challenge
The goal was simple to state and hard to meet: show that SpiceDB delivers consistent latency as both traffic and data grow by orders of magnitude. The team planned thirteen tests, starting at ten million relationships and one thousand requests per second, and working up to one hundred billion relationships and one million requests per second.
Every test ran for ten minutes and had to meet the same service-level objectives:
- CheckPermission: 95th-percentile latency under 25 milliseconds.
- WriteRelationships: 95th-percentile latency under 100 milliseconds.
- Reliability: at least 99.9% of requests succeeding.
Two secondary goals mattered just as much to teams planning a rollout: understanding how compute grows with traffic and data, and learning how long it takes to load existing permission data into a new system.
A realistic workload, not a synthetic one
Load tests are only as useful as their resemblance to production. Rather than an artificial schema, the tests used a permission model provided by the prospective customer, describing a social networking product:
- People share public or private content, follow each other and block each other.
- Conversations attach to individual pieces of content.
- Public content is visible to everyone; private content only to followers.
That model exercises most of what makes authorization hard: unions, intersections and exclusions, three levels of nesting, and wildcards that open content to everyone. It is a meaningfully demanding schema, not a single role check.
Shaping the data like real life
Real social graphs are lopsided. A small share of accounts produces most of the content and attracts most of the followers. To reflect that, the test data followed a Pareto distribution, where roughly 20% of users generate 80% of the activity. That matters for a graph-based system: very wide relationships and deep nesting are where permission checks get expensive, so a realistic shape is a much harder test than evenly spread data.
Shaping the traffic
The request mix mirrored the customer's expectations: 60% of checks asked whether someone could see a piece of content, 30% whether they could see another person's profile, and 10% whether they could see a conversation. On top of that, 1% of all requests were writes. That is several times the peak write share the Zanzibar paper reports, which makes the test more conservative, not less. Checks used SpiceDB's default low-latency consistency mode, and requests were drawn from a sampled pool of simulated daily active users so that caching behaved as it would in production.
How it was deployed
The tests ran on AuthZed Dedicated, the fully managed, single-tenant way to run SpiceDB, on AWS, with CockroachDB as the datastore. A few deployment choices made a noticeable difference:
- SpiceDB ran as a cluster of replicas on its own compute pool, separate from supporting services such as load balancing and monitoring.
- Replicas were spread across machines and availability zones, so a single failure affects as little as possible.
- CPU cores were dedicated to SpiceDB processes, keeping each core focused on the same work.
Infrastructure grew with each test. At the very top, serving one million requests per second against one hundred billion relationships took 21 SpiceDB nodes with 32 virtual CPUs each, alongside a six-node CockroachDB cluster.
The results
Every test met its targets, and latency stayed remarkably steady as traffic and data grew. Here is how the largest dataset, one hundred billion relationships, performed at each traffic level:
| Requests per second | Check P95 | Check P50 | Write P95 | Cache hit rate |
|---|---|---|---|---|
| 1,000 | 17.1 ms | 8.05 ms | 17.9 ms | 0% |
| 10,000 | 10.0 ms | 3.44 ms | 37.0 ms | 65.1% |
| 100,000 | 5.86 ms | 3.08 ms | 49.3 ms | 89.2% |
| 1,000,000 | 5.76 ms | 3.03 ms | 48.3 ms | 95.9% |
Results from the final round of tests on one hundred billion relationships. Write latency stayed well inside its 100 ms target throughout.
Checks got faster as traffic grew. SpiceDB breaks every permission check into smaller sub-problems and caches their answers. The more traffic it serves, the more those answers are reused: the cache hit rate climbed from zero at one thousand requests per second to almost 96% at one million, and latency fell with it.
The same pattern held at one billion relationships, where one million requests per second ran at a 5.89 ms P95. Data size, in other words, was not the limiting factor; the shape of the data and the traffic were.
What drives authorization performance
Raw numbers are less useful than understanding what moves them. The tests made the main levers clear, and they are worth keeping in mind for any capacity plan:
- Your schema. Every union, intersection, exclusion and level of nesting adds sub-problems to compute. A richer model needs more capacity for the same traffic.
- Your queries. These tests focused on permission checks and writes. Listing everything a person can access is a heavier operation and needs additional infrastructure, or precomputed permissions for very large lists.
- The shape of your data. Very popular objects and deeply nested groups are expensive. Zanzibar includes a dedicated indexing component for exactly this reason.
- How many people are active. The share of users making requests decides how much data must be read from the database and how well caching works.
- Consistency choices. Low-latency reads make heavy use of caching; requiring the very latest data on every check trades some of that speed for freshness, which is the right choice where it matters.
These factors interact. A higher cache hit rate lowers SpiceDB's CPU use and the load on the database, which frees capacity for more requests. Planning around that interplay is more reliable than extrapolating from a single benchmark.
Getting existing data in
Most teams adopting a permissions database already have years of access data to migrate, so import time is part of any rollout plan. In these tests, one billion relationships loaded in about two hours and forty minutes, and one hundred billion in about five days. Once indexed, the largest dataset used about 8 TB including three-way replication.
That work led directly to native bulk import and export in the SpiceDB API, which speeds up initial loads considerably. A rerun of the largest test on ARM-based AWS Graviton instances also delivered about 20% more throughput.
Improvements that came out of the tests
Pushing a system this hard surfaces bottlenecks that ordinary traffic never reaches. Several fixes found during the tests became part of SpiceDB itself, including better balancing of database connections, spreading cache refreshes over time to avoid sudden bursts of database traffic, steadier routing of work between SpiceDB nodes, and more even distribution of that work across the cluster.
Lessons for running your own scale test
- Test with your own schema and data shape. Results from a different permission model or evenly distributed data will mislead you.
- Model real activity. Decide how many users are active and how traffic is distributed across them; it changes the results more than total data size.
- Measure at fine granularity. Some problems only appear when metrics are collected every second rather than every thirty seconds.
- Plan the data load. Large imports take hours or days. Make sure credentials and links used during the import don't expire midway, and pause backups that could compete for resources.
- Budget for scaling the database. Adding or removing nodes from a cluster holding this much data takes time while data rebalances.
- Watch cloud limits. At this size it is easy to hit default instance quotas, and cheaper spot capacity can cost more if a test has to restart.
What this means for your application
Authorization sits on the critical path of almost every request, so it has to be fast and it has to stay fast as you grow. These tests show that a dedicated, Zanzibar-style authorization system can serve one million permission checks per second against one hundred billion relationships with single-digit-millisecond latency, and that capacity grows predictably with traffic and data. The practical takeaway is that centralizing authorization does not have to become your bottleneck, provided you plan around your own schema and traffic.
These tests were run in 2023 on SpiceDB 1.21 and a CockroachDB release from that time. SpiceDB has continued to improve since.
Explore SpiceDB
See how relationship-based authorization works in practice, from the ideas behind Zanzibar to interactive examples you can run in your browser.
Explore SpiceDB · The Zanzibar paper, explained · SpiceDB vs. Zanzibar
Based on AuthZed's engineering report by Victor Roldán Betancort, originally published in July 2023. Read the full report for every test, infrastructure table and configuration detail.