This is the blogpost of a talk I gave at DWeb Camp 26 and LocalFirstConf 26.
I’ve long been a believer in the local-first movement, an approach to software design in which user data lives on personal devices. Prior to Bluesky I worked on a number of local-first projects. These included SSB, a p2p social network, and Beaker Browser, a p2p web browser.
I’m a local-first believer because I’m an advocate for open source and the web. Most of my professional career has been about open computing in some form. The local-first movement has codified 7 ideals and I like them a lot.
Now I work on AT Protocol, a network built with the goal of “locking open” the information commons.
However, the protocol is a departure from local-first. Where we previously might have used peer-to-peer networks, in this case we chose to rely on federated servers.
Why didn’t we keep using local-first designs? It had to do with our needs for scale. We knew that this project needed to handle much larger systems than we’d ever built before. We had to change our approach to the problem.
For my background, moving away from local-first felt like heresy. What’s interesting is that by setting those new requirements, we ended up finding ways to preserve what mattered to us — openness and user autonomy — while meeting those new design goals.
This is how atproto became a high-scale open network.
It’s a distant memory now, but Bluesky was founded to produce a protocol which Twitter could adopt. None of us came from Twitter — we came from the dat & ipfs world — but this was the contract we were hired to complete.
We simplified these requirements into the phrase “no steps backwards.” We knew we had to create something which felt extremely natural to Twitter users, and which could handle the scale of Twitter without sacrificing the network’s open purpose.
We also wanted to make something bigger than the Twitter use-case. The Internet is a global information commons. It is inherently a high-scale endeavor. We want a world where you can write a query against that commons and get streams of useful information.
Before we could do that, we needed to look at the job in front of us. Here’s the numbers we were given:
The Twitter of 2022 wasn’t the largest service on the Internet, but it wasn’t a joke either. Six thousand tweets per second likely undersells the volume of events by at least 10x, based on the post-to-like ratio that we see on Bluesky. That’s a high volume of events.
The Twitter model is extremely dedicated to realtime delivery of public posts. Just serving a timeline involved a complex mesh of services.
Reasonable people can disagree about the merits of microservices, but this design isn’t significantly different than most high scale applications. Different tasks need different solutions, and breaking into multiple services can help isolate the load and specialize the solutions to specific needs.
This is a simplified version of the diagram:
At 240M DAU, you get scary red lines.
Driving all requests into a single database will saturate it. You can scale the db into a cluster or invest in sharding techniques, but if your system can accept non-transactional replication then it’s reasonable to redistribute the data into each service’s disk. This helps isolate read load between tasks.
Purpose-built services also help to optimize for specific needs. The userdata cluster’s job is to keep the data durable and available. The user recommendations server needs to answer graph questions. The ads server needs to handle statistical aggregations. Each of these benefits from a different set of on-disk structures over the same canonical data.
You handle this with data replication.
The general wisdom is to use Kafka for something like this instead of a message broker like RabbitMQ. There’s nothing wrong with message brokers, but they’re better for work scheduling than they are for data replication. Logs offer durable replays with minimal coordination between the services.
With logs, we can sync events and datastores across the datacenter and get the benefits if isolated, purpose-built systems. There’s a lot more at play when handling scale, but we’ll checkpoint here to focus on these specific properties of a high-scale system:
We’re using a variety of services which decouple read workloads from each other as much as possible.
We maintain a canonical userdata store which we then replicate to a variety of secondary databases.
We project that data into more purpose-fitting stores (like graph databases or OLAPs).
With that established, let’s look again at our protocol needs. We’ll create a triforce symbol to represent a single service:
Our goal is to get these various services collaborating — not just on the edges of the experience, but presenting the exact same network to their users.
But of course we’ve now introduced a tough constraint: the public wide-area network (WAN).
The public Internet is much slower than the links within a data center. It suffers from outages — either at the connections or among the nodes. Harder still, it’s an adversarial environment. Participants are incentivized to lie about user actions, and they will abuse resources with the intent to freeload or to disable peers.
Keep in mind that we’re building to meet the scale of 2022 Twitter. This means we’re thinking about possibly hundreds of thousands of events per second, and we’re aiming to now add the public Internet to the equation. How do we handle that?
After a little hemming and hawing, the Bluesky team landed on this theory: we can treat the challenges of the WAN links as equivalent to the challenges of a particularly faulty internal network. That is, the sync between services inside a datacenter can be seen as the sync between organizations. The only thing we have to do is address the question of trust, which we solve through defensive API surfaces and at-rest signatures.
The problems of the WAN are a bit more extreme than within your DC, but they follow similar patterns — outages, bad links, disk loss, recovery flow requirements, and so on. So our solution for atproto looks like the kafka solution in a regular service:
Instead of a userdata cluster, we created the Personal Data Server (PDS) network. Instead of kafka, we built replication into the protocol.
Viewed another way, it’s a world of PDSes flowing into apps, which in turn write back to the PDSes.
Behind the scenes, the PDS hashes all the JSON into a merkle tree and then signs it. That way, we can prove the authenticity of the data as it stores-and-forwards across the internet — thus the name “authenticated transfer protocol.”
With signing, we can safely forward content through relays. The relays aren’t strictly necessary — you can directly sync from the PDSes — but it’s immediately obvious that a large network is going to be running a lot of sync streams, and funneling them into dedicated services can help reduce the number of consumers on the PDS hosts. Again: start from scale.
Writes are committed once they’re acked by the PDS. The writing application can immediately update its internal indexes; the writes will subsequently replicate into the network to other listening applications.
This pull-based sync model follows the same logic of kafka. Referring to the slide again:
Applications syncing via AT Protocol have low coupling and can recover from outages fairly easily by replaying from the PDS hosts. There’s no communication between applications at the time of transaction; the cooperation is asynchronous. It’s perhaps analogous to writing to the user’s filesystem, and allowing other apps to receive an fanotify about the file update.
Data replication is really at the core of atproto. Apps log into the user’s PDS and then send JSON over to be written. Once written, the JSON replicates out over the sync streams to all of the interested applications. It’s a simple, low-coordination solution to creating a shared network.
We’re now averaging 500-600 evt/s in our systems, and have seen up to 2k/s during large events. We’re still far below Twitter’s peak scale, but there are now over 3,000 PDS operators in production, and some hundreds of applications.
The Internet is a high scale system. We want it to be widely accessible as a commons of information — not something that can be locked up behind egregious API costs, as Reddit and X recently did.
As we now develop permissioned data, we’ll begin to address how to scale down and tackle the more private use cases for applications. As much of a journey as nonpublic data has been, we’re pretty sure it’s more straightforward to scale down via essentially small side-networks than it is to scale up. That’s why I’m glad we decided to start from scale.
—
Recent other good reads:
—
Learn more at atproto.com.