In 2007, Tim Berners Lee write the essay The Giant Global Graph. Quote:

There are cries from the heart .. for my friendship, that relationship to another person, to transcend documents and sites. ..Then any other site or program can use that information.

It seems appropriate as X is sending cease-and-desist letters to Nitter to remember TBL's essay. Nitter is - was - a simple frontend to X which allows users to view tweets without logging in. Even that small use of proxying to the pages is enough to receive threats of legal action.

Twitter's API in 2007 was famously open, which meant thousands of developers building clients, tools, and analytics for free. So, what happened? Why was it pulled? Simple: the network won. The developers stopped being an asset, and the API progressively closed. Rate limits, pricing tiers, login requirements, then technical blocks on the workarounds, and now letters from lawyers. Meta ran the same playbook a decade ago and it's now hard to remember there was ever a Facebook or Instagram API worth building on.

This is why Brewster Kahle, founder of the Internet Archive, has been calling for over a decade for us to lock the Web open.

Nitter started off using X's APIs. When that closed, it read public web pages. And now that there's nothing left to close, the demand is that the source code come down. A program that displays public posts is being treated as a circumvention device under computer-crime statutes.

We have a walled garden problem. It isn't going to change, and the only option in front of us is to start fresh.

The good news is, atproto continues to grow, activitypub remains resilient, and our community is full of believers and builders in the open social web. Since I work on atproto, that's what I'll talk about next.

Interoperation by SELECT *

SELECT * FROM internet.blogposts

The walled garden problem is downstream of a simple question: how do I SELECT * FROM internet?

If you've never written database code, SELECT * FROM users is how you ask a database for everything it knows about its users. Once you have it you can filter it, sort it, and join it against anything else you've got.

The web doesn't historically work that way. The web is a few dozen companies, each holding a filing cabinet, each with a receptionist posted out front. He'll read you one file at a time, but only files you can name, as fast as he cares to read, and as long as his boss allows.

Nitter was a lightweight X reader that worked fine right up until X turned off the access it depended on. Every API (the "receptionist") is a business decision that hasn't been reversed yet.

But Impermanence isn't the only problem. Even a permanent, free, generously rate-limited API wouldn't be enough. Applications need much more meaningful access than APIs can provide.

  • You can only ask questions someone already thought to answer. An API is a fixed menu. It gives you getPosts(user) and getFollowers(user). If your product idea needs "posts from people my followers follow, ranked by how often they get quoted," there is no endpoint for that, and there never will be, because nobody at that company is building for your product.

  • Even the right questions come back in the wrong shape. Followers come 100 at a time. A two-million-follower account is 20,000 round trips. At any polite rate limit that's hours of work to answer one question about one user — so anything interactive, anything that has to feel instant, is off the table before you start.

  • You can't join across "cabinets". The interesting questions are almost always cross-service: this person's posts against that person's photos against a third service's reviews. Two receptionists in two buildings can't cross-reference anything, and neither can you.

  • You can't index data you don't hold. Search, ranking, recommendations, feeds, moderation tooling — all of it is built on indexes over the whole corpus, laid out for the specific questions your product asks. You cannot build an index through a keyhole.

To actually build a service, we need the whole dataset rather than a view onto it; we need it live, arriving as it changes instead of polled for; we need to index it however my product demands; we need to write back into it; and we need all of that guaranteed in a way no single company's quarterly priorities can revoke.

Desktop apps handle this by sharing the filesystem. Internet apps don't use files; they use databases. We need to share the database.

As a user, I don't want to be locked into an app anymore than I'd want to be locked in the trunk of a car. I want an actual free market.

So then, here's another set of needs.

  • Persistence of identity.

    • My presence and relationships are built around my identity. It needs to outlive the app I signed up with.

  • The export of living (not dead) data between services.

    • Exporting archives of your tweets is useless as an account migration solution because data doesn't live in isolation.

    • If data is no longer operable - capable of additional operations by participants in the network - then it's a static archive and useless to another application.

    • You could always print your tweets and look at them I guess.

If we want data to remain operable even outside of its original service, then we need to share the database.

These are all issues atproto is designed to solve, including open data access, account migration, and a live firehose of network activity.

How atproto makes SELECT * FROM internet happen

How do we share the database? We don't. We share a lot of them. We create a whole network of personal data servers (PDS) which applications interact with.

How do we handle apps sending complex SELECT * queries to our personal data servers? We don't. We replicate the data on logs. We have each application aggregate copies of the data to query locally.

How do we have apps write to those databases? In this case- we do! We have the apps send writes to the PDS, which in turn replicate back out to the other apps.

This last one is the core of the intuition about atproto: the write/ingest loop. Almost every atproto app has code that looks like this:

// write
pds.putRecord(post)

// ingest
onPut(‘app.bsky.feed.post’, evt => {
  mydb.put(‘posts’, {...})
})

Rather than waiting for the ingest to come back over the wire, you can use a "short circuit" so your app's database can update more quickly. The 200 OK from the PDS is a transactional go-ahead.

And so the more efficient pattern looks more like this:

// write
pds.putRecord(post)
mydb.put(‘posts’, {...}) // ← optimistic

// ingest
onPut(‘app.bsky.feed.post’, evt => {
  mydb.put(‘posts’, {...})
})

Does it work?

Yes. The network exists. It's live, it's public, and you can read all of it right now — from a laptop, without asking anyone's permission. This is exactly how Bluesky, Tangled, Leaflet, and a bunch of others work now.

Let me hit you with some stats. At time of writing, there are:

It's never been easier to tap into the data with the new jetstream service.

import { Jetstream, isCreate } from '@bsky/jetstream';
import { app } from '@bsky/sdk/lexicons';

const jetstream = new Jetstream('https://jetstream.us-east.bsky.network');
const collections = [app.bsky.graph.follow, app.bsky.feed.repost, app.bsky.feed.post];

for await (const event of jetstream.live({ collections })) {
  if (isCreate(event, app.bsky.graph.follow)) {
    console.log(`🌱  ${event.did}  follows  ${event.commit.record.subject}`);
  } else if (isCreate(event, app.bsky.feed.repost)) {
    console.log(`♻️  ${event.did}  reposts  ${event.commit.record.subject.uri}`);
  } else if (isCreate(event, app.bsky.feed.post) && event.commit.record.reply) {
    console.log(`💭  ${event.did}  replies  ${event.commit.record.reply.parent.uri}`);
  }
}

If you want a fast way to get into it, try it out here.

Stop getting cease & desists. SELECT * FROM internet.blogposts instead.

And, oh, if you're looking specifically for blogposts on atproto, you probably want to use standard.site.

👉 Follow me on Bluesky.