San Francisco Syndrome

The residents of Jerusalem are familiar with an odd phenomenon. A major tourist destination, the holy city receives between 3 and 5 million visitors a year from around the world. People from all countries and walks of life. The vast majority of these visits proceed perfectly normally, but for a small fraction of people, something strange happens.

It is said that travel changes you. And in some rare cases of Jerusalem visitors, a visit to the city brings about a profound change. Some start hearing voices. Others are so moved by the experience that they transform their entire worldview. The characteristic they all share, however, is that they start to believe they are in fact the Messiah, a messenger sent by God to represent Him on Earth.

If you've never heard of this phenomenon before, it may sound crazy. But there are plenty of documented cases over the centuries, in fact so many that there's a name for it: Jerusalem Syndrome.

Today, something similar seems to be happening with regularity in another city of pilgrimage. And the results bear a certain similarity, so I've taken to calling it San Francisco Syndrome.

God is dead? Hold my beer.

In the Western world, organized religion has been on the wane for decades. Perhaps relatedly, instances of Jerusalem Syndrome also appear to be diminishing, at least per capita. But as Voltaire said:

If God did not exist, it would be necessary to invent him.

Since the West killed God, and since Nature abhors a vacuum, it was inevitable that people would look to replace Him. And so it is that San Francisco has become a place of pilgrimage, conversion, and occasional Messiah-dom for a susceptible set of our smartest and most ambitious young nerds, optimists, and sociopaths.

They have become converts to the Church of AI.

The Church has its dogma, as all religions do. The most important one is that no problem is as interesting or as important as the problem (and promise, depending on your P(doom)) of AI.

• Global warming? Who cares? On the timescale of ASI's arrival it's too slow to worry about. ASI will fix it on the way to paving the planet with fabs and solar panels.
• Public debt crisis? No problem, ASI will generate 15%/year GDP growth for decades.
• Cratering worldwide TFR? Also too slow. If we want to solve it (and maybe we don't) then ASI will give us artificial wombs or whatever.
• Healthcare costs? ASI will get us to functional immortality as a side quest to launching von Neumann probes to colonize the galaxy.
• Decreasing trust in institutions? War? Human government is dead. The ASI(s) will run everything with the beautiful efficiency of the Culture's Minds, so the only wars will be brief skirmishes as we assimilate non-believers into the collective.

Unlike Francis Fukuyama's metaphorical End of History, believers in the Church of AI believe in the actual end of human history. Humanity a mere way station on the road to the machine rapture. Like any good religion, much of its fervor comes from internal tensions. In this case, it's between the optimists who believe in the AI utopia I've described above, and the pessimists who believe it will turn us all into paperclips, or some other arrangement of atoms more suitable for its inscrutable machine purposes.

Either way, the Church of AI functions precisely as millenarian cults have functioned for... millennia: (a) the end of the world is nigh, (b) only the believers fully grasp this fact, and (c) they will guide us.

Indeed, San Francisco has an acute case of eschatologitis.

A brief history of the Church of AI

Others can do a better job of writing the history of AI: the thread that runs from Norbert Wiener and his cyberneticists, through John von Neumann, then to Marvin Minsky and John McCarthy. It's worth noting that the latter grew up in a household with two active Communist Party members as parents; he learned math, Russian, and politics from Soviet-era books. I note it because we will see that the theme of a totalizing, utopian political philosophy returns repeatedly throughout the history of AI.

I entered the picture as a direct observer sometime in the 1980s. As I've described elsewhere on this blog, I've been obsessed with computers since I can remember. Predictably for a child of the 80s, my initial contact with the idea of AI came from TV and movies. Most memorably the Matthew Broderick vehicle WarGames (1983), but also The Terminator (1984), D.A.R.Y.L. (1984), even the quirky and oddly endearing Electric Dreams (1984). The idea of superhuman AIs (of varying degrees of human alignment) exploded into popular culture in that decade.

But underneath the Hollywood-sanitized version was an undercurrent of more serious attempts at prediction. Hans Moravec and quickly thereafter Ray Kurzweil wrote books making rather specific predictions about the course of artificial intelligence in the coming decades. In the 90s and early 2000s, an even deeper underground of thought and debate about AI lived in obscure Usenet forums and in particular mailing lists like Extropians and SL4 ("Singularity Level 4", already presaging the end-of-times worldview that would form the basis of the modern Church). The latter served as the training ground for at least a few of the modern prophets of AI doom, including Nick Bostrom and its founder Eliezer Yudkowsky.

I started paying attention to these forums, and to Eliezer specifically, around the mid-2000s. For those who are too young (or choose not) to remember, it's important to realize that Yud was from the beginning a prophet of the end-times searching for a mechanism by which it could actually happen. His first prediction for the technology that would end humanity was the evocatively named "grey goo": nanotechnology robots that could disassemble and reassemble matter at the atomic level.

When nanotechnology research visibly stalled, his end-times thinking shifted more strongly toward the idea of AI "FOOM". Eliezer Yudkowsky has always been a gifted writer. He understands his audience incredibly well and like our current president, he has an ear for language and for labels in particular. FOOM refers to the theory that the "source code" for self-improving AI is actually quite compact. So it's conceivable that someone can discover it, set the AI to self-improve, and (over the weekend while the human is off doing human things) the AI self-improves to the point of taking over the world. And because the AI's value system isn't aligned with that of humans, again it will turn our atoms into things it prefers. Not grey goo this time, but paperclips (a highly-successful Nick Bostrom meme).

Like any good prophet, he set about building his church: searching for and obtaining donations to build an institute (MIRI), initially to work on the research project of how to build AI safely with alignment to human preferences. Two things are worth noting at this point:

1. Again we see the tendency, the instinct to totalitarian thinking. What are "human preferences"? Well presumably, whatever MIRI decides they are. Whatever right-thinking Bay Area transhumanists consider them to be.
2. It is a difficult problem to actually build a machine that, you know, does all the good things. So MIRI pivoted to a series of supposed problems that had to be solved before the machine could actually be built. Writing a completely new programming language, solving various theoretical/mathematical paradoxes about self-modifying agents, also training people to "be less biased" (via another non-profit CFAR), and a variety of other not-obviously-core activities.

On paper, MIRI has been a resounding and colossal failure in its 26-year history. As far as I can tell, the humans it claims to have trained are just as biased as ever and it has produced no useful theory of self-modifying agents who maintain value alignment. There's no question they've produced words, lots of words: papers, technical reports, blog posts, fiction. But no hard technical output to show for all the money donated to it: no new AI-building programming language and certainly no hint of even a prototype of a "safe AI".

And yet, in spite of its manifest failure in these attempted projects, MIRI and its associated constellation of non-profits have been extraordinarily successful at proselytizing for their worldview. Their fledgling church (I'll try to refrain from calling it a cult, but I can't promise anything given the existence of the Zizians) produced incredibly powerful adherents. In fact, those adherents, people such as Dario Amodei, are perhaps the apotheosis of the failure of Yud's dream of "protecting humanity from AI extinction". They have managed to marshal levels of resources undreamt-of in the mid 2000s, all devoted to racing to develop supposedly unsafe AI. While wringing their hands about how they wish they didn't have to.

And then something happened

I'm getting a bit ahead of myself. Because up until roughly the mid-2010s, all of this AI eschatology was just a bunch of slightly deranged transhumanists talking to each other on listservs, waylaying the occasional mathy kid on his way to Jane Street, and generally behaving like Bay Area barnacles siphoning donations from the Bay Area profit machine.

And then transformers happened. A few researchers at Google discovered an AI technology into which you could pour infinite compute (in the form of highly parallel GPUs) and data (in the form of the Internet and any digitizable data you could find, legally or not). It worked, and it just kept on working. As Dwarkesh Patel writes, the scaling era had begun.

The God of Straight Lines on exponential axes turned the cult of AI into the Church of AI. Early belief in the God is what funded DeepMind and then OpenAI. A few more orders of magnitude and it funded Anthropic. After a few more, a half-dozen other new foundation labs have been funded at billion plus pre-product valuations. So here we are, in 2026, with trillion-dollar pre-IPO AI "labs" and endless debate about what it all presages for the future of humanity. At this point, it's important to see clearly the symbiosis at work in the SF AI scene.

1. Unlike the AI FOOM dreams of the past, modern AI technology requires massive resources to achieve vaguely human-like performance. And exponentially more resources to achieve ASI, it is expected and hoped. The God is nothing if not exponential.
2. Problematically, those resources come in the form of capex: money which has to be spent on GPUs, servers, data centers, power, and massive training runs before a single cent is seen in revenue.
3. The only way to justify this amount of spending is to believe in even larger amounts of revenue from the technology.
4. Those larger amounts can only come from an extraordinary revolution in the world's economic structure.
5. Extraordinary claims require extraordinary evidence, or in its absence, an extraordinary story.
6. Like every successful religion must, the Church of AI provides that extraordinary story.

Crucially, even the pessimistic, doom-inflected sect of the church is useful. Essential even. The debate about AI safety increases attention paid to AI in general. It attracts smart ambitious people because that's where the action is. It provides a geopolitical tension angle ("dangerous open-weight Chinese models") which focuses even more attention. In the modern vibes-based economy, the Church of AI is winning the meme war precisely because it has utopian and dystopian versions.

How it looks on the ground

People working in the frontier labs believe they're working on the most important technology in world history. This isn't hyperbole. Anyone who's interviewed at Anthropic will tell you that there is a strong cultural fit component. Candidates are openly and explicitly rejected if they're not AGI-pilled enough. In fact, as Ruxanda Teslo so eloquently describes, in SF AI circles, being insufficiently AGI-pilled is low-status, a reason for scorn and social ostracism.

Like any good religion, the more extreme the required belief, the better a signal it is of adherence to the faith. Extreme beliefs are costlier signals than more moderate ones, so they work better. How extreme? I will tell you:

I currently have a bet with a well-known investor about the possibility of humans building a Dyson sphere around our Sun by the year 2030. For the purposes of our bet, the definition of a Dyson sphere is "something that can capture at least 10% of the Sun's energy". You will be unsurprised, dear reader, that I'm on the "no" side of the bet. But the bet is probably a good investment for my counterparty. It's a costly signal he can truthfully show the rest of the faithful as evidence of his bona fides.

As always, the most important signal is to ourselves. Even a cursory look at human history, literature and philosophy teaches us that we're extremely good at telling ourselves stories. In this case, part of SF AI culture is that there is only one correct answer to the Hamming question. You can see this is true since literally the first result when searching "Hamming question" is a link to a LessWrong post. Everyone should always be working on the most important thing they can. So working in AI became tautologically important: if it weren't world-changing, I wouldn't be working on it. So it must be world-changing.

Of course, the money also helps. If one were to suggest, as I have, that the AI rapture isn't actually imminent, one would of course be met with stares of bewilderment at best and pity at worst. But as Upton Sinclair famously said: "It is difficult to get a man to understand something when his salary depends on his not understanding it." When AI new grads from good universities can receive 7 figure comp packages, when experienced researchers aspire to 9 figures, it's not especially difficult to find reasons to say yes, and to say yes to the cultural, moral and eschatological requirements of the job.

What now?

Where does this leave us? I honestly don't know. The God of Straight Lines may yet have a few more orders of magnitude in store for us. To give the church credit where it's due, the lines have stayed straigher than I would have expected four years ago. So the world may yet be turned into computronium, and our atoms disassembled to make room for that. Whether our new AI overlords deign to scan and upload us as a little favor, as in Roger Williams' The Metamorphosis of Prime Intellect, remains to be seen.

Or perhaps this ends with an AI bubble-popping bang which will be felt as a sad whimper by the believers. The Church of AI is also a Cult of Intelligence, reifying it above all other desirable human qualities: knowledge, wisdom, empathy, even basic social skills. We may find out that intelligence isn't actually fungible into everything else that matters. In such a world, AI tools will take their place as yet another in a long line of wonderful technological developments born of humanity's Promethean spirit.

But whichever way it goes, extrapolating a line on a graph does not, and cannot, form the basis for a moral philosophy.

In the meantime, I'll continue consuming prodigious amounts of tokens every day, await the next model release with bated breath, and keep trying to figure out how maximize the value these systems provide. But also, I've decided to re-read William James' The Varieties of Religious Experience. It feels instructive in a way that might surprise many people.

I'll close by updating Voltaire. It can be interpreted in at least two different ways, depending on whether you stand inside or outside the Church of AI:

If God is dead, then we seem compelled to invent him. Even if it kills us.