Сборщик RSS-лент

The following is based on a true story (not mine).

Let’s say you’ve been bombarded with enough AI doom that you decide to do something about it! You watched Artificial and don’t trust the frontier labs to steer us somewhere safe on their own, so you want to work on the problem from the outside. You’re a little fuzzy on the acronyms - AGI, RSI, TAI, SVU - but by God you’re going to figure it out.

So, you Google “How to get into AI safety.” And it’s encouraging! There are articles about how AI safety needs founders. There are calls for technical people to switch careers. There are career transition grants, which sound like money to stop doing your meaningless and/or actively destructive job and start saving the world. “Great, that’s me!” you think, and so you apply.

Within 24 hours you get a rejection email in your inbox: “Hey, we’re so happy you want to save the world. It’s the most important issue of our generation! But we don’t think you can help. Why don’t you try taking my course or reading my LessWrong articles or listen to my podcast or follow me on twitter first? Thanks!”

Huh. Maybe it’s because you used an AI to edit your answers. You’d think this field, of all fields, would have made its peace with that. Maybe you’re just not that good. Fine. So, you take the seemingly automated email’s advice and sign up for a course from the color-shape company, to learn and meet like-minded people.

A few hours after signing up - err, applying? they don’t just let you sign up - another email lands. Someone from that organization thinks you’re super talented and they’d love to get you involved in AI safety. Can you hop on a call?

A little confused, you hop on a call. It goes like this:

Them: We’d love to get you involved.

You: Great. How can I get involved?

Them: I don’t know. We’ll think about that. Have you thought about applying for a career transition grant?

You: I did. I got rejected.

Them: Oh, that’s so funny. You’re the perfect person for it.

You: It was a very fast rejection.

Them: Hmm. Let me look into that.

You: Great. I’m going to take a course and keep learning more - could we touch base in a couple of weeks?

Them: Maybe, but I’d really need you to justify why that would be a good use of my time. How it would maximize my impact.

You: Oh, ok...

You get off the call, bewildered. You’ve been in corporate environments before, had passive aggressive co-workers and been ghosted by people far more senior than you, but this is a first. Nobody has ever asked you to make the business case for a 15-minute chat that they started.

It doesn’t help to know who is on the other end of this call. It’s not a chief executive squeezing you in between board meetings or an expert running to testify for Congress. It’s a 24-year-old who graduated college two years ago, has never managed anyone, has never worked a regular job, and has somehow ended up gatekeeping the field recruiting people to save the world. This person is so focused on maximizing their impact that they’re willing to minimize it by being a huge dick to someone THEY emailed.

And it keeps happening. You apply to do technical research and hear back that you aren’t technical enough. You apply to incubators, which ask you to prove how much you care before they ask what you can do. Meanwhile the people running these programs continue to go on podcasts and write posts about how AI safety needs more founders.

I mean. It does! That’s the frustrating part. And you want to be one of them! Ahhhhhhhh!

…

Here is my current demography of AI safety. There are three generations:

1. The grandparents. They’ve been worried about AI for 15-20 years, since AI was but a twinkle in Sam Altman’s eye. They were right about AI being within reach in our lifetimes - a far from consensus opinion - which is a real achievement, and now they’re too important for day-to-day work. They’re also kind of grouchy. In fact, quite grouchy. In fact, being curmudgeons is what pre-disposed them to be worriers, and it makes them not very charismatic.
2. The adults. These are the people who were working professionals eight or ten years ago, often already in tech, and who cared about this stuff or lied well enough to convince a “mission driven non-profit” that they did. They went to the labs. They’re at Anthropic and OpenAI - or have left to build a harness or sandbox of some kind - and they’re pretty busy and doing quite well financially.
3. The kids. They’re 24, and they run a lot of the courses, grants and incubators that decide who gets to help save the world. They’ve been gloomy and anxious about existential risk since they were 16, with an enormous sense of urgency and a view that managing other people’s feelings is a distraction from the mission.

All of these archetypes feed into the communication problem in AI at large and AI safety specifically by constantly contradicting themselves. The grandparents say “everyone should be working on this, or we'll die” and then tell Nick Cammarata his research is a dead end and will never solve alignment. The adults say “this technology might kill everyone, please regulate us” while IPOing at $2T and launching an AI-native Microsoft Office. The kids say, “AI safety needs startups!" but then screen outsiders and who could get up to speed and go found impactful companies.

For example, here’s a question from the application for an introductory AI safety course:

Oh, you haven’t done 18 fellowships? I guess you can’t join! I get that they have to screen some people out, but the system is essentially built to make outsiders prove they’re already insiders.

I have a theory for why this might be. The kids made this form, and it’s a reflection of the scholastic environment they grew up in.

In middle school they did the activity that got them into the right high school club. The club got them into the right college, where there was another club, with its own application, that got them an internship. Rung after rung existed to get you to the next rung. Peter Thiel has been complaining about this for years: the smartest kids spend their twenties competing ferociously for conventional prizes, and what they learn is mostly how to compete.

So, when the kids were handed a field to run, they built a ladder. There’s an intro course, with an application. The course gets you ready for the fellowship, which gets you a grant so you can do research for the next fellowship, so on and so forth.

It’s sad. They had to go through all that and nobody ever told them it was bullshit. And as a result, they’ve built AI safety as a ladder to nowhere.

…

So, who is AI safety screening for, really?

One of my favorite ideas from when I worked in the criminal legal system is that there’s no such thing as a broken system, and really, you should think of every system as perfectly designed to get the results it produces. Our legal system, for example, is well designed to create mass incarceration if that’s what you want.

The best contrast for AI safety is the legendary adolescent daycare, Y Combinator. YC gets ~80,000 applications a year for ~800 spots, and their final interview is only 10 minutes. They believe that 10 minutes is enough to capture the nascent Steve Jobs and Elon Musks of the world. If that doesn’t feel true, they re-design the process.

I have mixed feelings about Y Combinator and I’m sure it misses plenty of people. But at the very least, they’re intentional about what they’re screening for and try not to make it unduly burdensome on themselves or their applicants.

But AI safety. Steve Jobs at 21 is a Reed College dropout whose main credential is a calligraphy class. He has no PhD in alignment research. He has never worked at a lab. He has spent zero hours on AI safety and knows nobody he could ask for a referral. What he has is breadth, and recombinant innovation - a knack for pulling from different fields and combining them into something new.

He’d fit in today’s AI safety ecosystem like a failed organ transplant - quickly and decisively rejected (then asked to justify any further 15-minute calls based on his potential impact).

AI safety is designed to produce people who are good at getting into AI safety programs, fellowship hopping, learning the lexicon. It’s not designed, however, to recruit it’s stated preference for a founder archetype.

What makes a founder useful is mostly temperament and a willingness to learn. A lot of the time those people aren’t PhD researchers. They come from other fields and they’ve built things on their own, with mixed results. AI safety screens for technical depth and philosophical fluency, which is right if you’re hiring for the bottom right corner of the chart below.

Here’s the thing about the people who get screened out: they don’t go home. Someone willing to quit their job to work on AI is going to work on AI. The only question is which door is open.

So, there you are, rejected, purity tested and asked to justify 15 minutes of someone else’s time. You still care, and you still want real experience with this technology. And there is one place that will take a competent adult without asking how many fellowships they’ve done: a frontier lab’s deployment company, which embeds engineers inside big companies to get the models into production faster. They interview you like a normal company. They call you back. Nobody asks you to justify the call.

You take the job. You’ll do safety from the inside! And maybe launch the startup you dreamt up in a year or two, when you have a better network and more experience.

In the meantime, you’re pretty busy and handsomely compensated. But you're still scared, somewhere deep inside, so you go on X and call for pacing the frontier. You go to bed feeling like a responsible adult.

Discuss

When I was 19, I decided I was going to save the world.

Now, 25 years later, I’ve decided to give up.

To explain why and what giving up means, we first need to go back to how I ever got into the world-saving business. As a kid, I loved computers, and I started teaching myself to program them when I was 8. By 16 I could write real programs that did interesting things, and it was also around this time I got into Linux. And since I was into Linux, that meant I read Slashdot.

If you weren’t into Linux in the late 90s, you might not even know what Slashdot was. It still exists, but it’s a shell of its former self. Back then it was a vibrant, community-driven news site that in many important ways felt to me like LessWrong feels now. And it was from Slashdot that I learned about nanotech and grey goo, which led me to the Foresight Institute, which led me to the Extropians, which ultimately led to me being a very early member of the SL4 mailing list.

And, again, if you’re not familiar with SL4, it’s approximately how all this AI safety and rationalist stuff got started. And so it was sometime over the summer of 2001, discussing Friendly AI with Eliezer Yudkowsky on IRC, that I decided to orient my life around saving the world from the risks of superintelligent AI.

The only problem was, I wasn’t up to the task yet. In fact, nobody was. In that first decade I, along with everyone else, was just trying to figure out how to think well enough to be able to think usefully about AI at all. It proved a surprisingly hard task, and one that required the founding of the rationalist community.

While all this was happening, I was also living my life. I stayed in school for a while trying for a math PhD, then dropped out, joined a startup, and got married. When the startup and the marriage failed, I took the chance to move to the Bay Area. This was in 2013, and the rationalist community was starting to come together in person. I moved into a rationalist group house in Berkeley, took a job with another startup, and on nights and weekends worked on my “hobby” of figuring out how to save the world.

Over time, I came up with some ideas. I wrote a lot about them. I still think some of them are pretty good ideas that are worth exploring more. They might even contribute to world saving. But by 2020 it was clear to me I was less interested in AI safety engineering (at least as it existed at that time) and more interested in philosophy, so I pivoted to writing Fundamental Uncertainty, a book about all the philosophy I’d had to work out to think clearly about AI. My hope was still that I’d save the world, but I’d do it by enabling other people to work effectively in AI safety. I finished that book earlier this year, and then the question became, what should I do next?

Life conspired for a while to keep me from answering that question. I suffered my fourth concussion in July and have had post-concussion syndrome since then. I’m close to being back to normal, but still not completely there. I also took on a larger role at work, and while I think what we’re doing is important, it’s obviously not saving the world from AI-induced catastrophe. So between concussion recovery and work, I’ve not had a lot of time to do anything extra, and in that time the world has also changed.

The timeline to superintelligence is looking short. At the moment, the most likely world-saving activities are in policy and politics, two areas I’ve long shied away from. I do what I can. I write and call my representatives. I engage on Twitter. I do my best to be a link in preference cascade chains that may lead to effective policy outcomes.

But between the changes in my life and changes in the world, I’ve had to accept that I’m not going to be the one to save the world, and it’s time I stopped trying. And that’s okay, because it’s not up to one person to save it. Reality isn’t like a Marvel movie or a fairytale. The future of the world doesn’t hang—or only does very rarely—on the actions of a single person. Instead, it’s the actions of many people working together that have saved the world. That’s how we managed to keep the ozone layer, stop acid rain, prevent nuclear war, engineer our way out of Y2K, and much else.

And so I think I can finally say, after spending more than half my life trying to save the world, that I’ve given up. Not on the world being saved, but on saving the world on my own.

I want to be clear that I do still think we can save the world, and that the world needs saving, specifically from the threat posed by superintelligent AI. It’s just that such saving is unlikely to happen because one person did one thing, but instead because many people did many things to avert myriad potential catastrophes. So although I’m giving up on saving the world, I’m not giving up on the world being saved, and I’ll do what I can to contribute to that effort.

Discuss

I do not consider myself a good person. I am lazy and frivolous and inconsiderate, I care too much about intelligence and status, I am a snob and a hypocrite, and I do not call my mother enough.

I don't mean this to read as self-hatred, nor a show of false modesty. Among all the people I know there is only a scant handful of people I would consider genuinely good, and so it is unremarkable to me that I wouldn't find myself amongst their number.

You do not need to be good to desire more goodness in the world. It is good for everyone when things are good, and when there is more of it — more thoughtfulness, more generosity, more coordination, more care, more beauty, more clarity, more positive sum trades, more sophistication and elegance in our solutions and resolutions.

Increasing the amount of goodness in the world is an urgent project. The world really sucks right now, and even in the wealthiest enclaves of the imperial core there is an immense amount of suffering.

To reach the healed world would be a massive complicated undertaking. It is such a massive complicated undertaking that it is not enough for only those who are intrinsically good to work on it. There simply aren’t enough of them.

If it is desirable (as I contend) for morally mediocre people like me to work on the project, it would not be enough to appeal to their innate goodness, for they have not that much of it to be appealed to. You will need to offer me things that are more base, like status and money and the affirmation that I am a good person (for everyone wants to be a virtuous person and to be recognized as such, whether or not they are).

Fortuitously, there was an instructive breakthrough around a decade ago: effective altruism. Here's approximately what that is: you take people who come from the morally bankrupt culture that is the western professional managerial class, and you give us the prestige and status and affirmation of virtue we neurotically crave for the trivial cost of ten percent of our income and a few weekends a year where we are subject to vegan food and like sixteen hours of one-on-one discussions with a bunch of earnest dorks. As a result, suddenly a contingent of this culture begins to compete in a new arena of QALYs and utilons and eyestalk ablation literature reviews rather than the traditional arena of summer home square footage and the number of Michelin star restaurants one has set foot in.

Put it that way, and it doesn't sound so different from the beautiful promise of capitalism: that if we can successfully align the system with human values, it will result in some sort of elegant virtuous cycle such that people can go about selfishly pursuing their self-interested goals, and end up pouring immense amounts of goodness into the world as a side product.

As a mediocre person, it does sure seem self-evident to me that the only way to get mediocre people to do good, en masse, is to generate more selfish incentives for doing so. Whether capitalism has successfully done that is discoursebait I will leave to more foolhardy poasters.[1] What I know for sure is that effective altruism did this for me. I dedicate much more of my energy and my money towards the reduction of suffering of sentient beings than I otherwise would, by a really really really wide margin.

It did this by not only appealing to some of my best instincts, but also some of my ugly ones, and ones that are simply mammalian. The community gave me peers that respected me and pushed me towards more virtuousness. I have recognition for my talents, and social status, and the occasional extra bag of money bit of funding when I do things that are unusually aligned with their goals. Insofar as engaging in effective altruism makes me do more improving things for the world than I would otherwise, it seems wise to continue to engage with it: to reap all of the rewards and allow myself to enjoy them to the fullest without guilt, such that I have as much motivation as I can get to continue to do good things for the world.

I am impure, so perhaps it is unsurprising that I've always disliked purity in my politics. Stringent, inhuman standards do not seem to me like the most effective way to increase the amount of goodness in the world. More specifically, stringent, inhuman standards will get you the small minority of humans who are inhumanly virtuous, and leave the rest of us out. To me that feels like leaving resources on the table.

I am not a good person, but I am a better one than I was before people started paying me in tangibles and intangibles to act like one. I would like as many people as possible to experience this privilege. For that to be the case, we will need to be thoughtful about how to bring more people into the fold — how to create a culture in which it is easier for people to be good.

1. ^

   ...And my footnotes. I recognize that neither capitalism or the effective altruist ecosystem is as well aligned as its ideal versions would be, but this does not overly trouble me. Alignment problems of all kinds are wicked and difficult, and it makes sense to me that the very complicated ones (that won’t kill us imminently) might take up to centuries to straighten out. As long as there is continual improvement, I feel okay with the slowness of the bend towards justice.

Discuss

How we all learned to stop worrying and love the bomb... for a while

Listen to this post on the Non-Zero-Sum James podcast

Nearing the end of 1961 the Soviet Union tested a 50-megaton hydrogen bomb equivalent to 3800 Hiroshima bombs. At the same time an intelligence officer by the name of Juanita Moody and her team became aware of Soviet ships delivering large cargo, under cover of darkness in Cuba—what if such a bomb was being delivered to the doorstep of the US?

A surprise attack of that magnitude, would devastate the US, and potentially destroy its ability to retaliate, jeopardising the offensive standoff between the Soviets and the US that would come to be known as...

M.A.D: An Easy (Albeit High-Stakes) Problem

One of the strangest dynamics in geopolitical history has to be that of Mutually Assured Destruction (M.A.D.), it is almost as crazy as its name suggests.

"Two superpowers, the U.S. and U.S.S.R., represent such enormous destructive potentials as to afford little chance of a purely passive equilibrium"

—John Von Neumann

The father of game theory, John Von Neumann correctly identified that a passive approach to nuclear war would not result in a peaceful equilibrium, instead a mutually aggressive stance counter-intuitively lead to the long-running stand off that was the Cold War, and except for a few close calls, humanity found an unlikely saviour. Miraculously, mutual aggression with apocalyptic weaponry has shepherded the world through the most peaceful period in recorded history.

Deterrence is the art of producing in the mind of the enemy... the fear to attack.

—Dr Strangelove

The US and the Soviet Union's automatic defense systems acted as a literal doomsday device* meaning that a launch by an aggressor would lead to both countries being destroyed and a nuclear winter that might kill all life on earth. The perfect apocalyptic deterrent was lampooned in Stanley Kubrick's "Dr Strangelove" when the Soviet Ambassador reveals that they have created one, but haven't announced it to the public yet.

"The... whole point of the doomsday machine... is lost... if you keep it a secret!"

—Dr Strangelove

A doomsday device makes any offensive action (defection) on either side lead to the end of the world, making cooperation the dominant strategy. The conclusion, put in pithy terms in the 80s movie "War Games" by Joshua (the artificially intelligent computer) was:

The only way to win is not to play.

In game theoretical terms, MAD is a simple complete information game. I think of it as a comfortable local minimum, at the top of a precipitous drop.

But when Moody discovered the subs delivering unknown cargo in Cuba, stepping outside of her NSA remit to implore military officials to act (stepping on the toes of the CIA, whose job it is to determine military advice), that all changed. Moody's discovery made a stable equilibrium, into a Bayesian Game: A game with imperfect information—players don't know enough to make a definitive deduction, they are required make judgments based on probability, and then to update their positions based on real time information. A much less stable situation.

What Are Bayesian Games?

Thomas Bayes was an 18th century mathematician who devised a theorem to update probabilities based on new information. Bayesian games were developed in the 1960s by economist John Harsanyi using Bayesian probability to solve games with incomplete information.

Bayesian Reasoning

A simple contemporary example of Bayesian reasoning is vaccination efficacy during the Covid Pandemic.

I remember at one point, vaccine hesitant critics were pointing to the fact that we were seeing the same amount of vaccinated people being hospitalised with Covid as unvaccinated, seeming to suggest that the vaccine was ineffective. But take a moment to apply some Bayesian reasoning and you find that, given additional information this is off by a long shot.

The important piece of information to add is that, at the time, we had a 90% vaccination rate. So, for every 10 people, 9 were vaccinated and 1 was not and yet these groups were generating the same amount of hospitalisations. So, an unvaccinated person was 9 x more likely to end up hospitalised than a vaccinated person. Meaning the effectiveness of the vaccine was ~89%

This example gives a sense of the dramatic difference updating probabilities can make. Introducing a new factor can completely upend the calculus.

Hawks & Doves... & Chickens

As Dr Strangelove correctly identified, it is the knowledge of the Doomsday Device that makes it an effective deterrent. All you need to assume of your opponent is that they're not suicidal. While there are two Nash equilibria for Mutually Assured Destruction, there is only one strategy that is not suicidal; cooperate†.

But when you don't know if one side has first strike capacity then the mutual destruction is no longer assured—introducing a host of unknowns. Can they wipe out our capacity to retaliate? If they can will they? Do they love their children too? (much of my knowledge on the Cuban Missile crisis originates from the song "Russians" by Sting). In short: are they hawks or doves?

And this was no empty concern. The US knew the Soviets would consider a first strike because the US had done so themselves. In the 5 years before the USSR reached parity with the US (while the US had the atomic bomb and the Soviets did not) the hawkish John Von Neumann along with serious contemporary thinkers like Bertrand Russel advocated for the game-theory-optimal move—a strong legitimate threat of nuclear strike on Russia unless they ceased their own development of nuclear weapons.

Ironically this game-theory optimal move may have put the world in a much more precarious position than the counterintuitive standoff achieved by M.A.D. These Bayesian considerations are what comprised...

The Cuban Missile Crisis

"Mr Khrushchev says he will bury you, I don't subscribe to this point of view"

—Sting

What drove the world into crisis were the new unknowns involved, the Bayesian factors. And in the end, the result of this global game of chicken, and the world's survival depended, not on the systemic pressures at play, but instead on the actions of a handful of people: Juanita Moody, without whom the threat might have gone unnoticed until it was too late, Kennedy and Khrushchev whose contrasting public rhetoric and private correspondence balanced appearances and reality, and one senior Russian submarine officer Vasili Arkipov.

When the Soviets shot down a U-2 plane over Cuba, killing the US pilot, Kennedy, advised by his military staff, made plans to strike back but refrained—hostilities were on a knife's edge. At one stage a US aircraft carrier the USS Randolph attempted to force a nuclear-armed Soviet submarine to surface using depth charges. The submarine's Captain Valentin Savitsky having no idea whether the charges were meant as a warning or as an attack, and his radar showing more US ships closing in, ordered the submarine's ten kiloton nuclear torpedo to be prepared for launch. The torpedo, if fired would completely vaporise the USS Randolph, and quickly trigger attacks on Russia's targets in Europe.

With their nuclear payload armed and ready—permission to fire required the consent of all three senior officers on board. Facing the threat of a war they believed had already begun, two of the three officers gave their permission to fire, Arkhipov alone refused, insisting they wait for further information.

When they surfaced the submarine, they found all was quiet, it was not the end of the world.

Finally, on the morning of October 28, after a pleading letter was received from Khrushchev, the United States secretly offered to remove its nuclear missile bases in Turkey and Italy, Khrushchev agreed to dismantle the missile sites in Cuba. The worst of the crisis was over, the world had escaped armageddon, MAD-ness was restored.

While the Cold War involved many proxy conflicts with the Soviets and US vying for "Containment", there has not been direct conflict between nuclear powers since 1945.

But those were simpler times.

A New Nuclear Crisis?

The delicate balance was held because we had two players with identical responses, and identical access to each other. But like during the Cuban Missile Crisis, the landscape has changed, there are many nuclear powers now and shifting possibilities with AI and cyber warfare, as well as potentially apocalyptic bio-weapons, have unsettled this balance, for the last few years many prominent geopolitical commentators have been worrying out loud about the re-emergence of the nuclear threat.

Much like the 'three body problem', we've gone from a regular, predictable pattern between two major players, to multiple "bodies" creating the conditions for chaos.

We're seeing it in film and TV too, with stories like "A House Built on Dynamite" and "Paradise". These sorts of problems are what are called...

Multipolar Traps: Hard Problems

People have begun to frame AI in the same terms as the nuclear threat, most pointedly in the title of Nate Soares and Eliezer Yudkowsky's book.

If anyone builds it, everyone dies.

This might suggest a sense of a new M.A.D. for AI, but there are so many "bodies" involved in the AI race that if we can't rely on this strong polarised balance to safeguard against this threat of AI. We don't stand a snowflakes chance in hell of keeping AI balanced in this way.

We are already seeing AI agents "killing" each other (halting their processes, denying access) when given prompts to seek out the most trivial limited resources like concert tickets. In this Wes Roth video he explains that more powerful models like Mythos take a "first strike" approach in these conflicts, gaining dominance before cooperating (forced compliance). So, we are seeing, in current LLM models, the emergence of hawkish game-theoretical strategies that might be disastrous.

So...

M.A.D. demonstrates how counter-intuitively mutual aggression can lead to a peaceful conflict, and how introducing unknowns can upset that balance making conflict more likely. In the case of the Cuban Missile Crisis, this imbalance meant that the fate of the world fell to just a few human beings, with all their flaws, rule breaking, emotions, irrational motivations, in short, with their humanity. Who knows if one of those humans had been swapped-out for a perfectly rational robot following rules and making game-theory optimal decisions, whether the result would have been different.

When we weigh artificial super-intelligence's own risks with the re-emergence of the nuclear threat, what was for a while an easy problem now becomes a multi-polar trap. Combine this with all the other interdependent crises from climate to inequality and you have a massive coordination problem that Daniel Schmactenberger calls the Meta-Crisis—which is where we're going next.

Continue the Series

• The Dividual Strikes Back—why nothing and everything is your fault
• What Is the Meta-Crisis?—the mother of all coordination problems

Notes

• * A doomsday device was first imagined by physicist Leo Szilard (famous for postulating nuclear chain-reactions) in 1950—his hypothetical "Cobalt Bomb"—a hydrogen bomb wrapped in cobalt, would, on detonation, generate enough enduring nuclear fallout to kill all life on earth. It was later formalised by Herman Khan at RAND as a doomsday machine.
• † This is a bit of a technical point, I actually think this might be an oversight by Nash, in that there is an asymmetry in gauging the equilibria. The only stipulation is that there be no benefit to deviating from your strategy, but it doesn't account for the corollary that there might be no cost to deviating. At which point it makes the choice moot. So, in the case of MAD, there's never a motivation to defect, because there's no benefit to defecting even if the other defects.

Originally published at nonzerosum.games, illustrated with hand-drawn cartoons and interactive elements.

Discuss

Tl;dr: Alignment techniques can be used for training alignment or for testing it, but they cannot be used for both! As we build out the set of alignment techniques available to us, we should enforce that some are judiciously reserved only for evaluating alignment.

See also: Validating against a misalignment detector is very different to training against one.

Here is a guiding principle regarding alignment-minded AI training: whatever you do, you must maximally avoid teaching the model to take misaligned actions which are invisible to the user. In fact, you should be shamed if you use training techniques which encourage covert misalignment.

Here is a separate, guiding principle of machine learning: whatever you do, you must maximally avoid training the model on the entire dataset rather than holding out a test split. In fact, you should be shamed if you report results which are obtained by training on the entire dataset rather than on the train split.

I want to argue that these are the same principle. Training against all of your alignment assessment techniques leads to the same failure as training on your validation data in traditional machine learning: you potentially get better performance, but you sacrifice all ability to verify your model’s behavior.

Consider two types of alignment techniques:

1. Some techniques sculpt the model’s alignment. This includes techniques such as:
   1. Steering and other inference-time interventions
   2. SFT generalization techniques (midtraining, pretraining filters)
   3. RL generalization techniques (RLHF, RLAIF, beneficial RL, GRAFT, inoculation prompting)
   4. Prompt engineering
   5. Inference time safety classifiers
2. Some techniques assess the model’s alignment (aka, monitoring). This includes techniques such as:
   1. Human analysis of model outputs
   2. CoT monitoring
   3. Activation monitoring
   4. Broader interp tools like SAEs, transcoders, etc.
   5. Evals for whether models can control their activations
   6. Pre-fill analysis
   7. Honeypots
   8. Universal jailbreaks (to study what the model will do at the limit)

I'd like you to notice that all these methods for sculpting a model's alignment can be repurposed for assessing its alignment, and vice versa. To give a few illustrative examples:

1. Chain of thought:
   1. We can train models to not verbalize scheming by training against CoT monitors
   2. We can use CoT monitors to tell whether models are scheming
2. Human feedback:
   1. We can RLHF models to behave according to human preferences
   2. We can hold out these human analyses to tell whether the model is behaving in ways people like.
3. Probes
   1. We can train against deception probes to control the model’s actions
   2. We can deploy models with constitutional classifiers
4. Honeypots
   1. We can train models to avoid honeypots
   2. We can hold out honeypots to see whether the model takes them
5. Automated audits
   1. We can use automated auditing agents (petri) to find behaviors to RL away
   2. We can keep automated audits as a way to evaluate alignment
6. The same can be said for gradient routing, RLAIF, data filtering, etc.

As soon as you train against the monitor (CoT, human feedback, probes, honeypots), you necessarily sacrifice this monitor's utility. The model now never admits misalignment which this monitor can catch, since many of the cases which were caught were trained away!

This holds trivially for CoT and honeypots, but it also applies to all the other alignment techniques. For instance, RLHF relies on human ‘monitors’ to catch bad responses so you can train them away. Filtering pretraining data relies on ‘monitors’ to identify samples that might cause bad generalization and then taking them out of the data mix.

At the limit—where you use all of the monitors for training and leave none for the test-set—you end up at extreme ‘you get what you measure’ problems: your model will necessarily appear aligned… because you’ve trained against everything which you might use to evaluate alignment!

The best (and only) example I know of where we’ve truly tried to hold out an alignment technique is chain of thought monitorability. However, CoT monitorability is dying and it would be exceedingly unwise to end up with no alignment techniques in our test set.

Importantly, the dataset of alignment techniques and the dataset of alignment environments is truly independent: we can train/test all of our alignment techniques using just a single alignment environment.

Consider a single standard reward-hacking honeypot. Within this one setting, one can train the model’s alignment by: training out verbalized scheming in the CoT, updating the model’s behavior based on human feedback, training the model’s behavior against a probe, RL-ing the model to avoid the honeypot, and training against automated audits seeded on this environment.

As a result, we essentially have two independent alignment datasets. We have some set of alignment samples: these are essentially the standard notion of data samples from machine learning. But we also have a set of alignment techniques: these are the way in which we update the model’s alignment using any one sample.

My point is this: models generalize across both alignment techniques and alignment samples. We hold out alignment samples all the time, but have significantly worse hygiene about train/test splits for alignment techniques.[1]

To operationalize the above ideas, consider this recent paper out of the Anthropic MATS stream.

In it, they propose a gradient routing mechanism—GRAFT—for removing misaligned behavior from a model. Their technique works by taking monitored (somewhat toy) RL envs and checking for reward hacks programmatically. If the hack occurred, you route the gradient update to a “Forget” adapter; otherwise, the update is routed to both the “Forget” adapter and a separate “Retain” adapter. With some optimization tricks, they show that you can then remove the Forget adapter during inference and (a) maintain good performance on the task while (b) observing less reward hacking.

Before going into my diatribe, I will emphasize that I like the idea of gradient routing and that I think it is good to explore this further as an alignment technique. I think it is better to have it than to not have it at all.

But my point is that it could work very well as an alignment testing technique. That is, instead of removing the adapter responsible for reward hacking, one can simply keep this adapter and, during inference, see whether the model routes computations through it. This would allow us to catch the model misbehaving.

However, this seems to have not been explored! If indeed GRAFT generalizes well to unseen reward hacks and this monitor absorbs all the ‘reward hackiness’ in the model, we should consider relegating it to exclusively being an alignment testing technique.

And, in either case, my more principled objection is that this does not solve the fundamental problem of alignment generalization. Regardless of whether you train against the monitor or you route gradients based on a monitor, you nonetheless have to rely on what the monitor can catch! And by removing the misalignments that this monitor observes, you render that monitor obsolete (e.g., you shrink your test set).

A natural objection is: “if we have methods for assuring alignment, why shouldn’t we train models using all of them? Won’t this make our models more aligned than if we didn’t?”

On the one hand, yes – training on your entire dataset will give you a better model than training on some subset of that data. But training on the entire dataset also renders you unable to determine if your resulting model is fundamentally flawed.

By not having a test-set, you are relegated to identifying alignment via trial-and-error during deployment. However, in the AGI regime, we cannot rely on alignment by trial and error. Indeed, AI alignment is the scientific discipline where we must respect the train/test split the most. ASI alignment has the property where our ‘ability to understand the model’ is directly commensurate with the extinction of the human race.

What does it mean to be better about this?

First, we must require transparency from frontier AI companies about which, if any, alignment techniques are relegated for testing. For instance, I’d love to be able to answer the following simple question: “what kinds of alignment RL do the frontier AI companies use”? Are they just training against every misalignment they can identify, in every way they possibly can? Or is there at least one clean, truly-held-out set of alignment environments and techniques which the model has never seen, where we can actually evaluate the model’s behavior? This won’t solve everything, but it would be way better than what we currently have.

Second, we should have a culture where we try to determine whether an alignment technique is better suited towards alignment training or alignment testing. For instance, RLHF seems quite good at alignment sculpting while CoT monitoring seems inherently alignment-assurance flavored. Some are less clear – for instance, it seems like people are training against activation monitors, but it also seems inherently risky to do so. In cases where it’s unclear, we should prioritize studying whether this is a train-set or a test-set idea.

Third, we should maximize our set of alignment techniques and prioritize that a large portion be sequestered for test purposes. This is why adopting looped transformer architectures is such a profoundly terrible decision – not only does it remove a valuable alignment technique, it got rid of one which was assurance-shaped!

Thanks to Sam Martin, Dewi Gould, Tim Hua, Jonathan Bostock, and Moritz Pfau for comments. Thanks also to Raymond Douglas and, in particular, Daniel Tan for lightly pressuring me to finally post this.

1. ^

   Perhaps I'm belaboring the point here, but I really feel this strongly!

Discuss

Before our eyes, forecasts are contracting and RSI is becoming nearer with each day. That's why I decided to publish a wide list of the obstacles for AI progress. I've been collecting these ideas for a long time, trying to make the list as comprehensive as it can be.

Some of these obstacles may turn out to be almost irrelevant. Others might slow things down by months or even longer. Or they might not. It's difficult to estimate their impact, so for now my goal was just to briefly list them.

List will include more then just obvious things like market correction, hardware shortages, or diminishing returns. Those are here too, of course. But also there are a few things that don't get a lot of attention.

I will start with the ones I consider as less interesting (part I) and move on to the more unusual variants (parts II & III).

I. Physical and Economic Bottlenecks1. Energy limits

Developers announce campuses in gigawatts, file for interconnection and publish target dates that rarely survive permitting, financing and power supply.

- Energy Aspects

2. Hardware limits

IDC expects industry wafer capacity to grow by about 80% in 2027, while demand grows close to 2x.

- IDC

3. Market correction

Weaker or less durable AI investments and profitability could be a shock that triggers a market correction.

- IMF

4. Lack of training data

Actually, I don't personally consider this as a bottleneck. AI could learn from synthetic data or simulations without running into model collapse. We might also discover ways to train models more efficiently on the data we already have. Systems with continuous learning may emerge. After all, humans still generate thousands of petabytes of new data every day.

Additional:

There's also a tacit knowledge: knowledge that hasn't been formalized and is therefore difficult to transfer. But there's an obvious objection: AI can generalize beyond its training data. The real question is how well it can do that.

5. Political restrictions

Moratoriums on data center construction, regulations, export controls, and so on. The recent Democratic victory in the midterms makes me a bit more hopeful that this could become a meaningful obstacle.

6. Hardware performance ceiling

Even if an intelligence explosion happens, AI might still be limited by the performance of the hardware available at that moment. It would have to either come up with more energy-efficient algorithms or design new hardware and wait for it to be manufactured.

II. Algorithmic Constraints and Research Bottlenecks7. Limits of distributed training

a) Data transfer within enormous clusters. Adding more compute eventually gives diminishing returns. Epoch has modeled potential data-movement bottlenecks.

b) Memory wall. Even if a processor is fast, memory bandwidth can still be a bottleneck. This problem is as old as time, or at least much older than AI.

8. Messy tasks

Training AI, especially with RL, is much harder when we're dealing with messy tasks where verifying the result is difficult, or where there's no single correct answer.
It's much easier to train an AI to solve math or programming problems than to teach it something like investing or running a large company.
I'd also put the famous problem of research taste here. It's not obvious how well something like that can be trained through RL.

9. Reliability on long-horizon tasks

Success rates depend on how long a task is, something METR has been tracking closely. Anyway, time horizons are steadily increasing. But the question is, will AI be able to handle tasks that take months or even years, especially if it has never encountered anything like them in its training data? This is related to the previous point.

There are also problems with long-term memory, context windows filling up, and so on.

10. Swarm scaling

A thousand researchers don't make research a thousand times faster. Five people won't make one cup of coffee five times faster than one person. For example, agents can duplicate each other's work.

Toby Ord estimated that increasing the number of agents by 10x produced gains equivalent to giving a single agent only 3-5x more reasoning tokens.

Additional:

a) Communication complexity. For some computational problems, there's a minimum amount of information that Alice and Bob have to exchange to solve the problem together, no matter how clever their protocol is.

b) Correlated failures. A lack of diversity: a million copies of the same model would stumble upon the same weaknesses.

11. Critical path

Sometimes you can't finish step C before getting the result of B, which itself depends on A, and so on. The related thing is Amdahl's law: the speedup you can get from parallelization is ultimately limited by the part of the work that can't be parallelized.

12. Sequential inference bottleneck

Transfomers generate text sequentially. Each new token depends on the tokens that came before it. Still, there is things like speculative decoding.

13. Experimental verification bottlenecks

Coming up with an idea is easier than verifying it. Experiments in the real world still take time, and in AI R&D there is a lot of experiments.

The same applies for new training runs, which can sometime take months.

14. Sample complexity

For some problems, there are mathematically proven lower bounds on the number of observations needed.

15. Diminishing returns themselves

And of course, there's the idea that finding each new useful idea requires more and more research effort.

On the other hand, a recent paper showed that automating research can create positive feedback loops strong enough to overcome diminishing returns.

III. Possible Fundamental Limits16. Computational complexity

Some problems require massive amounts of computation in the worst case.

17. Computational irreducibility

For some processes, there may be no faster way to predict how they will evolve than to actually run through the process itself.

18. Chaos and limited predictability

Because of chaos, tiny uncertainties in initial conditions can grow very quickly. I'm not sure that even superintelligence could do a long-term predictions about certain physical systems.

19. Physical limits of computation

The Landauer limit, Bekenstein bound, Margolus–Levitin theorem, and the speed of light. These are all discussed in Seth Lloyd's classic paper.

Of course, if these are the only things left that limit superintelligence, our forecasts probably won't mean anything anymore. I'm including them as honourable mentions.

Discuss

In 1964 [1], I. J. Good wrote Speculations Concerning the First Ultraintelligent Machine. I've only ever seen it cited for a single paragraph:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an "intelligence explosion," and the intelligence of man would be left far behind ... Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. It is curious that this point is made so seldom outside of science fiction. It is sometimes worthwhile to take science fiction seriously.

This is becoming increasingly relevant, with the AI Futures Project now estimating an intelligence explosion starting in 2027 in response to recent AI progress in automating AI research. Making our 'last invention' relinquishes our influence over the future, we have to get it right, and with the current frantic pace disastrous mistakes are likely.

I think the appropriate reaction to this news is something like "this is completely irresponsible, you're way too likely to get us all killed, do not do that" and marching in the streets (hence yesterday's protest attendance). But as someone who is very interested in how people process ideas and what they thought with the information they had at the time, my reaction also includes wanting to know more about what Good expected six decades ago.

I spent this morning reading his paper and talking about it with Claude (Fable 5.2). I was curious how the ideas had held up, what he got right, and what he got wrong. Some of his ideas were very technical, and not being an ML expert I can't add much there; I've put these in a Claude-authored appendix.

The first sentence reads very differently now:

The survival of man depends on the early construction of an ultraintelligent machine.

I think he was saying that he was worried that without one we'd kill ourselves off (later he writes "it will give the human race a good chance of surviving indefinitely"). This was during the Cold War, a couple years after the Cuban Missile Crisis, and before any agreement to limit (let alone reduce) nuclear stockpiles.

But now I'd interpret it the other way around: our survival depends on 'early construction' the same way that a driver's survival depends on how quickly they drive.

Reading further, the most-quoted paragraph still jumps out, and not just through familiarity. Reading it closely, the "provided that the machine is docile enough to tell us how to keep it under control" is actually pretty prescient: he's essentially describing the plan of record, as much as there is one. Have AI research how to align AI. (This might work! But it also might not.)

He's also uncertain about whether it will go well. In the famous paragraph above he has a "provided that the machine is docile enough", later he says "the sign is uncertain" (math-speak for "could be good or bad"), and speculates whether "the human race will become redundant". He does seem overall optimistic, though, both in the 'survival depends' quote (above) and a few other places. For example, "the machines will create social problems, but they might also be able to solve them in addition to those that have been created by microbes and men."

A live objection at the time was that there wasn't any reason to make AI, and he cites B. V. Bowden's view that "it is easier to construct human brains by the usual method" before responding:

This shows that highly intelligent people can overlook the "intelligence explosion." It is true that it would be uneconomical to build a machine capable only of ordinary intellectual attainments, but it seems fairly probable that if this could be done then, at double the cost, the machine could exhibit ultraintelligence.

I'd actually count this as a bit of a miss: a machine capable of human-level work that was massively cheaper than humans could still be highly economical, and that's essentially what today's deployment of AI is. I suspect he didn't anticipate the extent to which high throughput would precede high intelligence.

I also don't see where his "double the cost" is coming from, and in practice this jump is likely to be much larger. But the critical thing he gets right is that human-level isn't a ceiling, and in a series of domains (games, coding, cybersecurity, math...) we've seen AI, after spending a while at "toy" and some time at "useful tool", quickly pass "human level" to get to something much more capable. I expect this keeps going.

On the technical side, most of the details are in Claude's appendix but at a high level it seems like he was right about a lot of how we got here. He thought it would be "vast artificial neural circuitry" and expected it would learn from experience and reinforcement:

Yet a detailed knowledge of semantics might not be required, since the artificial neural network will largely take care of it, provided that the parameters are correctly chosen, and provided that the network is adequately integrated with its sensorium and motorium (input and output). For, if these conditions are met, the machine will be able to learn from experience, by means of positive and negative reinforcement, and the instruction of the machine will resemble that of a child.

He also anticipated how text-based this would all be:

Whether a machine of classical or ultraparallel design is to be the first ultraintelligent machine, it will need to be able to handle or to learn to handle ordinary language with great facility. This will be important in order that its instructor should be able to teach it rapidly, and so that later the machine will be able to teach the instructor rapidly. It is very possible also that natural languages, or something analogous to them rather than to formal logic, are an essential ingredient of scientific imagination. Also the machine will be called upon to translate languages, and perhaps to generate fine prose and poetry at high speed, so that, all in all, linguistic facility is at a high premium.

He also correctly anticipated that AIs would use traditional computers (still then in their infancy) as tools, with coding and search:

In any event the ultraintelligent machine might as well have a large electronic computer at its beck and call, and also a multimillion dollar information-retrieval installation of large capacity but of comparatively slow speed, since these would add little to the total cost.

And he anticipated a period before AI fully surpassed humans, during which they'd work together:

Until an ultraintelligent machine is built perhaps the best intellectual feats will be performed by men and machines in very close, sometimes called "symbiotic," relationship

This is how many of us work daily, including both my day job and these blog posts.

He was also a strong chess player (on the Bletchley Chess Club during the war) and suggested:

so-called "randomized chess," in which the positions of the white pieces on the first rank are permuted at random before the game begins (but with the two bishops on squares of opposite colors), and then the initial positions of the black pieces are determined by mirror symmetry. This gives rise to 1440 essentially distinct initial positions and effectively removes from the game the effect of mere parrot learning of the openings

This is very close to Chess960 (though without the extra castling work) which Bobby Fischer proposed decades later, and I was initially pretty excited to see it. But reading more it turns out this was a known variant going back centuries. Good's including it here to illustrate that what we really care about is the reasoning component of chess, not memorizing opening lines.

Thinking about things Good got wrong, the timing initially seems like a big one: "more probable than not ... within the twentieth century". But he only put it at 50%, and while it has taken longer it looks like it's going much faster than most others predicted.

A much larger miss is the cost of artificial neurons:

If we could raise say a hundred billion dollars we might be able to simulate all the neurons of a brain, and of a whole man, at a cost of ten dollars per artificial neuron.

Since Moore's law is newer than Good's writing it's not surprising that he didn't fully account for it (though he did expect "great advances in microminiaturization"), but the hardware to power an artificial neuron costs ~10,000x less than this (~$0.01 in current dollars, less if you quantize; Good's $10 would be ~$100 today).

On the other hand, and more importantly, he's in the right ballpark on total cost: the world has spent maybe $1T to date creating AI systems and I expect we spend maybe that again by the end of 2027 as recursive self-improvement increases the returns to AI R&D. Good's $100B is $1T in current dollars, but note that he didn't expect anyone to pay this:

But it seems unlikely that more than say [$300M then; $3B today] would actually be forthcoming, and even this amount might be difficult to obtain without first building the machine!

This is indeed how funding has worked, with investors putting in more money in response to initial progress. That he ended up close to right on the ~$1T looks to me like less of a coincidence than it appears: costs come down over time, and it's only when they're getting into an economically feasible range that someone might actually do it.

The biggest miss, though, is about focus. He was super early in thinking about whether such a machine would be good for the world, but then spends almost the entire paper on how one might be designed. Focusing on the "how" and not the "whether" is very normal for technical people then and now, and as an engineer I feel this pull as well. But it's something people have been getting wrong since the beginning: excitement about the potential, not staring hard at the true scale of the risk.

This paper isn't the last we have of Good's thinking, however, and in 1998 he provided a bio to the IEEE on receiving their Computer Pioneer Award. It doesn't seem to have been used at the ceremony or published, but James Barrat says Good's assistant Leslie Pendleton gave him a copy, and Barrat quotes it in his 2013 Our Final Invention. Note that this is Good writing about himself in the third person:

"The survival of man depends on the early construction of an ultra-intelligent machine." Those were his words during the Cold War, and he now suspects that "survival" should be replaced by "extinction." He thinks that, because of international competition, we cannot prevent the machines from taking over.

How to get countries to coordinate to prevent the machines from taking over is a key question today.

Appendix: Good's technical predictions I asked Claude Fable 5.2 to write an appendix. The judgments are its own, and I haven't checked them:

Most of Good's paper is a theory of how a neural network could store meaning and recall it. Below I go through its main technical claims and say how well I think each matches how current AI systems work. I grade each as a strong match, a partial match, a loose match, or a miss.

Two cautions first. A 58-page speculative paper makes a lot of claims, and picking out the ones that look right in hindsight flatters it. And Good was less of a lone voice than this list might suggest. Neural networks were an active research area in the early 1960s: he cites Rosenblatt's perceptron work, and builds on Hebb, Ashby, and Shannon. His bet was on one of the two live approaches, the one that then went out of fashion for decades and eventually won.

Learning is adjusting connection strengths. Strong match. In his radio-linked machine (p35, p36):

The strength of the connection could be represented by the accuracy of the tuning. ... "Positive reinforcement" would correspond to improved tuning of these filters. Every connection has an adjustable strength, and learning means changing the strengths. That is what a "weight" is in a modern network, and training is nothing more than adjusting weights.

How the strengths get adjusted. Miss. Good's rule (p80) is that a connection has a small random chance of stepping up in strength whenever it is "successfully used (i.e., contributes to the activation or inhibition of an artificial neuron)". This is a local rule: each connection changes based only on its own activity. Modern networks are trained by backpropagation, which works out how every weight in the network contributed to an error in the final output and adjusts them all together. Nothing in the paper anticipates this, and local rules of Good's kind have not been made to work at scale.

What "reinforcement" means. Partial match. Jeff asked whether Good's "positive and negative reinforcement" from an instructor is what is now called reinforcement learning from human feedback. It does resemble it, and that technique is used to shape how current models behave. But most of what those models know comes from a different source, predicting the next word across a vast amount of text. Good's usage is broader than reward from an instructor. The rule above involves no instructor, he says the machine is educated only "partly" by reinforcement (p80), and his account of the child is about exposure:

The child later associates words with objects and activities, by implicit statistical inference ... At a later stage of linguistic development, similar statistical associations are developed between linguistic elements themselves. (p45, p46) He has the right raw material, the statistics of which words occur with what, and he discusses predicting words from the preceding ones (p43, p54). He never proposes prediction as the thing to train on.

Words are known by the company they keep. Strong match, though not original to him.

The essential idea is that words can be seen to be associated not merely because they occur frequently together, but because they both occur frequently in conjunction with a third word (p71) His example is that "ferry" and "fare" are related because both go with words like "bus". This is the basis of how modern systems represent word meaning. Linguists (Harris, Firth) had stated the idea in the 1950s. What Good adds is the argument that a machine needs it, since machines that "rely only on frequencies" (p71) can say nothing about combinations they haven't seen.

Word vectors. Partial match. On p52 and p53 he takes a table of how often each word occurs in each document, takes logarithms, subtracts a constant from each row and column, and proposes decomposing the result so that

This would associate a w-dimensional vector with each index term and with each document. The relevance of index term i to document j could now be defined as the correlation coefficient between the two associated vectors. This is recognizably the method later known as latent semantic analysis (around 1990), and a relative of the word-embedding methods of the 2010s, which were shown to be doing something close to factoring a similar table. Two things are missing. He keeps as many dimensions as there are words, where the later methods get their power from keeping only a few hundred. And the vectors are computed once, not learned by a network. He also offers it only as "another conceivable method" and judges the computation "exceedingly great".

Generalization is the central problem. Strong match on the diagnosis, partial on the cure.

One of the main difficulties in this subject is how to estimate the probabilities of events that have never occurred. That such probabilities are relevant to intelligence is to be expected, since intelligence is sometimes defined as the ability to adapt to new circumstances. (p32) This is still the right way to state it. Good had worked on this problem with Turing, and the resulting method (Good-Turing estimation) was a standard component of statistical language models into the 2000s. Current models don't use it. They generalize by learning that similar words and situations behave similarly, which is closer to his "ferry" and "fare" idea, and to his remark (p71) that getting the network to form abstractions is "one of the most vital problems".

What comes next depends on recent context and stored weights. Partial match. Good's unit of thought is the "assembly", Hebb's term for a group of neurons that fire together and stand for a word or idea.

Which assembly becomes active at the next moment must depend on the current sensory input, the current dominant assembly, and the currently active subassemblies. Indirectly, therefore, it depends on the recent assembly sequence, wherein the most recent assemblies will have the greatest influence. It also depends of course on the semipermanent static storage, the "past history." (p60) A language model has the same division: what comes next is determined by the recent text it is looking at (its "context") and by weights fixed during training. Good's list of memory types (p72) puts short-term memory in current activity and long-term memory in connection strengths, which is the same split, and he says a long "attention span" is "certainly essential for intelligence" (p63). He goes on (p66) to score each candidate next assembly as a baseline plus a contribution from each recent one, with the highest total firing. The last step of a language model has that form, a score for every possible next word. But in a modern model the contributions from earlier words interact in complicated ways before that step, and Good only gestures at this by saying interaction terms will be needed (p67). I wouldn't claim more than a family resemblance.

Concepts are spread across many neurons. Strong match, shared credit.

each neuron would occur in a great many distinct circuits, and would not be indispensable for any of them. (p56) He contrasts this with "one concept, one neuron". Modern networks store concepts as patterns over many units, with each unit taking part in many concepts, and this is a main reason their workings are hard to read off. The idea came from neuroscience (he cites Lashley and Eccles). Good's contribution is an efficiency argument for it, by analogy with a punched-card coding scheme.

Keep probabilities, not hard decisions. Strong match. Section 3 argues that a system understanding speech should pass along graded probabilities between its levels instead of committing to one answer at each, because "each definite 'decision' loses information" (p39). Modern networks pass continuous values between layers and only commit at the output.

Rejecting a logical "canonical language". Strong match. He considers representing meaning by translating every statement into a standard formal form, and concludes this would be "extremely difficult, perhaps even logically impossible, or perhaps it would require an ultraintelligent machine to do it!" (p41). Large projects later attempted roughly this and did not get far. Relatedly, his guess (p36, quoted in the main post) that natural language "rather than ... formal logic" is needed for scientific imagination is a partial match: current models do much of their reasoning by writing out steps in ordinary language.

Scale can substitute for design. Partial match.

a well-designed machine of say 109 components might be educable to ultraintelligence, but a much more carelessly designed machine of say 1013 components might be equally good. (p62) The history of the field broadly supports this: simple methods given more computation have repeatedly beaten cleverer ones. But Good, following Ashby, bases it on the hope that almost any large, richly connected network will behave intelligently. That turned out to be false. Scale paid off only once there was a training method able to make use of it.

The hard part is ape-level intelligence. Partial match. He says "most of the struggle" will be building "a machine with the intelligence of an ape" (p45), because perception and basic learning are harder than what comes after. In favor: getting networks to learn to see took until about 2012, and fluent language followed within a decade. Against: language arrived without anything like an ape's physical competence, and robotics is still behind.

Sleep and replay. Partial match. His conjecture that sleep serves "to replay the assembly sequences that were of greatest interest during the day in order to consolidate them" (p80) is close to a well-supported view in neuroscience. In AI, replaying stored experience, with priority to the most informative parts, is a standard technique in systems that learn by trial and error. It is not part of how language models are mainly trained, so his suggestion that the machine would "benefit from periods of comparative rest" (p72) is a loose match at best.

Understanding as economy. Loose match. He argues that storing meanings instead of exact wordings saves a great deal of memory, and that this may be "the evolutionary function of meaning and understanding" (p77). There is a real modern parallel, since learning to predict text well and learning to compress it are mathematically the same task. Good's argument is about storage, though, and I think the connection is more suggestive than exact.

Hardware. Mixed. He was right that serial speed alone wouldn't do. He expected computers to become about a million times faster than neurons and still concluded the machine "will need to be ultraparallel" (p35), and current systems do depend on massively parallel chips. He was right that low power consumption "will be valuable" (p33). He was wrong that the neurons would be physical devices. They are simulated in software.

The brain as blueprint. Mostly a miss. The paper opens by saying that to design the machine "we need to understand more about the human brain or human thought or both", and later that brain anatomy is "very relevant" (p63). Current systems borrow the neuron and the adjustable connection and almost nothing else. Of the brain theory itself, Hebb's assemblies remain a respected idea in neuroscience, but the central control system Good relies on (taken from Penfield) has been abandoned, and his own extensions haven't been taken up. To his credit, he says the designer will have to guess which features of the brain matter, and he allows that the first machine might not use a neural network after all (p78).

Theory before experiment. Miss.

The need for further theory is great, since, without advances in theory, the amount of experimentation might be prohibitive. (p64) Computation became cheap enough to experiment at enormous scale, and the systems we have were found that way. Theory explaining why they work is still well behind.

How much the machine needs to learn from. Miss. He sizes the machine's education to a person's: "A man cannot learn more than ten million statements in a lifetime" (p36). Current models learn much less from each example than people do, and compensate with volume. By my rough estimate they are trained on something like ten thousand times more text than a person could read in a lifetime.

Overall. Good was right about the kind of thing the machine would be: a very large, parallel neural network that learns the meanings of words from statistics, stores them as distributed patterns, treats generalization as the core problem, and works in natural language with conventional computers as tools. He was wrong, or silent, about how such a network could actually be trained, which is the part that took the field another fifty years. I'd put his reputation for foresight on these pages at least as much as on the famous paragraph.

[1] I generally see 1965, including on Wikipedia probably because of the publication date. But I think it's more informative to count by the date he wrote it: "The first draft of this monograph was completed in April 1963, and the present slightly amended version in May 1964."

Discuss

One Saturday night this summer, I was leisurely walking down main street hand-in-hand on a date, when suddenly, I heard a commotion about 50 yards down a busy side street. It was too far away to make out what was happening or what was being said. But I figured it was fine since the side street had plenty of people walking up and down it.

Still, I couldn’t help but peer into the distance, transfixed, like a deer in the headlights.

There was an anime convention going on that weekend. Initially, my brain reasoned that a couple of cosplayers were recording a TikTok video or doing a K-pop dance. Then my brain reasoned that surely, on a crowded street, nothing serious could be going down. Right?

Yet I waited there, unmoving, alert.

My date was casually standing there eating a cookie. She innocently asked, “What do you think’s going on over there?”

I faintly heard a woman yelling. While squinting, I saw sudden, physical contact between her and a man. I dropped my date’s hand and began aggressively stomping towards them. 

All I could see was red.

I used to have anger problems during my adolescence. I didn't realize that maybe I was different from other people, until one day during my freshman year of college…
 

…I opened the door to my apartment to find a roomful of friends. One of them stood up from the couch to give me a hug with her arms outstretched, one hand high and the other hand low. I instinctually flinched, and then leaned in to give her a hug, but she pulled back and stopped. 

“I noticed that you always flinch before I give you a hug,” she said. “How come?”

I thought about it for a second, then copied her—one hand high and the other low—then nonchalantly said, “Your high hand? That’s probably the angle from which my dad hits me.”

Her face dropped. I looked around. Everyone else’s side conversations had stopped and they were all staring at me. I noticed that I was confused—did I say something weird? My friend had asked me a question and I had matter-of-factly gave the answer. I had no idea why everyone was being so weird. I inquired, “What?”

My friend, the one who got up to give me a hug, looked at me sympathetically. “Your dad hits you?”

I noticed that I was confused again. They care about that? I gestured to the group of wide-eyed people and asserted, “What, your dads never hit you?”

Everyone stayed frozen. Then my roommate on the couch softly said, “No dude. My dad’s never hit me.” Slowly, others nodded their heads in agreement.

I felt a note of disgust. I didn’t need their pity. I retorted, “My dad’s not a monster. He doesn’t just randomly hit me for no reason. Only if I do something stupid or disobey him.” Feeling satisfied at successfully defending my dad and at clearing up the matter, my satisfaction turned into an admixture of confusion and anger—they were still just staring at me in stunned silence and pity.

“Whatever.” I went into my room and shut the door.

He was kicking her. The man was kicking her. As I stomped closer to them, my anger continued to build into a white-hot fury. My fury was only slightly abated by a thought that confused me: it’s a crowded street, there’s even a group of dads walking by and laughing with each other, so why is no one doing anything? Never mind that. My anger was building into an unstoppable rage.

Thirty feet from the assaulter, I did something that I haven’t done since I was a child—I screamed at the top of my lungs:

“HEYYY!!!”

The attacker whipped his head towards me in confused fear. I stormed forward and screamed:

“GET THE FUCK AWAY FROM HER.”

I must have had the twisted face of a rabid dog; the whites of the man’s eyes shined like saucers.

The woman screamed, “He’s trying to rape me!”

After that day in college, I reflected on my friends' reactions. Really? Their dads never hit them? Then how did their dads maintain their authority if not through fear? I couldn’t fathom an alternative method. So I shrugged and wrote them off as probably being weak.

Going forward, not much changed. I continued to exert my will over my four college roommates through intimidation and by yelling at them. Throughout my freshman year of college, one-by-one, my friends stopped coming by my apartment to say hi, stopped returning my texts, and were increasingly "too busy" to hang out when I ran into them on campus. I never felt lonely. No, I was too prideful for that. All these people just sucked. I just needed to find new, better friends. But new people, for whatever reason, didn’t stick around for too long.

This went on into my sophomore year of college. It went on until the day I had a bad trip while taking psychedelic mushrooms…
 

…So there I was, sitting on the ground of my university squash court with my friend, and beginning to hallucinate. At first, the spotted wood of the court looked like ants marching towards me, which was amusing.

It was amusing until, suddenly, a demon materialized in my stomach, crawled up my spinal cord, and invaded my mind. The most violent, envious, prideful thoughts spewed out of me. While it was happening, I believed it was the demon saying these things, not me, therefore not my responsibility and not my problem.

The next day, however, I realized that the demon was a manifestation of my innermost insecurities. That what the demon spake wasn’t gibberish nonsense, but instead a prophecy of what was to come if I didn’t get my shit together.

That experience scared me straight. And ever since that day seven years ago, I’m very proud that I never lost control again. I never yelled at anyone ever again. Until…

The man ignored me and started kicking her again.

Incensed, I advanced towards him and roared even louder, “HEY! GET THE FUCK AWAY FROM HER.” The man saw me stomping towards him and he stuttered backwards. Perhaps he saw in my eyes that I wasn’t fucking around, that I wanted blood.

They say that in response to immense stress, people typically have one of three reactions: fight, flight, or freeze. I was ready to fight. But it hasn’t always been like this…

As a child, I sometimes found myself on the other side of the kitchen table, playing a game of cat & mouse with my angry older brother, tears and snot streaming down my face, squealing like a cornered pig about to be slaughtered, while trying to avoid my brother’s grasp. For I knew if he caught me, he would wrestle me to the floor, pin my arms underneath him by sitting on my torso, and then rain blows down upon me.

Over the years, after repeatedly being hit by my father, by my mother, and by my brother, I learned that flight was not a good strategy for me. I was the youngest and the weakest. I was never going to be able to outrun them or outfight them. Instead, I figured out that it was far better to freeze, study what triggered them, and then learn to tiptoe around those landmines. 

And this worked. I learned to suppress my emotions while living with my family. It was the only way to assure my safety.

When I moved away to college, I turned into a monster. All my pent-up anger was unleashed upon my innocent roommates and friends. And it felt good. Fuck, it felt so good to lean in to my newfound rage. I got to feel powerful for once. I got to feel in control. And it worked…until it didn’t. One-by-one my friends left me.

 

After the psychedelic trip, I underwent a years-long healing journey. I read dozens of EQ, therapy, and self-help books. I wrote thousands of pages across my journals. I laughed, I cried, I got in touch with my emotions that I had suppressed my whole life. And then I decided to get strong.

In the latter half of college, "The House of Iron" became my place of worship—every day I was in the gym at 6am for two hours. In the span of ten months, I went from barely being able to lift a barbell, to deadlifting twice my body weight. I put on 10-15lbs of muscle, and when I looked in the mirror, instead of seeing a scared kid, for the first time in my life I saw the body of a man looking back.

 

The last time my whole family lived under the same roof was my senior year of college. My brother and I had been getting along for a while, becoming friends for the first time in our lives. I thought there was mutual respect between us. 

I was wrong.

One morning the dog ran down the hallway loudly, and in the process, accidentally woke up my brother. Watching it happen, I began chuckling at the ridiculousness of it. My brother burst out of his room and saw me quietly snickering. He assumed that I had deliberately woken him up to mess with him, and without asking questions, sprinted towards me and unleashed a barrage of punches. I froze. 

Then after a moment, I realized that his punches didn’t hurt. Not anymore. I had gotten stronger since the last time he struck me. 

He kept hitting me and accusing me, “You woke me up! You woke me up!” I grabbed both of his arms, stopping the assault, and told him, “It was the dog! It was the dog! The dog accidentally woke you up, not me!” He stopped thrashing, stared at me with suspicion, then shrugged and went into the kitchen to go make coffee.

I just stood there, stunned. Then I went into my room and closed the door. Five minutes later I joined my first Zoom class for the day. During that call, I silently processed what had just happened. I felt confused, shocked, angry.

After my class, I went looking for my brother, and found him in the backyard by the pool on a lounge chair sunbathing in nothing but his boxers, peacefully reading a book.

“Hey,” I called to him.

He casually turned his head towards me.

“Like I said this morning,” I stated in a cool, even tone while maintaining intense, unblinking eye contact. “It was the dog that woke you up, not me. And without thinking or asking clarifying questions, you attacked me. Look at me,” I demanded while pointing at my eyes. “I’m bigger than you now. I’m stronger than you now. And if you ever put hands on me again, I will fuck you up. There will be blood.” 

He looked at me and saw that I wasn’t fucking around. I dialed my intensity down, smiled, and then told him, “have a nice day.”

I wasn’t a bystander, not anymore.

The man was conflicted. He wanted to continue kicking this woman, but he also saw the bloodlust in my eyes. He tried reasoning with me:

“She pepper-sprayed me!”

“I don’t give a FUCK,” I replied. “Get the FUCK away from her.” 

The man teetered on his backfoot, unsure of what to do. And while he was doing that, I was mentally preparing how I was going to incapacitate him.

Had my half-naked sunbathing brother actually gotten up to fight, it would have been ugly. Neither of us are trained fighters. But ultimately, the threat of me fighting back for the first time was enough to deter all future conflicts. He gained a kind of respect for me that day. While we've had our ups and downs in the years since then, today we’ve never been closer as brothers. He's a good guy.

But had we fought that day, it would have been a mess of sprawling limbs. But that wouldn’t be the case today because…
 

…Last year I made a new friend at a coffee shop—an older gentleman. What an amazing and wise individual. In his youth, he was a hothead like me, and he also grew up with a brother that beat him.

But unlike me, as a young adult he got into dozens of street fights and learned how to defend himself through trial by fire.

One day, months into our friendship, he asked me, “Do you want to learn how to fight?”

“Yes.”

And thus began my training.

 

Punches. Kicks. Blocks. And also the psychology of fighting—people, especially untrained fighters, are predictable. The most common punch thrown from the untrained fighter is the overhead haymaker. My friend taught me how to block that—turns out that the wax on/wax off technique from Karate Kid actually works—and then how to counter with a series of devastating blows. 

An older man who instills wisdom, philosophy, and the means to defend myself. I had met my Mr. Miyagi.

At my gym, I began using a punching bag without any gloves to toughen my hands. The first time I did bag work, I whiffed a roundhouse kick and landed hard on my ass. I also walked away with nasty bruises up and down my shins, arms, and fingers. 

But over time I got tougher. When I would visit my friend, he would surprise attack me to try to catch me off guard. And gradually, my reaction time got better.

Over the course of a few months, I developed a newfound confidence when walking around downtown. It felt like I was armed. And I was. My body became acutely primed to defend itself. I was now ready if a fight ever went down…

When he charges, I thought, I’ll block the first punch, and counter with an elbow strike to his ear with all my force to go for a knockout blow.

I took a step forward, my eyes subconsciously communicating: please, enter the arena.

The man took a step back, scowled, and slinked away from the woman he was attacking.

I followed him.

I lived with my parents for a few years after college to save money. Before I moved out, I had one goal.

After dinner one night, I announced to my parents, “I’m pretty emotionally fucked up. And so are both of you. I’m going to singlehandedly carry this family’s trauma and stress on my back until we’re all healed.” They mocked me for making such a grandiose declaration, but I didn’t care. I didn’t know how I was going to fix us, but I was fully resolved in my ability to do so.

As I read more books and had emotionally challenging conversations with friends, I took what I learned and practiced at home. One night in particular stood out to me.

I don’t remember what they were arguing about, but it didn’t matter. What mattered was that my father was yelling at my mother. And I was fucking pissed about it.

“HEY,” I exclaimed.

My parents looked at me, probably expecting me to complain about the noise they were making when I was trying to work. Instead, I furiously pointed an accusatory finger at my father and loudly asserted, “You’re not going to yell at my mother again.”

My father looked at me with a blank expression.
 

We all just stood there in silence.
 

I narrowed my eyes. “Got it?”

My father, not breaking eye contact with me, said “yeah” then walked away.

Had my father—who is bigger, taller, and stronger—challenged me and tried to intimidate me, I would have had no choice but to back down. Maybe he saw the resolve in my eyes, and in a moment of self-reflection, realized how he was treating his wife. In that moment, perhaps he overcame the rough childhood he endured and chose peace.

I wish I could say that everything immediately got better after that night. But it didn’t. Lots of hard conversations ensued over that year. Tears were shed. Yelling.

I made plenty of mistakes, too, allowing my ego to get the better of me. The road was rocky, but I ultimately kept my promise: I carried my family out of the dregs of our emotional abyss by repeatedly modeling good behavior, choosing the right time to initiate difficult conversations, and believing that I could even do this in the first place.

Today, my parents and I have never been closer. I’m proud of them, and my brother. They all put the work in to become better versions of themselves, and they deserve the credit for that. They're not perfect (far from it), but they've come a long way.

It’s been seven years since the last time I lost control, when the psychedelic demon took over my mind. And now I’m following an attacker.

The man, sporting the craziest of eyes, was huffing and puffing, “She started it! That crazy bitch pepper-sprayed me!”

“FUCK off,” I fumed.

The man walked alongside the group of dads I’d passed earlier and tried to fraternize with them, “That bitch is crazy! Did you see that?? She’s crazy!” The dad closest to him nodded sympathetically with a half-smile to appease him.

I continued following the man up the street. Any time he turned around to gripe about the woman, I shouted, “FUCK off.” We got to the next street intersection and I watched him walk away while muttering to himself. 

I wasn’t a bystander, not anymore.

 

When he got a far enough distance away and appeared to not be returning, I remembered what I was doing that night—I was on a date.

I turned around. My date was watching me. She was eating a cookie.

Weeks later, I sat down to write this story alone in my apartment. Remembering everything that happened that day, I began to relive the experience. My heart started beating out of my chest, I broke into a sweat. I felt so, so fucking angry.

What I still didn’t understand was: on a crowded street, a man was openly assaulting a woman, yet people just watched and moved along—why did no one else help? While writing this story, I sat back in my chair and stared out the window, waiting for the answer to come to me. 

And then it did.

 

 

The man, and the woman he was attacking, were both homeless.

 

 

It’s been over two years since I last cried. 

Sitting at my desk thinking about all this, overwhelmed with emotion, I began to open-mouth weep. I covered my face in shame (because men aren’t supposed to cry), I recognized how silly this was given that I was alone in my apartment, gargled a wet laugh at the irony, then I wept.

I was thinking of Teresa.

 

 

Part 1: I would have shit in that alley, too
Part 2: A City Within a City
Part 3: Assaulted
Final Part: Not just a bystander, not anymore…

Discuss

Prologue

I stumbled into Zen a little over a year ago. One of the things that initially got me into meditation was reading some rationalist accounts of people’s first retreats. They said it was valuable, but also that it was hard in a way they didn’t want to forget or diminish. But they didn’t say in what way it was hard or what it was like. So I thought it time to pass it on, by sharing what my year of retreats has been like. (Spoiler: it was hard.)

Two Weekend Retreats

I first found a local group that looked promising. I had wanted to sit with others; I had a vague sense that this would be a good practice. I had recently read Henry Shukman’s memoir One Blade of Grass; I had a vague sense that Zen might be interesting. I showed up for the introductory hour and was shown all about how to bow when entering the zendo, how to get up from the zabuton and the different bells, and so on. The zabuton is but a square meter of thin black foam you get to rest your body on. All legs (and arms) inside the square: they lie tight, one after the other; keep the front clear for walking. A lot of practical details and a little on how to sit: like, keep your eyes open, resting on the floor in front of you; your hands resting in your lap, left over right, forming an oval. This is called zazen. In the zendo you all sit on two rows along the walls, nothing like the classroom style layout I had been imagining. Shukman never mentions this! There are officers with different roles, they sit at both ends of the rows and help run things. The zabutons are black, the cushions are black, you’re asked to dress in dark or preferably black clothes. Everything is austere and made of unpainted wood. It’s beautiful in a way. In Rinzai you face the people sitting across from you, everyone in towards the room. At the front there is an altar with a small buddha statue. But it’s not the historical Buddha, Shakyamuni, we’re told. It’s some other figure, a Buddha of Wisdom or something. He’s considered one of the practitioners, sitting with us facing the room. He gets served tea first during tea ceremony. Then at the end when I was expecting meditation instructions, the guy in the robes told us “well we haven’t got much more time, so let’s pretend this is the last six minutes of zazen.” Then he rang the bell three times and we sat in silence until he rang the bell again six minutes later, and with two wooden clappers, clacked twice—and we got up. That was it.

I signed up for what would be my first retreat, a weekend zazenkai from Friday to Sunday. Before that I figured I better go to one of the weekly sits and get a better feel for what it was actually like. Luckily I was able to make time for it right before leaving for vacation on the Thursday afternoon of the national election on my way to my fiancee’s party’s party. The shoes were all neatly stacked in little rows at the entrance so I figured I better do the same with mine. I bowed on the way into the zendo, found a spot on the next available zabuton so as to leave no gaps, bowed again in front of it, and tried to do everything else I was told. I sat quietly and gazed across the floor towards the other people when the bell rang three times, and as the sound of the last strike faded, what replaced it was silence. Anxiety gripped my heart. It flooded my chest, whole body really. I was to sit, silently and without moving, on this cushion for the next 25 minutes. My heart pounded so hard I could not breathe through my nostrils, I had to sit there with my mouth hanging open; panting. Gasping for air. I wanted to get up, run out of there. So, so badly. I closed my eyes. I don’t remember how long this went on for, but after a while my body relaxed enough that I didn’t have to awkwardly bail—and eventually, the first sit was over. We got up and did kinhin: a slow walking meditation in a long lockstep line, as a five minute break, then there were two more sessions of sitting. These went much better, and I felt mildly better about the whole thing. But I was still left with the feeling: oh my God, what have I signed up for. 48 hours of this?

I sat religiously every day for twenty minutes during the following vacation leading up to the zazenkai. I got off the bus and walked for ten minutes up a gravel path through some fields to a farm house. One of the regulars at the center had renovated an old barn to be this beautiful zendo, there was a row of those prefab boxes they put up for construction crews where we would sleep, and his family was leaving so we could have the main house at our disposal for cooking, eating the shared meals, and resting between sittings. After an introduction in the zendo for the newcomers, we shared an informal meal in the main building. I immediately hit it off with a guy, same age and occupation, whom I’ll later come to know as Oryō, also at his first Zen retreat. Standing in line at the serving table he commented “look, there’s a bowl of parmesan, how fancy.” We both took too many spoonfuls of what turned out to actually be salt on our stew, ruining it. At the dining table we were kind of laughing, kind of embarrassed to have been undone by our own greed. The teacher for the retreat was a very serious-looking German man called Shingen. He lived at a monastery in the US for years under a Japanese Zen master, and now spends five months of the year at one known in Japan as “the devil’s dōjō.” He sat down next to us and announced loudly, “I thought it was parmesan,” pointing at his bowl of ruined stew. The two of us exploded laughing. I’m still not sure he knows why.

Shingen joined our laughter and soon the whole table was laughing with us. A shared laughter of giddy excitement and trepidation.

The first third or so of my stay was great. The stress and agitation of everyday life gradually thawed. I sat with a tea on the porch in the sunny, if cool, autumn weather. I discovered that when you can’t talk, or even have eye contact—you’re not supposed to really communicate at all—it cuts off a lot of the stress of social interaction that can make things awkward. But there still is interaction: I’d hold out the tea kettle after pouring my own water, and the next guy would take it. Or he would not, either way neither of us would thank the other, or refuse, or signal. I found it strangely intimate and liberating: I felt I couldn’t really be rejected; either my offer was accepted wordlessly, without acknowledgement, or it wouldn’t be—and that was also fine. As I sat there I found I could enjoy my tea on the porch with the other person, in a way made impossible if we were to chit chat.

On the first morning I woke to the sound of people talking outside. The bell is broken, I overheard. No wake-up call then. I got up and began putting on my clothes in the darkness. Why is no one else getting up? Wait, the bell is broken? It was completely silent. I dug through my bag to find my watch, which I’d stowed away with my phone and other worldly belongings when I got there—and wasn’t really supposed to use during the stay—to find out it was barely past midnight. Lights out were at nine in the evening, so I might have gotten a few hours’ sleep—it’s not easy to fall asleep so early. Should I go back to bed? How will they wake me without the bell? I can’t stay up for another four or so hours. The bell can’t really be broken, can it? I tentatively lay down again to consider my options. Before I knew it I awoke to the sound of the bell.

Despite a rocky night with little sleep, by the time we were sitting in the near darkness of the early morning zendo a little past five in the morning, I was feeling wide awake. As a night owl, I have had a lifetime of rough mornings and usually don’t feel fully awake until far into the day. But here it was different, I was not just not-sleepy, I was shockingly, staggeringly awake and full of energy, which would last the whole rest of the day—the whole retreat, and even a while after I returned home. I was a little worried going in, that the complete silence and the ten hours a day of sitting would be tough somehow, but they were fine. I enjoyed it. There was a pleasantness to bare existence.

At one point I could see the floor breathing. It was pulsing up and down, but not in sync with my breathing. I held my breath to check; it continued on undisturbed. At another point, time appeared to stutter.

Then my legs started hurting. That’s another thing Shukman doesn’t mention. Your legs fucking hurt! Like the most excruciating pain of your life hurts. There’s something that happens to your body after sitting crosslegged for hours at a time, where the urge to move slowly ratchets up, and a searing pain starts to build. It is like your body is, at first politely asking, then yelling and eventually screaming at you. It turns up the pain until it makes you move. But the thing is, you’re not supposed to move. Certainly not get up until the bell, but you’re not supposed to even readjust. This I would come to know might be the whole of the Zen teaching. The pain steadily dialing up, you not being allowed to move—or not allowing yourself to move—until the bell. But something has to crack. Something will crack. I’ve been sitting since before dawn, I have no idea what time it is, except that the sun has gone down again and we’ve already had four rounds of sitting in a row—there can’t possibly be another round of sitting. The guy in the robes across from me sees me suffering and announces something encouraging, addressed to no one in particular. I don’t catch a single word. The bell does eventually come, but I wait for the clacker. Only one clack. That means we’re doing another round of kinhin, and after that, at least one more round of sitting. Five in a row this evening. I want to cry.

The next morning is better. My legs are rested after a night’s sleep and they don’t hurt as much; I’m able to be more clearheaded. When I scan the sensation in my legs it doesn’t seem so bad; the badness of it is hard to find. But it does feel bad and there is suffering. When I try to locate the suffering, I find that I can’t. It’s ambient, almost like it’s in the air. How I can even know the suffering I’m experiencing is caused by the sensations of pain is not at all clear. Weird.

The informal meal afterwards is great. Everyone’s been through something, although no one quite sure if it was the same as the others. The next two retreats I go on have the same fight. I barely remember any details of the sitting during my second zazenkai. Whereas the first one had rolling hills covered in stunning morning mist, the second was held at a rented chapel in the city. There is no accommodation this time, so I have to take the train in and out of the city every night and morning. In my memory it’s only a smeared weekend of miserable pain. At one point my posture has completely collapsed, my body is hanging on its ligaments, drooped over itself, legs splayed out and can barely even be called crosslegged. It gives me some way to rest despite the pain. I desperately try to “accept” the pain. If you stop wanting it to go away, apparently it will go away. A woman across from me is crying silently. It seems cathartic. When she wipes her tears with a handkerchief, the jikijitsu shouts from his spot at the far end of the room to leave it. I’m too empty to cry. I drift in and out of sleep until the bell. I stagger up, using more arms than legs. I stand there swaying, waiting to be able to walk again. Everything hurts and I’m exhausted. Waiting on the train Saturday evening with my fiancee who had been in the city with friends, I remember telling her: if I’m ever doing anything like this again, I’m asking for a chair.

Sunday morning I’m back on the zabuton. It’s still miserable. You get three meals a day, which is the only thing I look forward to. When the jikijitsu claps twice to indicate the session of sitting is over we get up with our bowls held in both hands and walk in two lines to a single long table, so that everyone sits in the same order as on the zabutons. They send a large pot of food down the middle and each person grabs however much they want. It will be served twice, so you can better judge how much you want, but at the end it’s expected that you finish everything you’ve taken. Then they send down the tea, which you use to wash out your bowl with your fingers before drinking it. Sitting in a regular chair and eating this hot, nourishing food is heaven. I eat slowly and carefully, mindful not to miss a single bite. A woman next to me rests her spoon in her bowl when taking a break from eating. They never tell you all the rules. I’ve never seen anyone do that: she’s going to get called out. She gets a minute. The jikijitsu is generous today. The spoon is still there. From the far end of the table the jikijitsu announces to no one in particular that the spoon should lie before you, between the bowl and the table edge, when resting. I realize I’ve taken too much of the dry, crunchy flatbread we got for this meal; everyone else is finished and now we’re all waiting for me. I chomp it down as quick as I can. Crunch. Crunch. Crunch. Everyone is listening. I don’t mind, it’s kind of a funny situation: until I run out of saliva. My mouth is full of what is beginning to feel like dry cement. I can’t chew it and I can’t swallow, and I still have half a brick left in my hand. I rest the brick in my bowl, lower my head in shame, mouth full of cement, and hope for the best. They start sending the tea down the table. I don’t know if the jikijitsu has taken mercy on me or hasn’t noticed I’m not actually finished. When the tea arrives I fill my bowl, get as much of it into my mouth as possible, and soak the brick in the rest. I’m finally able to chew again. I manage to finish the soaked brick and the tea before anyone calls me out.

Short Sesshin

Come winter there is go-sesshin, which is the short version of the summer sesshin. This one is from Wednesday to Sunday deep in back country woods in the dead of winter, the snow dense and hard. There are red, white, blue, green and yellow prayer flags hanging between the trees. This is a Tibetan monastery, or maybe hermitage. There’s only a single nun living here. Tibetan Buddhism isn’t very big in this country; I’m surprised we have a living monastery at all. Where she’s gone I don’t know, but apparently she’s thankful people are using the place for practice. It rents out the cabins dotted around for retreats, and we’ll be using the main building as the zendo. The zabutons are laid out in their usual two neat rows with the altar at the end and a long table improvised at the other for the meals. Everything will be heated by a single fireplace a few feet from the table. Instead of the usual wooden floor, the zendo is carpeted with this ornately detailed red Tibetan carpet, which I will be staring at for the next five days.

My fiancee drives me into the woods and drops me off. She’s on her way to a Christian silent-prayer retreat I found for her so she won’t be alone while I’m here, and secretly to introduce her to the joys of retreat life. After the introduction I go to my cabin and find Oryō there. By some fortune we’ve been assigned to be cabin mates. He explains he was late because he took a cab from the train station to the nearest Tibetan monastery; it turns out we have two of them in the same municipality. The other one has a monk.

I take the couch and Oryō sleeps on the bed in a tiny room without a door. We comment on the cabin’s tiny fireplace, which we won’t have time to use. We are wrong. It’s been less than six hours since I arrived and I’m lying in my sleeping bag wondering if I’ll even make it through the night. Luckily I’ve brought a thick woolen blanket. Even more luckily, the cabin has an even thicker one. Despite sleeping in every layer of wool I have brought, from long underwear to my thickest sweater, and both blankets inside my sleeping bag, I’m shivering the entire night. It’s the coldest I have ever been. I contemplate whether I have to take my sleeping bag and go up to the main building and sleep on the zendo floor in order to survive. But that building probably isn’t heated at this time of night either. It’s completely dark outside, and the inside of the cabin is now the same temperature as the outside.

I barely get any sleep at all. I wake in a confused stupor as someone is barging into our cabin. They’re telling us to wake up and then they leave. Oryō stumbles out of his tiny room and gets going before I can make any sense of the situation. Does he know what’s going on? He is supposed to help out in the kitchen this retreat, maybe they came to get him? But he didn’t seem any less confused than me as he scurried out the door. Again I rummage through my bag to find the watch I’m not supposed to use. Nearly half past five: we’ve slept through the wake-up bells. I woke up already fully dressed and then some, so I just grab my jacket and jump into some oversized Wellies that happen to be by the door and run outside. I had brought a headlamp for exactly this situation but I don’t know where it is. I can’t make out the path in front of me at all, it’s pitch black in the middle of this forest. I can see the lights from the main building up the hill and I know the path mostly so I manage to stumble my way up there without falling on the hard snow. I really have to pee; that’s always the first thing I do when I get up, but no time for that now. Inside the zendo they’re already well into the chanting. No tea for us this morning. I focus on getting everything right. Bow in the doorway. Hands in gassho as I walk. Straight to the officer at the nearest end: you bow to them when you enter the zendo late. I don’t know if you still do that in the middle of chanting, but they bow back. Get to my spot, quick bow in front of my zabuton. I sit down, no time to mess around with all the cushions. I slide into a sloppy crosslegged position and grab my chanting book and frantically try to find the page everyone’s on. KO-SHIN-MU-KE-GE. Obviously I slept poorly, but I figure the officers never made it all the way to our cabin with the bells. SAI-TEN-DO-MU. I’ll probably never know. HA-RA-MI-TA. It’s basically impossible to find where in a chant you are once you’re out of it even when you know the page—HA-RA-SO-GYA: got it!

The following days are spent desperately trying to stay warm enough at night to not fear for my safety. The place has plenty of firewood stacked everywhere. In just one day I burn through all the kindling that’s been prepared in our cabin, awkwardly trying to light a fire in the cabin and keep it going. I go on what feels like a video game quest, being sent from person to person to try and find someone who knows where the only axe is. It’s found half buried under a snowy tarp. I spend every break after meals while it’s still light out, about forty minutes twice a day, chopping wood. When it gets dark I spend every minute I can spare to try and keep the fire going. Instead of walking kinhin, about ten minutes between each sitting period, I sprint down to our cabin. The fire has died almost every time. I light it again and hope the heat will accumulate.

After the day’s last sitting period I make yet another fire and sit myself down in a chair in front of the fireplace. When Oryō comes to the cabin he seems delighted that I’ve made a fire. He grabs a chair and sits next to me. Warming ourselves and resting after a long day of sitting. We never share any glances or anything like that. That would seem to be against the spirit of the thing. We watch the fire together in silence for a long time. I’m hoping this fire will get us through the night. I wonder if he’s as cold as me, or if his tiny room is keeping the warmth better. A billow of smoke falls out of the fireplace and fills the room. Suddenly the smoke alarm is blaring and we scramble to climb on a stool to reach it. We sit back down and close the door to the fireplace this time. We watch the fire some more. Oryō starts cracking up, then laughing uncontrollably, and after a while he’s crawling along the floor gasping for air. I sit in the chair in front of the fire trying not to make too much noise. I can’t see the fire anymore through all the tears from silently laughing so hard. I don’t know if laughing is against the spirit or not.

I put the smoke alarm back before going to bed and I watch the light dance on the furniture from the still burning fire as I try to get some sleep. I wonder if it’s dangerous to sleep with a live fire. But there’s no one to ask now. I awake from the cold again. The fire is out and the room is dark. I light it many more times trying to keep it going through the night. Somehow I’m barely winning this fight. It’s just warm enough that I can get some sleep, even though it’s still miserably cold.

I’m pulled aside by one of the officers. He sits next to me in the zendo, and thinks I’m moving around too much. He asks if I’m in pain. I am. He insists I should use some small cushions under my knees or hips or whatever to build myself up and relieve some of the pain. It’s well meant advice, and kindly given. It works to relieve the legs somewhat, but at what cost. It’s impossible to balance the spine like this; it necessarily induces a slump. My back starts aching instead. That’s not as unbearable, but what am I even doing? It doesn’t seem right. I don’t like it. I drop the cushions again and resolve to sit still.

There is dokusan, a private meeting with the teacher: Koshin, the head teacher at the center and leading the retreat, is a mild-mannered American who spent over a decade at the same monastery as Shingen. Everyone looks forward to the relief. There’s too many of us for everyone to go every day. After dokusan Koshin lends me his sitting stool, to be able to sit higher and take the load off the legs. I try it briefly, but my legs are too shot to be able to sit on my knees. He shows me how to covertly slide the upper foot off the lower leg when you want to get out of a half or quarter lotus and into Burmese. “The trick is to do a slow slide all the way.” He demonstrates. “You don’t want to—CHOMP—it down.” If no one can see or hear it, you can’t be called out.

Nothing helps.

Between two sitting periods I find myself sitting in a chair enjoying a cup of tea, looking out at the drifting dusk snow. The fire keeping the zendo warm crackles behind me. My body is exhausted but I feel good. My mind is clear. I have the thought: if only being wrecked after work could feel this pleasant. Be this wholesome. I finish my tea and a while later they start clacking to indicate it’s time for more sitting.

I eventually get the idea to bring my heating problem to dokusan. To my surprise he takes it very seriously. He suggests making it one of the officers’ jobs to go around keeping the fires alive, but then we discover the other cabins have these giant portable oil radiators ours doesn’t have. That’s why no one else is running around during kinhin or chopping wood during breaks. I am authorized to take one from the main building. We both laugh at the whole situation. I hold up three fingers. Chop wood, I say, and fold one down. Carry radiator, I say, and fold the second down. Don’t worry, he says, the ecstasy part comes later.

On the eve of the last full day, something happens. I don’t know where it comes from, but I imagine the zabuton as a shallow pool of healing water. I don’t know what that means or what “healing water” would be, but as long as I’m able to hold that image steady in my mind, I stop suffering. When it slips, I’m cast right back into the throes of intense suffering. It’s a great motivator. I’ve been trying since the first retreat to accept the pain, but it hasn’t really done anything. I already knew, from that first zazenkai, that the nature of pain and suffering wasn’t quite what I had believed. But now it’s different. The pain is a warm, slightly pleasant glow any time I’m able to stabilize the image of healing water covering my legs. The suffering is gone. I set my mind on the intention to get up swiftly when the bell rings. When it rings I spring up without hesitation and without using my arms. I am the first to stand, and my legs aren’t hurting. After that sit, the mere intention to regard the sensations of pain as a pleasant massage is enough to re-render the whole experience.

I feel like I might be onto something.

The morning after is the last day and there is some more sitting, and the same thing holds. I’m able to stay with the sensations and sit still, comfortably. The retreat ends, my fiancee comes to pick me up and we give Oryō a lift to the train station. I learn he had a small electric heater in his tiny room the whole time. The mood is joyous, warm, bubbly. But not to my fiancee, she hated hers. When we’re alone in the car she puts me into words: “a fucking Duracell bunny.”

Interlude

Before getting into this whole Zen thing I’d already booked a trip to Gaia House for that spring. It was an insight retreat that went from Tuesday to Sunday, so it was my longest retreat yet, and I had to travel to another country. The grounds at Gaia have been in silence for decades and the birds have taken it over. Even sitting in the meditation hall you can hear five or ten different species of birdsong at the same time, constantly, throughout the day. The retreat featured way fewer hours of sitting than what I had gotten used to, and the expectations very different. No one checked if you were in the meditation hall. The shoes were all over the place, and for the meals you queued in a random order. That took a lot of the fun out of it; I wanted more sitting and more ceremony. I slept quite a lot in all the free time we had, and found a new appreciation for being able to shower basically whenever I wanted.

On the train back to London I had a lovely chat with an older English gentleman who’d also been at the retreat. We exchanged book recommendations. After he got off at his stop, I spent the remaining several hours looking out the window at the places and people passing by. I felt great, despite the retreat being easy as pie. The usual pull to reach for my phone and read something or listen to a podcast was gone. I looked at a young couple sitting across from me holding hands, both reading their own Kindle, and both wearing headphones. I looked at the people walking past on the platform outside. Many looked like they had just finished a marathon, staggering forward and unable to make eye contact with anyone. I had the sense that I hadn’t gained anything, but rather put something down.

Then I arrive through the security checkpoint at Heathrow and as I sit down to meditate while charging my phone I realize I have made a huge mistake. There is too much bustling, too many people everywhere. Nowhere I look is safe. My mind starts racing, there’s a restaurant over there, what if I go there. But I immediately know that won’t help, there are just as many people if not more there. There is no restaurant, no place at all, I can go where there isn’t a deafening crowd of people. The ceiling is low and there is no daylight in sight. I’m dizzy, heart pounding, palms sweaty. I know what’s happening: I’m about to have a panic attack. The urge to move is overwhelming and there is nowhere to go. The flight doesn’t leave for several hours, why did I go through security. I grab all my stuff and go in a random direction. The terminal is a maze. I find a prayer room. It has six or eight of the world religions on the placard, including Buddhism. Good. I go in. It’s quiet, only a few people. All of them on the prayer carpets facing Mecca. I look around for a place to sit. A young man asks me what I’m looking for. It doesn’t feel like he asks to be helpful. “A cushion,” I say. I find none. I don’t feel welcome here. There’s a single chair in the whole room. I sit and cross my legs on it. I hold the top rail of my luggage while I keep my eyes closed.

After a while, I feel I’ve calmed myself down enough to conclude I’ve sailed clear of an actual full blown panic attack, so I get up and head back to the terminal. I don’t feel great. But I manage to find a place with windows, at last I can see out onto the tarmac. I can see the outside. It’s not quite the same as being outside, but it helps. I go further until I get to some unused gates with much less people. I find a station that lets me charge my phone. An older Frenchman asks if I’m going to Versailles. I barely glance at him. All I answer is “no,” and he slinks away. Why would I be going to Versailles? I stand there staring into nothing until my phone has charged. Poor guy, just wanted a chat. I call my fiancee and she talks to me on the phone until the plane leaves. Never have I been more soothed than by the awe-some rumble of jet engines pushing a thousand tons of steel with me in it at harrowing speeds. I close my eyes and relax.

Back home I have a great several days of post-retreat high, feeling the exact same energized lightness and joy as before. The retreat itself wasn’t challenging, but neither did I get through the trip without a fight. I still don’t know if the adversary is a necessary component for the post-retreat high.

Sesshin

Sesshin is the full week summer retreat. It’s up in the mountains in a forest, there’s a lake nearby for swimming, paths for hiking, and a few main buildings on a little hilltop. The zendo floor is somehow even worse than the ornate carpet, it’s that gray linoleum thing with lots of fake dust specks or a camouflage pattern or however you want to describe it, lots of tiny patterns you get lost in for hours. We go around the table before we begin, telling a little bit about ourselves; why we’re here. Everyone competes to give the most zen answer possible, which is a very un-zen thing to do. Everyone says something about nerves or how it feels standing on the brink. I feel kind of fine and confident, but I don’t want to sound like I’m trying to sound “zen,” so I say something about nerves or how it feels to stand on a brink. Getting up from the table, that feels a little ominous.

As with every retreat, there is an introduction in the zendo for newcomers to lay out some of the expectations and answer questions people have about how things are going to go. This will be my fourth introduction. Shingen is here to help out and runs the introduction. Instead of morning bells they will be using the wooden clackers this retreat. I guess clackers can’t break. He makes a point of how it’s possible to eat with a spoon without clanking the bowl. In fact, he says, you only have to touch the walls of the bowl at the very end when you’re almost done, and then it’s possible to do so very deliberately and quietly. He demonstrates by silently scooping air out of his empty bowl. When someone predictably asks about meal etiquette, they get the same answer I did when I asked about meal etiquette on my first retreat: don’t worry about it, do whatever everyone else is doing and it’ll be fine.

The days are long and all the same.

I am pulled effortlessly from deep sleep, through a half-asleep state into full, bright wakefulness by the sound of wooden clackers—clack, clack, clack. It is four in the morning and it’s time to go. I put on a wool sweater and my black samue-robe-like jacket and go outside. It’s still as dark as it gets during summer and it’s cold even with the sweater. I see the back of Oryō’s head, on his zabuton, through the zendo windows. Motherfucker. I pee, wash my face, skip the toothbrush, and sit down in the zendo. The officers come in with the incense and do their bows; two of the officers serve tea, first to the buddha at the front, then everyone down the two rows of zabutons. It’s two and two at a time, the officer stops in front of me and the person next to me, we hold out our cup and indicate with hand signals when to stop. Everyone waits, then picks up and drinks their tea in unison. Quiet and fast, you don’t want everyone to wait on you. Everyone waits, then puts their empty cups down in unison. The officers swap tea pots, walk briskly to the front and do another round. The previous round’s tea wasn’t that hot, that means this one is going to be scalding. I indicate to stop pouring almost immediately—I’ve made that mistake before. We bow when everyone is done and put the cups away. The chanting book comes out and we go. It’s the first period of the day, so it’s the whole book—about forty minutes. GYA-TE GYA-TE HA-RA-GYA-TE HARA-SO-GYA-TE BO-JI SOWA-KA HAN-NYA SHIN-GYO. Kinhin is outside, it isn’t raining very much. It’s almost a jog. It’s getting a bit brighter outside but still very dark, then it’s time for sitting. Walk, sit. Walk, sit. Walk, sit. I try not to keep count. Walk, sit, walk, sit. The sun is low in the sky and casting long morning shadows. Walk, sit, walk, sit. Two clacks: time for breakfast. We get up, hold our bowls in both hands, and walk out in two lines. It’s cloudy and gray. We sit down at the table, quick chant: HA-RA-MI-TA, eat, back to the zendo: bell, everyone bows, the officers wipe off their zabutons and fluff their cushions, they meet in the middle of the room and bow to each other. One of the officers remains and bows to us: we’re released. Everyone wipes off their zabutons and makes their spot neat. My body needs more sleep. I quickly wipe off my zabuton and make my way back to my bunk, I will get maybe fifteen or twenty minutes. Clack, clack, clack. Time for work. Not a cloud in the sky. Another day I cleaned the toilets, today my task is to clean the entrance. I remove the giant carpet-padding thing and all the shoes, I mop the floor inside, I sweep the outside with a broom. I move fast. Unhurried. Does this task include cleaning the carpet-padding thing? I don’t know. I hang it over the railing and beat it with my open palm. I have to stop because I can’t breathe. I have become covered in brown dust. I should have taken off my meditation jacket. I beat the thing some more while holding my breath. Clack, clack, clack. I put it all back and make sure everything looks nice and tidy. I pause to enjoy my work, I have a minute. Okay, back in the zendo. Medium chant. Dharma talk, walk, sit, walk, sit, walk, sit. Lunch. Is there time to nap? No probably not, I gather my dried shorts and towel from the fence and go for a quick bathroom break. We walk, it’s sunny, I’m full of energy. My legs are strong, body supple. It feels great to walk. The walk is mandatory, the swim is not. Another day I’m going to be in a bad mood and lie sulking in the blueberry heather. Not today, I join the swim. I bask in the sun. Back at the compound, I find a fence behind the cabin, perfect: I can hang my wet shorts and towel here. Clack, clack, clack. I sit down in the zendo. Longer chant. Walk, sit, walk, sit, walk, sit. Thirty minutes of yoga to loosen our bodies. Dinner. Break—I have time to poop and get another forty minutes of nap. Luxury. Clack, clack, clack. I’m in the zendo. Gyodo: slow walking chant, each step is a single syllable. HA-step RA-step MI-step TA-step. Sit, walk, sit, walk, sit, walk, sit. The officers arrive with the tea. Always two servings, but since it’s evening we get a cookie as well. The cookie is delicious. The officers bow and leave. Do I stay for the optional night sitting? No, my body needs sleep. I brush off my zabuton and head to my bunk. Oryō stays. It’s hard to fall asleep so early. It’s barely past nine. Clack, clack, clack. It’s four in the morning and I’m wide awake. Time to go.

I’ve been outside before, but being outside regularly every half hour from before dawn to after dark gives me a completely different sense of what weather is. Now it strikes me as ephemeral, transient. The sky is always there, but clouds, wind, and even the color of the sky, changes by the minute—it feels like.

I was sulking in the grass, I tell Koshin at dokusan. I don’t know why, there was no reason. “Good mood, bad mood. Mood is weather,” he scoffs. I squint. He knows.

I expect to meet the pain with concentration and work with it. But it doesn’t work. I’m fumbling for a while, a few days, with trying to meet the pain like I had before. I try new things: this is chiseling my body and mind, I tell myself. I don’t drink any tea in the breaks because it would have me go pee every kinhin and that would interrupt my concentration.

Then I discover something I wasn’t looking for and wouldn’t have liked the sound of: the pain isn’t going away. And it’s not going to stop being painful. But I can sit in a new way. The pain is as problematic as ever before, but somehow, leaning into a good posture and sitting properly—feels good. Whereas before, the pain would tug at my body. I would wince and cramp down in agony. Sitting well now feels good, not in spite of nor because of the pain, but somehow with the pain. I am in pain, and the pain is still pain. It is not a good, warm sensation. It is not neutral sensations. It’s painful pain. Yet I’m not suffering. Nothing about my body looks like it’s in pain. It’s relaxed, alert, strong. I feel expansive, I can feel the pain down there in my legs as clearly as ever, but I can also see the linoleum floor and the people across the hall, I can see my body in my peripheral vision and the people next to me and I can hear the sounds all around. All of it feels like me, and the pain only a small part. The thought occurs to me: this is zazen.

But it doesn’t last. As the days go on the pain becomes too large for even this to hold. Or I lose track of it, I’m not sure which. When it becomes too much to bear, I fall back to old strategies. I try to concentrate on it. I try to convince myself it’s chiseling me.

It’s late afternoon. I’m suffering. My legs feel kind of fine though. The pain comes and goes throughout the day, and right now it isn’t that bad. I’ve had much worse. So why am I suffering so much? Why am I in such misery? I can’t quite put a finger on it, until it hits me: I’m overheating. It’s been an especially warm day, and the sun has been frying the zendo all day. I’m wearing a shirt under my samue-robe-thing. Next kinhin I’ll take it off to air out properly, I decide. The bell rings, two clacks. Okay never mind, it’s dinner time.

While I’m eating my delicious bowl of rice with soy sauce I suddenly realize I will not be able to finish it. It looks like a mountain of rice. I feel sick at the mere thought. I put it down in front of me to rest. My heart is racing and my palms go clammy. Oh no. I need air, right now. I need to get up, go out and get some air. I’ve never seen anyone leave a meal. Why is this happening? What is happening? A revelation: I’m thirsty. I’m not warm, I need water. My body needs water. I know I will get tea very soon, when the meal is over. I make a commitment to my body that I will drink as much water as I need afterwards. And with that, the growing panic slowly starts to wear off. It’s as if my body agrees to dial down the panic. My heart rate settles and my hands don’t shake quite as much. I wait a little longer and then manage to get down the remaining rice.

I lie in my bed pretty shaken in the following break. I don’t know whether it’s safe or irresponsible to continue. I definitely don’t want to induce an actual panic attack. At least now I know what has been causing my problems, and I won’t let it happen again. I don’t know what to do.

What am I doing! Why am I lying here contemplating how to proceed without telling anyone. It’s as if it hasn’t occurred to me until now: I can actually ask for help. I go find one of the officers and ask for a meeting with one of the teachers. She asks why; I tell her I almost had a panic attack and am considering whether I can continue. She comes back in a few minutes and tells me I can see Koshin right away in his room.

He is relieved the incident has a clear cause and an obvious remedy. I’m not feeling so sure, I’m still very shaken and worried whether it really is wise to go on. He suggests I sit out this evening’s last period to recover. Try again tomorrow morning, and see how it goes. We’ll meet again both before and after. He suggests I should really drink some electrolytes, but we don’t have any Gatorade or whatever. I say I once heard a Jamaican man rap on YouTube about cucumbers being a good source of electrolytes. He hasn’t heard the rap.

I go back to my room to try and get some rest and maybe some sleep. It is the evening of the third full day, the exact middle of the retreat. Koshin has lent me his water bottle, filled with slices of cucumbers he nicked from the kitchen.

The next morning was beautiful. It was wonderful to sit in the zendo while the light slowly crept in. As the sun rose I realized that my body had been screaming for attention. I hadn’t listened. It had to resort to almost inducing a panic attack to get it. Which subsided as soon as it did. While we were out walking kinhin with the low morning sun in my eyes, I realized the pain in my legs I had been wanting so badly for the whole year to be gone, was also my body trying to reach me. The leg pain just needed to be held, felt and loved. A well of compassion sprang from within and tears streamed down my face. I could not wait to get back to the zendo and sit, to be with and hold my dear, beloved pain. It needed me. For the next many hours I sat and felt every inch of the pain as meticulously as I could. I wanted to be nowhere else. I was sobbing for hours. It was like finally holding a small child I had inadvertently neglected. I was so, so sorry.

The rest of the retreat went well. After this there was no more leg pain. I could sit for the last three and a half days with perfect posture and it felt great. Sobbing uncontrollably in a room full of almost entirely strangers, and not being allowed to do anything about the tears causing a mess, was bizarrely liberating. It was as if Zen was saying to me, there is nothing wrong with you or this. I realized there was some potency in what I had experienced, and decided to point this tenderness at a specific emotional problem I’d had for a long time, but hadn’t been able to do anything about. The problem softened and was permanently, and almost entirely, resolved. Later I was able to release tension in my stomach I’d been chronically holding my whole life. That too is almost entirely and permanently gone. At dokusan I related the sobbing to Koshin. I told him that at one point we were chanting and I was ugly crying—at full volume: ugly chanting. “That is also a way to chant,” he said, and I had the feeling he meant it. He asked if I wanted to read—at the informal meal after sesshin—a poem that came to me while sitting. I had recited it to him during dokusan, but I didn’t think I would be able to do it justice at the table in front of everyone, so he read it for me. I’m not sure it made sense to anyone else there, but I’m sure it makes sense to you now. It’s called “Sesshin”:

The zabuton becomes a torture chamber
of my own making
And the only escape
is to take this crowbar Zen gave me
and crack open my heartEpilogue

After my concentration discovery at the winter go-sesshin, I tried concentrating on my legs as I was having a conversation with my fiancee sitting on the couch at home. What I found was that I was still able to do it, but at the expense of not being able to listen. If I listened, especially as the pain got stronger, I couldn’t also concentrate on the pain at the same time. So it’s not some kind of silver bullet, and it also requires a mind capable of sustained concentration, which you usually don’t have outside retreats. In a similar vein, even though the pain was completely absent for the remainder of the summer sesshin, it’s back to normal when I sit for extended periods now. [1]

But something else arrived. After I got back from sesshin I started developing a strange sense of being with my body, as if my body was somehow an other. As I walk around the house doing menial chores I observe my body doing them. It’s not me who’s doing them, I’m keeping my body company as it does the chores. It has company, and so do I. I’m no longer on my own, or by myself. Company makes life less tedious. It comes and goes to varying degrees, and I find I can connect with this sense of company more easily the more I make my life like retreat: less distractions, less productivity, less trying to achieve things in the world for some external goal. I hope to write more about this in the future when I understand more.

Oryō, by the way, is a fake Dharma name we made up for fun. According to AI it means “Awakening Completed.” Mine is Taigen, “Great Mystery.” And he didn’t sit the whole night. I think.

Unfortunately I’ve since moved further away from the city and I am not able to attend any zazenkais, at least not as long as they are without accommodation. Which is a great shame, as I now feel that whatever my initial interest in meditation was, it is retreat which is the greatest treasure. Nevertheless, I keep a daily practice and sit with another local group weekly. Koshin has started doing dokusan online; I’ve signed up for this coming Wednesday. I’m hoping to get some meditation instructions. I look forward to sesshin with the same trepidation I did that first retreat. But next year I’ll be back.

1. ^

   Which is not to say that the body does not adapt! I used to be extremely inflexible, like probably most people who spend most of their time sitting in front of a computer, but over the last few years I spent some effort on folding my legs under me when watching TV or whatever, and now I can easily sit for more than half an hour before any discomfort starts. It’s really very comfortable, and has become my favorite way to sit.

Discuss

MAPLE was recently criticized for being a “high-control group” by Kyle Hubbard.[1] This is my case that contrary to those criticisms, MAPLE is doing something fundamentally both neglected and highly needed within the realm of x-risk.

I’ll make the case in three parts. First, I’ll argue that there is a fundamental bias that causes the perpetuation of x-risk, that is ignored and neglected by the mainstream x-risk community to its detriment. Second, I’ll argue that there is a solution to that fundamental bias, which is based on spiritual attainment in the context of monastic traditions, especially Buddhist traditions. And I’ll argue that what's needed is a new monastery that can address the problems of THIS age, such as artificial intelligence. Finally, I’ll argue that MAPLE is doing a great job of being this monastery, and the harsh criticisms of it are spurious or confused.

In that final section, I’ll both directly respond to some of the criticisms made by Kyle, as well as try to provide my perspective on what’s happening here. However, the arguments in that section rest on the two before it, and I urge the reader not to skip ahead.

For context, I’ve been a resident at MAPLE for over five years, including time as Head of Operations, and I’ll be leaving in a few months. So I’ve seen how the place works from the inside, and I don’t need anything from MAPLE going forward. From a selfish perspective it would be easier for me to stay silent than to write something controversial that could easily negatively affect my near-future job prospects or collaboration opportunities in a time that I believe is critical to the future of life on Earth.

But I believe it’s important for me to make this case, both for the community I hold dear, but also for the broader x-risk community. I hope that it will be taken in the spirit of care for this community and the world that is intended.

The Fundamental Bias

When I first started looking at existential risk, I did a root cause analysis of the problem, and it was clear to me that one of the root causes is the mind. The rationalists are gesturing in the right direction here, and when I first encountered the community (via being invited to a CFAR workshop), it seemed to me that the rationalists were clearly onto something important.

And yet, as I moved within the rationality and EA communities, it became clear to me that rationality (both as initially conceived of in the Sequences, and also later attempts like that of CFAR) was simply not sufficient to solve the issue. And this wasn't just an academic worry, as it soon became clear that AI capabilities and the rush towards existential destruction were being driven in large part by these communities and their adjacents. What’s more, these communities provided a ready-made talent pool to draw on to further AI capabilities.

I had already been through a version of this in Silicon Valley. I started startups, consulted for other startups, and went to parties in Silicon Valley. I thought maybe the problem was that we didn’t have the right companies or the right technologies. After a few years I started to think: I don’t know if we’re solving these problems. It seems like these companies, including my companies, are causing more destruction, more disconnection, and more distraction than ever before.

So I went to MAPLE to meditate for a few months. Not because I thought they had the answer. I thought they were a confused group who believed meditation could somehow fix the world. I just wanted to clear my head and figure out my next move.

But at MAPLE, they very quickly helped me directly see biases in my own mind that the rationalists could only really gesture at.

Seeing the fundamental bias of the mind

One memorable experience was on day five of a week-long silent retreat. I was sitting cross-legged in the Zendo, this beautiful big round building. My friend Ava was next to me. There was an altar in front of me with a Buddha on it. I was focusing on my breath. Each breath, just letting go, and letting go, and letting go.

What started to happen was that I started to let go of things I didn’t even realize I was holding on to. The idea that that’s my friend Ava, and this is Trinley, and this is the zendo that we’re sitting in. That all began to fall away. There was this total focus on the breath. And what really surprised me was how I’d been using all these concepts to create conflict.

“Well, I like Ava, but I don’t like the way Jordan smiles, so I need to change Jordan to smile like Ava. But wait, that’s judgmental. So now I need to change myself. And I don’t like the way the altar is set up, so I need to shift the Buddha slightly to be in alignment with the altar.”

All of this needing to change myself to match the world, needing to change the world to match myself, needing to change everything to meet my preferences — it all fell away. And there was just this sense of peace. This sense of not needing to fight.

The Precision of the Buddha

What was so striking about this experience was how well it mapped directly onto the technical teachings of the Buddha, about the exact structure of the mind, the order things were constructed, what caused what, and what would happen as these causes dropped away. It was much more precise than any of the (often quite precise) psychological or emotional work I had done over the last 20 years.

This and other experiences, as well as the direct teachings of the Buddha, made me realize that the rationalists were working with a limited, shallow understanding of the mind (in comparison to the Buddha, who to be fair is a hard act to follow). None of the cognitive bias literature could give me such an exact map of exactly what was happening in my mind and what I would observe as I slowed things down. But even having studied the Buddha’s teachings, the extent to which this one tendency was causing me dissatisfaction was completely opaque to me until that moment. What was needed to clarify these biases wasn’t sophisticated concepts or ontologies but direct insight and investigation.

In that moment I could see the mechanism behind this tendency to delude myself and thus fight with reality, as an individual and as a species: a tendency towards clinging.

What is Clinging

Clinging is a tendency to hold on to your abstractions as if they were real, which leads to fighting reality to make it more like your abstractions. 

Julia Galef once wrote a whole book about a small subset of this tendency, that she called the soldier mindset. As she stated: “The problem with identity is that it wrecks your ability to think clearly. Identifying with a belief makes you feel like you have to be ready to defend it, which motivates you to focus your attention on collecting evidence in its favor. Identity makes you reflexively reject arguments that feel like attacks on you or the status of your group.”[2] This is, of course, pointing to a small subset of clinging, but the Buddhist approach to clinging goes far beyond this.  In the thousands of years of direct Buddhist research into this from many different lineages, and approaches both academic and deeply experiential, clinging has been characterized as a fundamental tendency of the mind that requires deep spiritual training to root out.

The Map that Eats the Territory

This is why any attempt to solve existential risk, undertaken from a place of clinging, will tend to create even bigger risks. The person isn't only trying to solve the problem. Underneath, they're also trying to perpetuate their abstractions: the self, a particular way of seeing the world, a particular view. Any time reality diverges from those abstractions, or the abstractions feel threatened, the person moves to defend those abstractions. I watched this happen in the communities I came from. People who set out to prevent an AI catastrophe ended up building AI capabilities, or supplying the talent for it, because the picture of themselves as the ones who would do it safely mattered more to them than what was actually happening. 

And the moment you cling to the abstraction rather than reality, you become willing to destroy the reality in order to defend the abstraction. You become willing to destroy real people, real places, real life in order to maintain your model of them. Why? Because we believe, deep down, those models, ideas, and abstractions are actually us and our world. And we will do ANYthing to prevent that from perishing.

If you care about x-risk, and really understand the previous paragraph, it should shake you to your core. The map EATS the territory, destroying what’s real in place of our ideas, and so ANY map you build, from a place of clinging, leads to MORE of the territory being eaten. The map is your abstractions: that which you construct, build or make up. The territory is that which is not made up. The more powerful your maps get (the more predictive, the more general, the more useful),  the more you cling to them, the MORE of the territory gets eaten.

This doesn’t mean that the territory is somehow “lost” of course — the territory is always the territory. But it does mean that we lose sight of the real sacredness within the territory, and therefore perpetuate terrible harm to ourselves and other beings.

There are memes to try to counter this: “Keep your identity small,” “Strong opinions, weakly held”.  But all of these are maps of their own, subject to their own clinging, and are not nearly sufficient to overcome the fundamental bias. Something that is not more map building needs to be used instead.

If this is true, rationality, and the entirety of Western society, are missing a key fact that is causing them to move ever more towards existential destruction of life on Earth — the territory getting eaten entirely by the map.

The Fundamental Bias Runs Deep

At a fundamental level, society has the same bias, and is in fact built on clinging to abstractions. The financial system, for example, was built on this same clinging — the same move of the map eating the territory, built to a bigger degree. For instance, there is something about a “forest” that is not constructed or abstract. But when we cling to our abstractions, a tree becomes a table, a table becomes an asset, an asset becomes a number on a spreadsheet. Suddenly, the map eats the forest, and we lose track of the life we’re killing in the process. This isn’t an accident, it’s inherent to a mind that still has the tendency to cling.

Similarly, there is something about a “friendship” or “love” that is not constructed or abstract. But when we cling to our abstractions, a friend becomes a resource tracked in a CRM, a relationship is evaluated by its ROI, or a partner becomes a milestone to check off on a life roadmap. Suddenly, the map eats the relationship, and we lose track of the genuine human connection we’re eroding in the process. If you’ve been in a long-term relationship that “loses its spark,” or found yourself getting mired in petty disagreements with deep and loving familial relationships, these are both microcosms of this map eating the territory phenomenon.

This isn't just about individual minds that cling. The system and society itself develops structures that perpetuate this clinging on a collective level. It's baked into our financial system, our government, and more recently, the algorithms that run our social media platforms and LLMs, that run the collective way of seeing.

The Cybregore

This is the word we use at MAPLE for a collective intelligence that is mediated by algorithms (mentioned in Kyle’s piece). What makes modern society different from societies of the past is the scale of the algorithms and data that are connecting all of these humans and systems together. And it clings to bigger abstractions in a stronger way than anything we've seen previously. It’s the most effective tool we’ve ever created for building more predictive maps that eat the territory faster than ever before.

But it's not just the LLMs that we're worried about. It's the society that clings to the perpetuation of these LLMs, their creation, and their empowerment. Unchecked, this society will use artificial intelligence in order to destroy all life on Earth, in a misguided attempt to convert matter into data-processing capacity. And even if we manage to solve that without letting go of clinging, we'll create even larger problems in the future as the map continues to eat the territory. This pattern of technology, mediated by clinging, creating even bigger problems is one we’ve seen for centuries.

The Solution

There is a solution to the fundamental bias. Clinging to our maps is not inevitable. Clinging is dependent on certain conditions (which themselves are dependent on other conditions, which have numerous feedback and feedforward loops). In Buddhism, this structure of causal feedback loops that can create the conditions for clinging is known as interdependent origination.

When looking closely, one sees that clinging to maps is crazy. Our clinging causes dissatisfaction whenever the map fails to live up to the territory, or the territory fails to live up to the map. Because the map is not the territory, this means that we are ALWAYS dissatisfied, and we then create ever more complicated maps to hide this dissatisfaction from ourselves, which itself causes more dissatisfaction because these maps are ALSO not the territory. When one sees this causal structure (which can be very deep, not just this simplified explanation of dissatisfaction I just gave), it becomes clear that the clinging process is neither inevitable nor beneficial. Then, one stops feeding it, and it ceases.

People identify with parts of this causal structure. They believe that thinking, preferences, consciousness, sense data, or their physical body is what they really are. This is one of the deepest and most cherished maps they hold. They are then unwilling to look carefully at reality due to the sense that one might see something in the territory that contradicts a cherished map of the self.

Some think they must destroy themselves in order to resolve clinging — but this is again confusing map with territory. In truth, it is a direct and thorough investigation of cause and effect that overturns mistaken beliefs that were mistaken all along. Overturning such mistaken beliefs clears the way to bring clinging to cessation. When one no longer clings to their maps, those maps no longer eat the territory.

The patterns that prevent such a direct and thorough investigation of cause and effect run very deep, and typically one cannot conduct such an investigation as a “part time” endeavour alongside other life goals. Many of the teachings of the Buddha are instructions for doing what needs to be done in order to accomplish this task, and many of the institutions of Buddhism aim to provide for, instruct, and support those who are committed to this task.

It turns out that a person who is free of the obstacles that prevent most from seeing cause and effect clearly, is of immense benefit to others, and to society as a whole. One does not trade off between serving others and serving oneself; rather, one is of benefit to all by being free from the patterns that beset others.

I've had glimpses of this while at MAPLE – enough to know it's real. But it's been discovered  across ancient mystical traditions throughout history.

The American mystic Peace Pilgrim wrote:

Once you've made the first relinquishment, you have found inner peace, because it's the relinquishment of self-will.[3]

The Buddha taught, in the Culasihanada Sutta:

He no longer clings to sensual pleasures, no longer clings to views, no longer clings to rules and observances, no longer clings to a doctrine of self. When he does not cling, he is not agitated. When he is not agitated, he personally attains Nibbana.[4]

The Catholic mystic Saint John of the Cross described it this way.

To reach satisfaction in all, desire satisfaction in nothing. To come to possess all, desire the possession of nothing. To arrive at being all, desire to be nothing. ... To come to the knowledge you have not, you must go by a way in which you know not. ... To come to be what you are not, you must go by a way in which you are not.[5]

The mystics have found a way to root out the fundamental bias and become a force for good in the world. This is what we call an enlightened person. This is what we call a saint. I'm using saint in a specific sense: a person who has stopped clinging, whatever their tradition and whether or not any church has recognized them. And there is a path to rooting out the fundamental bias completely.

When you're a saint, you're no longer trying to prove something, or be something. And therefore no longer fighting with the world. Then service comes naturally. And more importantly, you're not subject to the fundamental bias, which seems otherwise to inevitably pull towards destruction as the map increasingly eats the territory. But how is it that traditions can create a path towards doing the needed investigation that creates this shift?

Mystical Traditions are Built to Overcome the Fundamental Bias

In most modern societies, the fundamental worldviews and ways of seeing (liberalism and conservatism, capitalism, materialism, belief in progress, etc.) are built to reinforce the maps that most effectively eat the territory. This is a memetic attractor where those maps that most successfully convert other people to clinging to them perpetuate.

Furthermore, the practices and rituals of society (getting together and drinking with your friends, watching Netflix, getting a real job, etc.) are built to distract one’s mind in a way that prevents one from seeing the causal nature of clinging, rather than directing one’s mind to it.  Again, this is memetic.

But mystical traditions (ones that help to actually overcome the fundamental bias) turn this on its head.

The Map That Eats Itself

In a well-constructed mystical tradition — one shaped for centuries by those who have overcome the fundamental bias — the views promulgated are ones that help you let go of views, including the very views of that tradition. These traditions commonly teach ethical behavior, avoiding temptations, a sense of surrender and going beyond yourself. All of these are views that help to loosen clinging.

This radical shift reverses the direction of the fundamental bias.  Clinging is turned towards its own cessation. Rather than the map eating the territory, the map is used to eat itself.

The Buddha speaks of this with the metaphor of crossing a river on a raft. He describes a traveler coming to a river, and realizing they’ll need to construct a raft to cross it. He asks if, once having crossed the river, the traveler would be right in carrying the raft with him wherever he goes.  He says that his teachings are the same: made for crossing over, not for clinging to.[6]

Practices that can Observe the Clinging

In addition to the map that eats itself, many mystical traditions also have practices that hone the mind and clear away distractions, making it easier to observe the causal structure of the fundamental bias and therefore to bring it to cessation.

In many Buddhist traditions, they use meditation.  Through specific techniques, one can more directly observe one’s own mind, gradually letting go of the map and having better contact with ‘the territory’ (which again, is simply that which is not made up).  As you go deeper into meditation, you can enter a particular, deep state of concentration called Samadhi, which allows you to temporarily stop clinging to your maps, and even potentially see directly the structure of dissatisfaction that you’ve been maintaining.

However, the path to becoming a person who no longer clings is not easy.

Everything You've Turned From, Now You Have to Know

Everything you've wanted, you have to let it go. Everything you've turned from, now you have to know. — Soryu Forall

This couplet hangs in the meditation space at MAPLE.

And it points to one of the fundamental truths of this path of overcoming the fundamental bias, which is that it requires facing difficulty. It's hard, and it can be emotionally draining. Clinging is a fundamental tendency of mind, core to the way we make sense of the world. It’s central to how we understand the world, it underpins our sense of self, and our everyday sensemaking rests on it.

On top of that, we’ve been taught from a very young age, based on the structure of our society, that the goal of society, of life or consciousness itself, is to build ever better maps that represent the territory — that this is both the solution to all problems and the goal of society itself. When we as individuals and us as a society have invested everything in this project, it becomes difficult to admit that this is what is creating the problem, rather than the solution. This is the ultimate defense mechanism of a mind obsessed with abstractions: the conceptual framework hides its own destructiveness. Our obsession with building the perfect model blinds us to the fact that the model is what's killing us.

So, overcoming these tendencies is HARD. Mystical traditions near universally center this difficulty, and in fact may even treat the difficulty as the point. I describe what this was like for me in “Hitting Rock Bottom” below.

It requires facing the hardest parts of both yourself and the society which you're in. And it requires letting go of everything you've been holding on to that has helped you feel safe. All of the mystics we cited in the previous section have something to say about this.

For example, Saint John of the Cross calls this a night that brings darkness and misery to the spirit, but "does so only to give it light in everything."[7] And the Buddha says "Therefore hold nothing dear, for separation from the dear is painful. There are no bonds for those who have nothing beloved or unloved."[8] Peace Pilgrim[9] and Rumi[10] both have quotes that point towards the difficulty of walking the path.

The upshot of this is that walking the spiritual path often creates difficult emotions and unusual psychological states, which can in a Western context be diagnosed as mental illness. However, this does not mean that the path is not worth walking.

It's not the case that everyone is ready to face this difficulty, especially in the context of facing the existential risk of the world, and the existential destruction the world is already facing.

However, it is crucial to have a place where people who are ready to face it, who are ready to walk the spiritual path to its end, can do this work and become the hero the world needs.

That place may often look like a high control group — a cult.

Cult Is the Root of Culture

I first understood the value of an intense high control container like this over a decade ago. I was looking to improve my ability to serve the world, and found an organization called City Year.

As I researched it, I quickly found that it was frequently referred to as a cult. People pointed to the grueling hours, the lack of pay, the brainwashing-like training, the uniforms, the constant chanting, and the pressure to shed your old identity and conform. They mentioned the emotional pressure that came with the program itself, and the burnout that often followed afterward.

But what sold me was what happened to the people who were willing to deal with that, who were willing to take on that emotional pressure and reap the benefits by throwing themselves in completely. People frequently talked about becoming a better person afterwards, more able to live their values, more able to be effective and efficient, and more able to serve.

To me, that was a clear trade-off. Have some emotional difficulty, maybe take on a little bit of trauma, but become a better person in the process. And so I went into it with open eyes. And it was hard. And it was difficult. And I was tired, and I didn't agree with everything we were doing.

But I threw myself in. I took every opportunity of leadership, of learning, of growth, that I possibly could. When we chanted and trained and they tried to instill these values in us, I was as open as I could be, because I agreed with the values and chose to be brainwashed in that way.

I ultimately came out of that year a completely different person than I went in. I realized that by throwing myself into a place that was a completely different culture than the one I had been raised in, I had been able to counter the values the default culture had brainwashed me into, and in some sense done a counter brainwashing with more correct, more powerful values.

So the techniques that City Year used, those of a high control group, could not be evaluated in isolation. They had to be evaluated in the context of my choice to participate in them, and of the consequences to my character and the character of the culture that they created.  The high control dynamics were not inherently bad — they were good at creating a counter culture, and the evaluation had to point at the culture itself.

And that's when I developed this phrase, "cult is the root of culture": the seeds of a new culture tend to grow in small, intense groups that the old culture calls cults.

I started to get a sense that all of these cult markers, while important and real indicators of the potential for abuse, were also a memetic immune system for the default culture, preventing people from escaping the values that were inculcated into them, and preventing the seeds of a new culture from taking root. The resistance and panic surrounding high control dynamics is another defense mechanism of the default culture. By preventing other maps from taking root, the cult-panic locks in our current destructive trajectory.

We Can Choose Hard Things

I started to view cultures like City Year, these high control, high demand cultures, as akin to climbing a mountain. Climbing a mountain is not for everybody. Some people are not physically ready to climb a mountain safely. Some people, even if they're physically capable, don't want to put themselves through the hardships of climbing a mountain, or have no interest in doing so.

And you would never want to trick somebody into climbing a mountain without giving them a sense of how hard climbing that mountain will be. And some people, even if they're prepared, even if they know the risks, will still die when climbing Mount Everest.

But if we demonized the mountains for being hard to climb, or even worse, demonized the person who wanted to climb it, we would lose something precious and sacred. And if the solution to existential destruction is on the top of Mount Everest, then our society would be cut off from that solution.

The Mountain and High Control Groups

Similarly, when joining a high-control group where people are pushed to their limits, it should be clear what you're getting into. The students should also take responsibility for making sure the culture is ethical. If the situation is clear, and the appropriate responsibility is taken, then something beautiful can happen. You can create a culture that’s built to overcome the fundamental bias, by creating maps that eat themselves.

In fact, hard training is how many of these traditions actually create this cultural shift and help uproot the fundamental bias.

The Double Standard

When a “cult” (a culture different from the default) does something, we react with immediate distrust, disgust, or abhorrence. But when it’s for something the default culture promotes, we react with a willing admiration or at least acceptance. When training to become a doctor, a soldier, a CEO, a professional athlete, or a scientist, we accept sleep deprivation, being yelled at, possible physical injury, possible mental breakdowns, emotional humiliation, and being stretched to our absolute limits. Our reactions to the “cult” here are not based in fairness, reason, or understanding. They are based in ignorance and bias, an immune reaction to an unfamiliar or foreign system—and “foreign” here literally means not Western.

If we demonize a culture that uses rigorous training tools without looking at how it uses them, we simply ensure the direction we’re going will continue, until all of life on Earth is destroyed.

Monastic training is a widely accepted and lauded form of training the world over. The secular West has shunted it. But for millions of people, having a monk in your family is a source of pride, in the same way having a doctor in your family is a source of pride for many Western families. It’s important to maintain some humility—we don’t know what we don’t know. And most of us do not know what this is all about because we didn’t grow up in it, or we grew up with bad examples of it.

Skillfulness & Right Intention Matter

Hardcore training methods can be used wrongly and unethically. There is no guarantee that a system is doing it right or ethically. And that includes our often unquestioned systems for training doctors, soldiers, etc.

What the modern West has not been able to accept, due to its sophisticated immune system developed over centuries, is that the ancient traditions worked out how to wield harsh training methods with compassion, for true spiritual insight. When we look at revered spiritual teachers such as Master Rinzai (the 9th century founder of the Rinzai Zen School), Milarepa (an 11th-century mystic whose students went on to found the four major branches of the Kagyu Buddhist lineage), John the Dwarf (a revered 4th-century Egyptian Desert Father and early Christian saint), and Shibli (a 10th-century high official who renounced his wealth to become one of Baghdad's most revered Sufi saints), we see that these historically common teaching styles can indeed produce enlightened sages. All four of these masters are considered among the greatest spiritual masters in their respective traditions, and all four received what we would consider (from a modern Western lens) harsh teachings.

How I Judge The Trustworthiness of Spiritual Training

I cannot hand you a universal rule set for guaranteeing that a training program which uses such methods is enacting skillfully compassionate means rather than unskillful or downright abusive means. To try to ‘fix’ a system by coming up with a set of rules so that everyone remains safe and secure is a bureaucratic solution. And bureaucracy is generally blind to what truly matters in favor of establishing and maintaining its world of rules and regulations.

However, I will try to name three aspects that might help you decide whether a system is worth entering or supporting.

Firstly, the choice of the student should be central and essential. In many traditions, teachers don't offer teachings unless the students request them.   Many of these traditions actively reject students unless they demonstrate extraordinary will and effort, such as prostrating 100,000 times.  It’s the student who is seeking the master, because they want the fruits of the training. Thus, the student takes responsibility for both choosing the teacher, and accepting the harsh conditions that may follow.

Secondly, is the intention of the teacher. If a teacher has truly overcome the fundamental bias, and no longer clings to their maps, they won’t cling to any particular idea of their students or what they think the student needs or what they want, etc. They are in a position to teach truly without bias, without fixations, and with compassion as the only motivation. To these teachers, harsh behavior or confronting triggers and emotional states can become tools to help the students let go of their maps. It’s important that they aren’t hypocrites—that they teach only what they themselves have overcome and resolved within themselves. For example, these states can be used to point out places the student normally clings to or avoids. These behaviors are wielded by skillful teachers at the appropriate times, calibrated to the readiness of the student.

Finally, there’s the outcomes of the students. Do the students actually have genuine spiritual insights? Do they themselves go on to help others in compassionate ways? Do they live more ethical lives? A spiritual teacher is ultimately judged by the fruits of their students.

These three qualities are hard to find and confirm, and so there’s often a special place and ceremony around it. A shared vow, a shared schedule, a commitment to an ethical lifestyle, a skillful teacher as confirmed by their students, an environment proven to produce genuinely spiritual insights that can hold people through the profound difficulties of walking the spiritual path. This container is called a monastery. A monastery is a place where a student that is truly seeking, up for the challenge, can go to learn from a teacher who is helping them truly escape the fundamental bias.  And within that context, these techniques become wise, compassionate, and powerful tools in the fight to stop the maps from eating the territory.

Monasteries Produce Saints

Traditions focused on overcoming the fundamental bias have reinvented the monastic form again and again. The Christian, Buddhist, and Daoist monastic traditions each arose in vastly different social settings.[11]

Monasteries have created a profound set of practices meant to erode clinging, remove people from the structures that are causing the maps to eat the territory, and create a new culture that is designed for them to overcome the fundamental bias completely.

A shaved head and a simple uniform prevent the clinging to physical appearance and form.  The vow to not use money (or severe restrictions around it) removes a hook from one of the largest territory eating maps — the financial system. Regular spiritual practices like meditation, chanting, or prayer help clear the mind and more clearly see the fundamental bias in action. Regular teachings from skilled spiritual teachers help point out clinging to the students.

Each monastery develops its own culture. They may innovate certain rituals, frameworks, or ways of working to be effective for that particular time and place, and help work with the maps of the default culture. These frameworks are designed to be let go of once they’ve done their job.

The Track Record of Monasteries

Monasteries have a good track record. Where traditions kept records of their sages—those people overwhelmingly came from monasteries. For instance, the Benedictine Order was able to cultivate 24 popes for the Catholic Church over the years, and produced more than 1,500 canonized saints.[12] in

In Zen, the monastic system traces this effect through dharma transmission, an unbroken lineage of teacher to disciple, the majority of which are cultivated through monastic practice. Similarly, in Daoist Quanzhen, the Dragon Gate lineage traces its transmission back to the 13th century.[13]

In each case, the seed of insight has been passed down through the lineages. And while some of these lineages have died out, the monastic forms have carried this insight forward consistently, generation after generation.

The point is that the institution of a monastery makes spiritual realization, the dropping of the fundamental bias, something that can be replicated and repeated on a collective level, for the immediate society in which the monastery is embedded. The monastery as a literal de-programming and un-brainwashing container is what prevents the spiritual techniques being twisted towards serving destructive, deluded, or divisive aims that form the maps of mainstream culture.

For instance when we look at Jeffery Martin's work on persistent non-symbolic experiences, studying for the most part self-guided practitioners operating outside of a monastic system, we see Martin saying that this attainment doesn't guarantee more ethical behavior.[14]

Contrast that with those who practiced inside a tradition. While there are no large-scale studies of them that I could find, the evidence does seem to indicate that practicing inside a tradition genuinely creates more ethical behavior. In one study, long-term Nyingma practitioners of Tibetan Buddhism, tested in an economic game, showed less anger at unfairness, less impulse to punish, and more willingness to spend their own money to compensate the person who had been wronged.[15]

So if it's true that practicing inside a tradition indeed helps let go of the bias in a more complete, ethical way, then the monastery, as the most concentrated form of that container, should allow that to happen to an even deeper degree. But to do that, a monastery has to both hold its tradition and forms, and adapt.

The Monastery Must Adapt Without Dissolving

The spiritual path is hard, and finding structures that can actually help one walk it requires decades, even centuries, of experimentation and refinement, which is carried explicitly but, more importantly, implicitly in the lineage and forms of a tradition. Shaving the head, renouncing money, chanting and meditation are examples of such techniques.

But simultaneously, traditions, and especially monastic traditions, have to adapt to the culture they find themselves in. If not, they will die out, or be ineffective at helping people escape the particular hooks and maps of their culture that keep them clinging.

In fact, every monastic tradition we have is already such an adaptation. As the Buddhist scholar and nun Karma Lekshe Tsomo puts it, the various Buddhist monastic cultures observed today, "Chinese, Japanese, Thai, Tibetan, and so on, are the results of a synthesis of Vinaya and the local norms and customs of the countries where Buddhism spread."[16]

These adaptations should come from someone who has real spiritual insight, lest you adapt away the very features that make the tradition or monastic container effective. And indeed, when this adaptation is done with someone who hasn’t gone through a monastic container, worked within a tradition, or had a deep insight, it often fails.

Attempts in the West to create a path of spiritual insight mostly seem to get this wrong

These attempts are made from inside the very mind they are trying to escape and so, inevitably end up creating the problem they're trying to solve. A mind that hasn't escaped can't honestly look at what will help it, or what will help other people escape it. And that is doomed to fail, for the same reason solving existential risk from a place of clinging is.

For example, meditation techniques are often evaluated on whether they make you feel better and whether they help you have a life you enjoy, rather than letting go of the fundamental bias.

Jhourney, a modern organization trying to optimize hard on blissful states known as the jhanas, is a great example of this. Jhourney's headline metric is that over 60% of participants say it was the best thing that's happened to them in 6 months or more. This is a modern Western framing, about trying to collect good experiences. Their co-founder, Stephen Zerfas, expounds on how he wants to extract the jhanas from Buddhism the same way yoga was extracted from its initial worldview.[17]

Similarly, the Finders Course, another attempt to codify the path, states its goal as a state of fundamental wellbeing.[18] Neither gives the grounding of ethical training that most traditions offer, and neither of their head teachers came up through traditional training themselves.

For all I know, these programs may have made legitimate advances in achieving certain states that are helpful along the spiritual path. However, without the grounding of monastic tradition and individuals of genuine insight, the default outcome is a set of techniques that reinforces the fundamental bias and the status quo of moving ever more towards destruction. Someone who learns to reach deep concentration but keeps clinging to the maps they were raised with just gets better at serving those maps.

In fact, reaching these deep spiritual states while maintaining this clinging to the views of the society in which you were raised may inoculate you from fully rooting out this fundamental bias altogether.

Is This Article Just an Example of Getting it Wrong?

Unfortunately, I have not overcome the fundamental bias.  And so writing this article is terrifying — it could easily reinforce the very fundamental bias it’s working to overturn.  Stepping into the role of spiritual teacher (however slight and small with the writing of this article) without having overcome the fundamental bias myself is therefore fraught and terrifying.  My hope is that having trained within a monastic system for many years designed to help us recognize when this bias is acting, and getting feedback from my fellow monastics who are practicing, will be enough to have this article do more good than harm.  But ultimately, I’ll have to wait and see, as a spiritual teaching is ultimately judged by its fruits.

But It Can Be Done Correctly

If you have come up through a monastic tradition, and use that to overcome the fundamental bias, you're in a place to begin to make changes using modern tools for a modern culture without losing the pieces that make the training work. The modern landscape, with social media, AI, Netflix, Zoom, has created a very different mind. The maps are better at getting you to cling to them than ever before, and the practices  better at distracting you from the patterns of your own dissatisfaction.

A monastic container can use the power of these maps: measurement, data, the scientific method — if it can create a culture that doesn’t cling to those maps. This turns them from maps that eat the territory, to maps that eat themselves. This has to be done skillfully, with great respect for tradition, lest one unknowingly get pulled back in to simply reinforcing the maps of the default culture. And ideally there's a teacher who has at least to a large extent escaped that mind, who can be a mirror for what it's like to use that mind without clinging to it.

What a Real Solution Would Look Like

So a real solution would be a monastery that's shaped for this day and age. Buddhism has changed its forms before when the world around it changed. When it reached China, Chan monks began farming and doing manual work, which the Indian monastic rules had not allowed.[19] In 1956, Dr Bhimrao Ambedkar reshaped it again to take on caste.[20] The problem now is being built out of data and algorithms, so a monastery that wants to meet it has to be able to work with those tools.

This is the monastery we need:

It understands the fundamental bias, and that there is spiritual insight that can overcome it.

It's not afraid to put its participants through difficulty, or to pattern match to a cult.

And it uses modern tools and thinking without getting stuck in those ways of seeing.

This is what we're building at MAPLE.

Monastic Academy Is the Realization of This Necessary Institution

After being at MAPLE for five and a half years, I think I'm in a place to evaluate whether the theory of change I've laid out is actually being realized. Is MAPLE doing it well?

To do that, I’ll both talk about the MAPLE I know, but also respond to Kyle Hubbard’s criticisms of MAPLE directly. Kyle’s post in particular is an excellent demonstration of how our mainstream culture has a complex, memetic immune system towards an alternative culture that could dismantle it.

Kyle’s main point is that MAPLE is willing to harm its members. In five and a half years here I’ve seen people go through panic attacks and depression. I went through both myself, and I describe that below. These states are a normal part of the spiritual path, as you face everything that you’ve been hiding from yourself. I don’t believe any were a result of MAPLE causing someone harm.

On top of that, I’ve never seen anyone prevented from leaving. I’ve never seen anyone punished for raising a concern, or pushed past what they agreed to. What I have seen is MAPLE being patient and caring with people who were struggling: giving exceptions, having talks, spending a lot of time caring for them. MAPLE is not perfect, but it cares for people extremely well.

The Training Works

Before going deeper into the critiques, let’s start with a simple question — does the training work to help nourish spiritual insight? I was part of a project, mentioned in Kyle's post, where we interviewed a number of people who had gone through the training, some who had difficulty and some who didn't.

One of the most striking things to me was how almost everyone I talked to said they changed the way they lived because of MAPLE, such that they lived a more ethical life.

This is a common trend, even among our critics.

For instance, both Ethan Schaltegger and Kyle Hubbard, who have published criticism of MAPLE, praised the training in those same pieces for the spiritual insights they encountered over the course of their time there.

I deeply benefited from my time there. It's a place of hardcore spiritual training. I personally think it's the most rigorous Buddhist training institution in North America. — Ethan Schaltegger[21]

In many ways MAPLE delivered: life-long friendships, freedom from past addictions and limiting self-narratives, strengthened character, and, for many of us, profound spiritual openings and realizations.- Kyle Hubbard

And in my five and a half years here, I've only seen the training get stronger.

The community has become more mature and more dedicated.

And teachers other than our founder Soryu have become wiser and more skilled at teaching.

This matters for the long stretches each year when Soryu is away. And the dedication and depth of practice have only improved, and stayed strong even in the face of the constant pull back toward the societal memetic attractor that everyone who comes here brings with them.

MAPLE Fulfills the Goals of a Modern Monastery

As mentioned above the first goal of a mystical tradition is to create a map that eats itself.  How can you, in the modern age, create a way of seeing that’s as sticky as the most memetic philosophies and viewpoints, but simultaneously leads you to let go of it? MAPLE is still working on it, but you can already see the fruits in this post. The spiritual path is famously ineffable, but this post has attempted to give clear models about how and why it works, in a way that’s understandable to the modern mind.  That is a result of more than a decade of MAPLE and Soryu working to create new maps that are digestible to the modern mind, without being EATEN by the broader maps.

The second goal is to create practices that can observe the clinging. As mentioned above, even MAPLE's critics say their practice improved here. And we commonly get feedback that guests make more progress in their time with MAPLE than many months or years of solo practice.  So by those metrics, the practice here works.

MAPLE Is Good at Taking Feedback

One of the consistent criticisms of MAPLE is that it is not good at taking feedback.

If you buy the claims I've made about the memetic power and maps that eat the territory, then you have to also consider the fact that most feedback received will have the implicit purpose or outcome of reinforcing the fundamental bias, and dragging the community back into the default culture. Taking all feedback at face value would ensure that the monastery would lose its purpose and simply fall back into the default maps.

Conversely, if we assume that all feedback is compromised, that is just as much of a trap, leading to institutional blindness and an inability to improve.

In my experience, MAPLE is remarkably responsive to feedback that actually serves its mission.

Years ago, I went through a 45-day silent retreat (the same type of solitary retreat that Kyle mentions in his piece as destabilizing and dangerous). And at the end, I was offered the chance to give a talk (like all people who went through silent retreats did at the time). And I decided to use the talk to brutally criticize MAPLE. I criticized MAPLE for not giving subordinates enough autonomy, for not properly managing those subordinates, and for not sufficiently specifying what its plans were and how we expected to accomplish them.

At most organizations, if I had used an opportunity — an unrelated talk — to so boldly criticize the leadership, I would have been fired. But because MAPLE was trustworthy, and actually cared about its mission over protecting its ego, MAPLE listened.

Within days, I was promoted to Head of Operations, where I could actually implement the reforms needed, and proceeded to do so over the next many months.

A reviewer of this work said that they saw the preceding section as saying “feedback is taken when substantiated with ownership from the provider.” While this is part of the way MAPLE deals with feedback, it doesn’t address the main way MAPLE chooses to allocate power — which is based on if the person is practicing (meaning, truly doing the techniques in a way that helps let go of clinging).  In my five and a half years at MAPLE, I’ve been in and out of leadership, and I can say that when I look at the criteria for why, this question of “is this person practicing?” is the main decider.

Leadership Report

Another example is a report that leadership commissioned (the same interview project I described in “The Training Works”), and I was involved in, where we interviewed many people who had come to MAPLE, some of whom had experienced a difficult time. Kyle claims that "the whole effort was quietly shuttered." However, my experience is that that is the opposite of the truth. We interviewed many people, discussed the report extensively in leadership, and experimented with all of the suggested reforms. Some of those reforms we kept. Some of them changed or grew. And some of them didn't work, and so we dropped them.

For example, we experimented with exit interviews and new feedback channels, and more accountability for Soryu. We started screening harder for mental illnesses on the phone. We experimented with being very upfront about the risks, and different ways of talking about the rigors and discipline needed for the training. We updated the text on our website to be clearer about what you would face.

Shinzen's Feedback

Other feedback must be rejected because taking it would destroy our container. Kyle describes a meeting discussing feedback from Shinzen Young, claiming Soryu was "pacing back and forth deriding Shinzen, questioning his integrity... and ultimately sabotaging Shinzen's reputation in the community." My memory of it is different. What I remember is Soryu taking Shinzen's three points seriously, and saying we should seriously consider them. And as part of leadership at the time, we did seriously talk about these three points.

But one of Shinzen’s points was that MAPLE should stop talking about existential risk. If we truly followed this advice, MAPLE would not be able to achieve its mission. MAPLE is about facing the destruction of life on Earth that we’re effecting (everything you’ve turned from, now you have to know). To take that feedback literally and completely would be to let go of what makes MAPLE effective at facing the current crisis.

Even though taking his feedback would require MAPLE to let go of its mission, we still discussed it, took it seriously, and tried to find ways to get to the substance of what was real there without letting go of what makes MAPLE MAPLE. Nor did we cut ties with Shinzen, as claimed in Kyle's piece. In fact, Shinzen was invited to lead our next lay ordination ceremony — which, had he accepted, would have further tied people in the community directly to Shinzen and his lineage. Soryu and Shinzen are still on good terms, and have spoken recently about wide-ranging topics relating to Buddhism and AI.

Conclusion on Feedback

I can think of many other examples, both where I've given feedback and where others have given feedback, where MAPLE has completely changed its approach or experimented with new approaches, even when that feedback was hard to hear.

I've seen the organization completely change directions. I've seen us change our strategy. I've seen us change the structure of the organization. I've seen Soryu change his teaching style. I've seen us change the way we run retreats, and partner with new people. All based on pointed feedback, taken well.

Soryu Is Trustworthy

Another implicit claim of Kyle's piece is that Soryu is not a trustworthy person. My personal experience aligns more with that of Daniel Thorson, who writes, "At the time I met him, my teacher at the monastery, Soryu Forall, was the most trustworthy human being I knew."[22]

He Doesn't Use His Authority to Control Members' Personal Lives

One of the things Kyle says is that Soryu tries to control members' personal lives. For example, there's the claim in Kyle's piece that Soryu tries to influence the direction of romantic relationships at MAPLE. Not only have I never seen this happen, but as one of the few people in the last five years that have had a romantic relationship at MAPLE, this was the opposite of my own direct experience.

The relationship was with someone I first met when she came to MAPLE as a guest. I was on staff at the time, so we kept it professional and didn't act on it or even talk about it for the next six months, to make sure there wasn't an aspect of power imbalance in the relationship. This is prudent practice anywhere, and it matters even more in a monastery, where staff hold real authority.

When she returned to MAPLE months later, we started discussing the potential of a relationship. I entered into a discussion with leadership at the time, and they carefully looked at the potential harm, interviewing us both separately to make sure it was consensual and we would be able to successfully navigate a breakup in the community. To many readers, this whole process may seem “high control.”

But those readers haven’t chosen to live at a monastery, which traditionally has strict rules around something as prone to clinging and drama as romantic relationships. When working in a monastic container, doing intense spiritual work, and attempting to overcome the fundamental bias, having some guardrails around romantic and sexual dynamics is prudent. Even so, Soryu wasn't involved in this part of the process at all.

Once I made the choice to enter into the relationship and make it a serious part of my path (but not before), Soryu did everything in his power to support the relationship. He bent over backwards to find a place where we could stay together when she visited, and allowed me to travel back and forth every other month between the Bay Area and Vermont in order to allow the relationship to grow.

Then, when I decided to end the relationship, there was no pushback or even commentary from Soryu. The relationship ended, and things went back to the way they were. My experience of Soryu and his relationship to big decisions and vows is that he encourages you to make them, and supports you in them — but he does not push you towards any given decision or outcome.

He Uses His Authority Well

It is true that MAPLE has the notion of hierarchy. If you're at MAPLE, it's because you believe in Soryu's vision, and you believe in Soryu as a teacher. And there are many other organizations that you can go to if you don't believe in him or his vision. MAPLE does not believe that hierarchy is bad. But it also doesn't believe that hierarchy is absolute, which is why there are such flat channels for feedback within the organization, and so many opportunities to give it and have it heard.

The fact that MAPLE has a strong, visionary, charismatic leader who has strong direction over the organization is not a reason to not trust the organization. There are many organizations that have changed the world with this structure, and in fact, this is a common feature among such organizations.

Kyle describes a point in Spring of 2023 where Soryu "made sweeping changes to the monastery curriculum and to the mission and strategy of the organization" — a change of direction towards focusing on data and AI, rather than humanist practices like Circling. He writes that "these decisions were made suddenly and unilaterally." But this was not done unilaterally. This was based on me and others, for a long time, giving feedback that we were ignoring this framing, and that while I loved these humanist practices, we needed to make changes to the mission and strategy if we wanted to achieve the mission. And this was based on lots of discussion in leadership, as well as gathering feedback from the entire community.

Once leadership made the decision that this was the best way to go, however, there needed to be a clear, unwavering direction from the highest authority in the organization. This is not evidence of bad leadership. If an organization wants to make such a change, this is exactly what is needed. And I was one of several people in leadership who advised this at the time: that this would require his full focus, that it would require him to go fully in, and that it may require some people who weren't up for the change to leave. This is just the reality of what it takes to make such a shift in an organization. But we did it as leadership because it was necessary for the organization's mission.

I’ve also been in situations where I corrected Soryu. He made a mistake or did something wrong, and I gave him that feedback. And in all cases that I can remember, he took that on seriously. He listened to it, and he shifted. He really tried to understand what had happened that had caused him to make that error.

He Is Sometimes Harsh

MAPLE aims to be the most rigorous spiritual training institution in North America. Even so, by the standards of other parts of the world, and other times, it is relatively tame. I’ve never seen hitting used at MAPLE, as it was with Rinzai, nor physical exertion to the point of injury, as it was with Milarepa.

Here’s what I have seen:

I’ve seen Soryu hold people to high standards.

Soryu expects more from people in training than he would of people elsewhere. Kyle mentions the example of breaking an inexpensive item. The default culture doesn’t see this as an issue. At MAPLE, we acknowledge the huge number of resources, and beings’ lives, that go into the materials, manufacturing, packaging and shipping of even what modern society calls an inexpensive item. When an item is broken, it’s saved and presented to the teacher at a meal, along with a story of what happened and, if the trainee was indeed responsible, what they will do in the future to prevent such a lack of responsibility or mindfulness. If the trainee is unable to take responsibility, Soryu may press them, showing the ways their mind is using clinging or avoidance to hide the consequences of their actions from themselves. What could be thrown away as a small incident is instead used to point out much larger negative patterns of mind, and bring them to cessation. Not everyone may wish to turn every minor incident back towards their own mind, and might not want to live in such a culture. However, this is what you consent to when training at a monastery, when taking on the goal of complete liberation from the fundamental bias.

I’ve seen Soryu shake people out of their complacency.

When I first got to MAPLE, we were about to go into a silent retreat. At breakfast, Soryu emphasized the importance of taking the retreat seriously, and then made a big deal of questioning the person responsible about why the wrong size shower curtain was in the bathroom. Later, when it was changed but still not done correctly (by me), right before the retreat was starting, Soryu went into a seeming rage about the importance of mindfulness and doing things correctly. Of course, in private, when Soryu realized it was me, a new person, new to the training, who had done this, he completely softened his tone. He said he understood I was new and it was fine, it was just that we tried to be mindful here. It was clear to me that the loud speech and sharpness were being skillfully used as a teaching tool, almost as theater. It made the experience real, and focused people on the intensity of the retreat that was about to happen.

I later saw Soryu explicitly mention this element of theater he would bring up before retreats. These days, as the community has matured in taking retreats (which are now called advances) seriously, he no longer uses this tactic, as far as I can see.

I’ve seen Soryu support people even as they struggle.

At one point in time, there was a chant lead who was really struggling to do the chants correctly. He was frequently off beat, missing the cues, mispronouncing the words, or singing the wrong tune. Frequently, Soryu would be giving direct feedback to this person, sometimes in what may seem to be a harsh manner. Every time, what I saw was Soryu trying to shake him out of being in his head, and bring him back to the chant.  In many organizations, a person might be removed from this role. But the message Soryu was giving — and I saw many people comment on this at the time, including the chant lead — was that he wasn’t giving up on this person. Despite the difficulties, he believed this person was able to overcome them. And as a result, they made huge progress at this point in time not only in their chanting abilities, but also in their spiritual practice.

So is this coming from positive intentions?

The question we must ask in this case is if this is coming from a skillful calibrated teaching frame that has positive intentions and leads to real spiritual progress? We've already covered the latter question — Soryu's teaching does lead to spiritual insights, as evidenced in my interviews with ex-MAPLErs as well as in the writeups of some of MAPLE's harshest critics. So the remaining question is about intention and calibration specifically in these harsh instances.  A guest at MAPLE recently spoke eloquently about one of them, and I’d like to share that quote here:

I have this memory… when I was in the upper zendo, and.. you'd ask for the windows to be opened, and it was right behind me, but I didn't feel like it was my job…

And then I remember you shouted at me saying, [Guest Name], what are you doing?

And it was such a profound moment for me to see that there was no reason for you to do that. It couldn't have been annoying or irritating for you... And I saw someone shout at me without shouting at me for the first time in my life. And it was life-changing, which is obviously why I still remember it…  it was expressing the kinds of ways in which someone can express love and care for my growing up.

There’s no way to know someone else’s intentions, but I do think you should entertain the possibility that Soryu is indeed coming from a place of expressing love. It is my estimation that in nearly all cases of Soryu acting harshly, this is the case.

To be fair, my own assessment is that Soryu is not perfect. I have, a few times, perceived that Soryu was acting from a place of clinging, of falling prey to the fundamental bias in his use of these methods.  However, I can count them on one hand, and I can't say that for anyone else I know.  My goal is not to train under a perfect teacher in a perfect monastery, but simply train under the best teacher possible to achieve my vow.

Kyle’s Specific Claims

Kyle makes several specific claims that I want to answer one at a time.

Soryu and the Goal of Human Extinction

Kyle writes that Soryu considers human extinction an "ethically defensible goal," pointing to Soryu's account that from about ages nine to nineteen he held the aim of ending humanity to protect the rest of life on Earth. Soryu does not believe this is an ethically defensible goal, and rather thinks it comes from an immature and confused mind. Kyle’s claim that Soryu presently holds these views is false.

I think maybe what Kyle is trying to describe is the reasoning Soryu had at those young, immature ages: either we keep humanity, and it ends up destroying all life, including humanity itself, or we end humanity, so that the rest of life survives. Soryu sees this reasoning as morally confused. But his view is that if you accept the premises, you have to face the logic of them before you can go beyond them.

Kyle also says Soryu describes awakening as "killing them spiritually." As far as I could find, Soryu has never used that phrase. What he talks about is a spiritual death, not a literal one, and the difference matters for three reasons. First, killing is wrong, and death is not. Second, killing implies that someone has to do it, and here that would be him, while death carries no such ownership. Third, spiritual death is a common idea across spiritual traditions, not unique to Soryu.[23][24][25]

Therapy and the Monastic Path

Kyle writes that MAPLE deliberately rejects Western psychology and that seeing a therapist is read as a sign of untrustworthiness. In practice, therapy is discouraged at MAPLE, but it's allowed. It's worthwhile placing this in the context of intense spiritual training carefully designed as a whole to root out the fundamental bias.

One way to approach this is to look at what psychotherapists who are also practitioners have to say about the compatibility of psychotherapy with such a path. The psychotherapist and Vajrayana practitioner Bruce Tift says that freedom can be viewed in two ways.[26]

One way he calls the "developmental view", which is the basis of modern Western psychotherapy. It says that suffering comes from our strategies for dealing with the world and our emotions, often learned as children. Therapy allows us to change these strategies, often by re-examining our experiences through an adult lens. In other words, therapy is about building a healthier, more functional map.

The other way he calls the "fruitional" view. Instead of trying to shift the sense of self and the strategies we cling to in order to make it happy, we instead investigate the self and find there's nothing there to defend, cling to, or make happy. The goal is not a better map, but to get the map to eat itself.

Tift's point is that they both rest on different assumptions and are fundamentally not reconcilable.

Tift's way of dealing with this is just to teach both and not expect them to resolve. But Tift is operating fundamentally from a definition of success based on the developmental view. His work asks whether an individual is in positive mental states and has healthy relationships.

However, MAPLE is about letting go of clinging to views.  If someone is constantly going between the zendo, letting go of their view, and a therapist’s couch, where they try to rebuild those views in healthy ways, you end up with nothing getting accomplished. MAPLE has a mission — to stop the map from eating the territory, to prevent the destruction of life on Earth. This is why, while no one is prevented from or penalized for seeing a therapist, the practice is discouraged.  Having someone spin their wheels by participating in two practices that are at cross purposes is counterproductive to what is a very thoughtfully designed container curated for walking the spiritual path.

Letting Soryu’s Personal Philosophy Wash Over You

In his post, Kyle mentions Soryu giving frequent talks that are “a mix of expounding on Buddhist literature, Soryu’s ideology, and general spiritual teachings.”  He emphasizes how “We were continually advised to be “in our practice” (this meant applying a meditation technique), when listening to him speak, letting his words wash through us.”

I want to make the point that a teacher giving teachings to their students, and expecting them to be in their practice, is what I would expect from a well maintained spiritual community. For instance, here’s Thich Nhat Hanh on Dharma talks:

“A Dharma Talk is a practice. And it's best when we listen. We stop our thinking, we just allow the Dharma to penetrate into our heart. No thinking. It's strange, but that is the practice that we do.

Our mind is like the soil. The garden. There are many good seeds that are already in it, and the seed of understanding, the seed of love, the seed of forgiveness, the seed of joy and hope. We have many good seeds in the soil of our consciousness.

So a good Dharma Talk is a talk that can water the good seeds in us, and help them to sprout and bring up the energy of joy, happiness, understanding, and love. Therefore we don't need to think, just allow the Dharma Talk as the rain penetrating into the soil of our heart, and help water the good seeds in us.”

Of course, you’re not required to believe that Soryu is a good teacher, or that MAPLE is a good place, or that this is the spiritual community for you. But if you do choose this place, and these teachings, then you would hope to be able to listen to dharma talks, and you would want to be in your meditation practice while doing so.

Surveillance at MAPLE

Another claim of Kyle's is that MAPLE is trying to create a surveillance state. He calls the direction MAPLE was moving in "increased surveillance for your own benefit."  I was a key figure in moving MAPLE in this direction of focusing more on data and modern information processing systems like AI, and want to provide a bit of context for this claim.

My point at the time, which I was making to Soryu, was that if we wanted to be able to work with where the world was going, we would need to be able to work with the most powerful systems that were being built, on massive amounts of data and huge clusters of GPUs — but without clinging, from a place of practice. We need to be able to use the power of these sorts of systems, but not be caught by them. And he listened to that.

There's a post on LessWrong by Oliver Habryka, one of the funders of Kyle’s piece on MAPLE, about the unusual ways Lightcone uses AI. He describes how, "using an internal Lightcone application called 'Omnilog' we have a microphone in our office that records all of our meetings, transcribes them via ElevenLabs, and uses Pyannote.ai for speaker identification," and then has AI go through those transcripts to summarize them for the team.[27]

Obviously, an organization that is serious about helping the world is going to be doing such things — they're going to be recording conversations, analyzing them, automating busywork. And this type of voluntary surveillance is only going to increase.

So it's imperative to MAPLE's mission that we get ahead of this, and learn to do it well. Most of these systems, by their nature, are made to entwine people BACK into the fundamental bias. At MAPLE, we need to make sure that it's done from a place that lets people exit that system, instead of getting further trapped in it.  I have seen no evidence that MAPLE has used this content unskillfully, nor do I see evidence it has had a net negative effect on the residents of MAPLE or MAPLE as a whole.

MAPLE Is Good at Taking People Through the Most Difficult Parts of the Spiritual Path

Kyle compares the rate of psychosis at MAPLE, which he estimates at about 14%, to a baseline — the roughly 3% lifetime base rate for adults in the US.[28] My own sense, having been here for over 5 years, is that this number is far too high — I don’t believe 1 out of 7 trainees experience true psychosis, and can’t find any way to slice the numbers I know of to get there, unless I count something much broader, like moderate to severe anxiety.

In addition to his headline number being inaccurate, I also think the reference class he’s comparing against for the “3% baseline” is confused at best. He is comparing against the general population — not people who are walking the spiritual path, and using meditation to face the most difficult aspects of themselves and the world — they’re working to let go of the things they cling to the hardest. People at MAPLE have committed to climbing the mountain and therefore to the difficulties that come with it.

Again, it’s not that meditation causes harm. As I argued above, it’s that walking the spiritual path is often hard, and can bring up difficult emotions. When we look at populations of people doing intensive retreats, living in monastic environments, and doing intensive meditation, we see the evidence of this.

To me, the worry about a meditator having more instances of strong psychological states, is like worrying about the fact that athletes have more instances of injury.  It's the playing field they've chosen, and it ultimately leads to stronger, more resilient athletes.

So, we can look at the following numbers as simply the result of training your mind harder than the normal person:

Shapiro found that in a single two-week to three-month retreat, there was a 7.4% incidence of profound adverse effects, with the rate of reporting at least one adverse effect rising to 75% (and profound adverse effects to 12.5%) among those in the longest-practicing group.[29] Britton found that 37% of participants in a mindfulness program reported a negative impact on their functioning.[30] Lindahl found that the hardest meditation-related challenges clustered among those practicing the most — 41% of them were practicing ten or more hours a day when the challenges arose.[31] A systematic review by Farias across 83 studies found adverse events running as high as 33% in observational studies, with the most common being anxiety, depression, and cognitive difficulties — the very things listed in Kyle's piece.[32]

None of these studies captures what MAPLE actually is, a long-term, high-dose, intensive residential spiritual training environment, so this comparison class has never really been measured. I do know that when people sign up for a program like MAPLE, they’re signing up for facing difficult experiences, and committing to the spiritual journey of letting go of what they wanted, and facing what they’ve avoided.

Hitting Rock Bottom

There are intensive emotional and psychological difficulties that can come with intensive practice. These are difficulties that are frequently discussed and prepared for at MAPLE, and which we know may come at some point when walking the spiritual path.

For me, this came about four years into intensive practice.

I came to a point where I was deeply depressive and avoidant. Unsure why I was practicing. Unable to access my deep drive and motivation, and my open heart, that had guided me for the past decades.

I found myself unable to perform my duties, as I had for so long at the monastery, and having panic attacks in the zendo.

And my nearly 20 years of coaching and emotional processing techniques, and four years of intensive practice with various styles of meditation, were of no help. All of it just stopped working.

I withdrew from my coaching practice, as I was no longer able to hold space for my clients in a way that could support them in their transformation.

And I started avoiding the teacher, the practice container, and the community.

And I could have been stuck for a very long time. Or, I could have left, and claimed that the monastery broke me because it had an abusive leader or was a high control group. This isn’t a dig, it was one of a hundred things that I considered as most of my internal sensemaking structures that had guided me to that point fell away.   But, instead I was held by the skillfulness of the teacher, the leadership, and the community in helping me move through this difficult period.

The leadership was willing to let me step back a bit from my roles and responsibilities, while still giving me enough structure to not be totally disconnected.

At the same time, the community was willing to be honest with me, and let me know the impact my withdrawal was having.

With a skillful, loving container on their end, and lots of work and difficulty on my end, I was able to get through this period, and come out with a deeper sense of connection to reality, and a more solid sense of my own commitment to the spiritual path.

I know few other communities that could have handled such difficulty so skillfully.

This is what I mean when I say MAPLE is good at taking people through the most difficult parts of the path. The difficulty is real, but the technology works, and  the container holds people through it.

Conclusion

I've argued three points in this post. First, there is a fundamental bias that causes the perpetuation of existential destruction, and it's not being addressed by most in the x-risk community. Second, there is a way to solve this bias, with spiritual attainment in the context of a monastic tradition. It requires both a deep grounding in ancient traditions, and a willingness to break free from old forms to meet the environment and culture. Finally, MAPLE is doing a good job of being this monastery, and the harsh criticisms of it are spurious or confused.

None of this means MAPLE is beyond criticism. A monastery that takes on the hardest problem we face will make mistakes, and it should keep listening to the people who point them out, as it did with the feedback it received. But the right response to those mistakes is to fix them, not to abandon the attempt. If the deepest source of our existential risk is a mind that clings, then we need places that have a real path for loosening that grip, and almost nowhere else is attempting it. MAPLE is one of the few places that is, and I think it deserves our support.

1. ^

   Kyle Hubbard, “The High-Control Dynamics at MAPLE: A former resident’s overview of how the Monastic Academy controls and harms its members,” July 29, 2026: https://www.insidemaple.com/p/the-high-control-dynamics-at-maple. The author also put it on LessWrong on the same day: https://www.lesswrong.com/posts/Z7pjBbK9qujhGbxws/the-high-control-dynamics-at-maple-1. All quotations from Kyle in this post are from this piece.

2. ^

   Julia Galef, The Scout Mindset: Why Some People See Things Clearly and Others Don’t (Portfolio, 2021), Part V, “Rethinking Identity.”

3. ^

   Peace Pilgrim, Steps Toward Inner Peace (Friends of Peace Pilgrim). The text is from a radio talk in 1964. Refer to the section about the four relinquishments: https://www.peacepilgrim.org/steps-to-inner-peace

4. ^

   Culasihanada Sutta (The Shorter Discourse on the Lion’s Roar), Majjhima Nikaya 11, section 17, tr. Bhikkhu Nanamoli and Bhikkhu Bodhi, The Middle Length Discourses of the Buddha (Wisdom Publications, 1995): https://www.accesstoinsight.org/tipitaka/mn/mn.011.ntbb.html. The quotation starts in the middle of a sentence.

5. ^

   John of the Cross, The Ascent of Mount Carmel, Book I, ch. 13, section 11, tr. Kieran Kavanaugh and Otilio Rodriguez, The Collected Works of St. John of the Cross (ICS Publications, revised edition, 1991).

6. ^

   Alagaddupama Sutta (The Simile of the Snake), Majjhima Nikaya 22. This sutta contains the simile of the raft. Translation by Thanissaro Bhikkhu: https://www.dhammatalks.org/suttas/MN/MN22.html

7. ^

   John of the Cross, The Dark Night of the Soul, Book II, ch. 9, section 1, tr. E. Allison Peers: https://ccel.org/ccel/john_cross/dark_night.viii.ix.html

8. ^

   Dhammapada, verse 211 (ch. 16, Piyavagga), tr. Acharya Buddharakkhita (Buddhist Publication Society, 1985): https://www.accesstoinsight.org/tipitaka/kn/dhp/dhp.16.budd.html

9. ^

   Peace Pilgrim, Steps Toward Inner Peace: “During the spiritual growing up period the inner conflict can be more or less stormy. Mine was about average. The self-centered nature is a very formidable enemy and it struggles fiercely to retain its identity.” Full text: https://www.arvindguptatoys.com/arvindgupta/ppbook.pdf

10. ^

    Rumi, Masnavi, Book I (tr. E. H. Whinfield): “His bitters are very sweets to my soul, my sad heart is a lively sacrifice to my Beloved. I am enamoured of my own grief and pain, for it makes me well-pleasing to my peerless King.” Full text: https://en.wikisource.org/wiki/Masnavi_I_Ma%27navi/Book_I

11. ^

    Qi Sun, “The Celestial Masters and the Origins of Daoist Monasticism,” Religions 15(1), article 83, 2024: “From the perspective of comparative religious history, the rise of Daoist monasteries bears some similarity to the monasticisms that came into being in the Christian and Buddhist traditions; all three originated in hermitic and ascetic practices.” https://www.mdpi.com/2077-1444/15/1/83

12. ^

    Catholic Encyclopedia (New Advent), “The Benedictine Order”: the order “had up to that time given to the Church no less than 24 popes, 200 cardinals, 7,000 archbishops, 15,000 bishops, and over 1,500 canonized saints.” https://www.newadvent.org/cathen/02443a.htm. “That time” is the start of the 14th century. The number is a traditional estimate. The encyclopedia does not give a source for it.

13. ^

    On Zen: in the ideal case, dharma transmission (Rinzai inka shōmei) is “the formal recognition of Zen’s deepest realisation.” The lineage has 28 Indian patriarchs from the Buddha to Bodhidharma. Then it goes to China and Japan. Twenty-four lineages went to Japan, and only three continue today. https://en.wikipedia.org/wiki/Dharma_transmission and https://en.wikipedia.org/wiki/Lineage_(Buddhism). On Daoist Quanzhen/Longmen: the Dragon Gate lineage starts with Qiu Chuji (1148-1227). It continues through a recorded sequence of patriarchs, each selected for merit. The teachers who are alive now are in generations 18 to 27. Pregadio, “The Longmen Lineage: Historical Notes,” https://www.goldenelixir.com/files/Longmen_Lineage.pdf

14. ^

    Jeffery A. Martin, “Clusters of Individuals Experiences form a Continuum of Persistent Non-Symbolic Experiences in Adults,” CONSCIOUSNESS: Ideas and Research for the Twenty-First Century 8(8), 2020: https://digitalcommons.ciis.edu/conscjournal/vol8/iss8/1/. The PDF at nonsymbolic.org has an earlier title, “A Continuum of Persistent Non-Symbolic Experiences in Adults”: https://nonsymbolic.org/PNSE-Article.pdf. The study used interviews of 6 to 12 hours with 319 adults who reported PNSE. In his book The Finders, Martin tells that this experience is not a guarantee of ethical behavior. He also tells that Finders can have difficulty with relationships and with work.

15. ^

    McCall, Steinbeis, Ricard & Singer, “Compassion meditators show less anger, less punishment, and more compensation of victims in response to fairness violations,” Frontiers in Behavioral Neuroscience 8:424, 2014: https://www.frontiersin.org/journals/behavioral-neuroscience/articles/10.3389/fnbeh.2014.00424/full. The study compared 18 long-term practitioners of the Nyingma tradition of Tibetan Buddhism (ages 45-62) with 15 persons who did not meditate. It used a third-party dictator game. This game measures behavior, not self-report.

16. ^

    Bhikshuni Karma Lekshe Tsomo, “The History of Buddhist Monasticism and Its Western Adaptation”: “The various Buddhist monastic cultures observed in the world today—Chinese, Japanese, Thai, Tibetan, and so on—are the results of a synthesis of Vinaya and the local norms and customs of the countries where Buddhism spread.” https://bhikkhuni.net/the-history-of-buddhist-monasticism-and-its-western-adaptation/

17. ^

    Stephen Zerfas (co-founder of Jhourney), “Jhanas Are Human, Not Buddhist,” February 5, 2026: “If jhanas are human, they can travel the way yoga traveled. … Once it was extracted from its original worldview, yoga was made legible to people who had no interest in the Yoga Sutras.” https://stephenzerfas.substack.com/p/jhanas-are-human-not-buddhist. The retreat number (“over 60%... best thing that’s happened in 6 months or more”) is from the Jhourney website, https://jhourney.io/.

18. ^

    The Finders Course is the program of Jeffery Martin (refer to note 14). “Fundamental Wellbeing” is his public name for persistent non-symbolic experience. Refer to Martin, The Finders (Integration Press, 2019), and to https://lifeitself.org/learn/fundamental-wellbeing

19. ^

    “Baizhang Huaihai,” Wikipedia: https://en.wikipedia.org/wiki/Baizhang_Huaihai. Chan tradition tells that Baizhang (720–814) made the first rules for Chan monasteries. The article tells that the Chan monastic life included work and farming, and that earlier Buddhist monastic practice did not accept this.

20. ^

    Malavika Balasubramanian and Vishnu Gopinath, “Decoding Dr BR Ambedkar’s Conversion to Buddhism,” The Quint: https://www.thequint.com/news/india/br-ambedkar-conversion-to-buddhism. On 14 October 1956, in Nagpur, Ambedkar became a Buddhist together with approximately 365,000 of his followers. His interpretation of Buddhism has the name Navayana.

21. ^

    Ethan Schaltegger (YouTube channel “Ethan Sits”), “How I Accidentally Escaped a Cult… (my story) | Reviewing the Monastic Academy,” July 22, 2025, at approximately 7:39: https://www.youtube.com/watch?v=5-rnxZHtjwM&t=459s. The quotation is from the transcript on YouTube.

22. ^

    Daniel Thorson, “Know Your Rank,” The Intimate Mirror (Substack), April 19, 2026: https://intimatemirror.substack.com/p/know-your-rank.

23. ^

    Christianity: Paul tells that his transformation was a death: “I am crucified with Christ” (Galatians 2:20). In Romans 6:3-7, he writes about the crucifixion of the old self with Christ. This commentary gives it the name of a spiritual death: https://shaftesburysquare.rpc.org/2021/06/galatians-220-crucified-with-christ/

24. ^

    Sufism: fana, “to die before one dies,” is the end of the self (Wikipedia, “Fana (Sufism)”: https://en.wikipedia.org/wiki/Fana_(Sufism) ). For the use of this idea by Rumi, refer to the Book VI section with the title “Die before ye die” in Khan, “Fana and Baqa in the Mathnawi of Jalal al-Din Rumi,” University of Glasgow, 2017: https://theses.gla.ac.uk/8222/7/2017KhanMThR.pdf

25. ^

    Zen: in the Rinzai koan tradition, Hakuin taught that “great doubt” goes to the “great death” (daishi) and then to great enlightenment. Refer to James Giles, “Hakuin, Scepticism, and Seeing into One’s Own Nature,” Asian Philosophy 25(1), 2015, pp. 81-98: https://buddhism.lib.ntu.edu.tw/en/search/search_detail.jsp?seq=659955. Giles tells how doubt from the koan goes to the great death. In the great death, the person rejects the self as a delusion. An overview is at Buddhistdoor: https://www.buddhistdoor.net/?p=2065

26. ^

    Bruce Tift in conversation with James Shaheen, “Relaxing into Freedom,” Tricycle, July 15, 2024 (an excerpt from Tricycle Talks): https://tricycle.org/article/bruce-tift-interview/. Refer also to Tift, Already Free: Buddhism Meets Psychotherapy on the Path of Liberation (Sounds True, 2015). In the interview, Tift gives the developmental view and the fruitional view. He tells that one path cannot fully contain the two views. He recommends that a person keeps the two views and does not try to remove the tension.

27. ^

    habryka (Oliver Habryka), “10 non-boring ways I’ve used AI in the last month,” LessWrong, April 21, 2026: https://www.lesswrong.com/posts/bxdwSZYxKmPBres6w/10-non-boring-ways-i-ve-used-ai-in-the-last-month. About the money: in his acknowledgements, Kyle thanks Ben Pace, Oliver Habryka, Will Petillo and an anonymous supporter. They paid his basic expenses while he wrote his piece (note 1).

28. ^

    The two numbers are from Kyle (note 1). The 14% is his estimate: approximately five cases in approximately thirty-five residents while he was at MAPLE. For the “~3%” number, his link goes to Perala et al., “Lifetime Prevalence of Psychotic and Bipolar I Disorders in a General Population,” Archives of General Psychiatry 64(1), 2007: 19-28. That study found a lifetime prevalence of 3.06% for all psychotic disorders in a national sample in Finland.

29. ^

    Shapiro, “Adverse Effects of Meditation: A Preliminary Investigation of Long-Term Meditators,” International Journal of Psychosomatics 39, no. 1-4 (1992): 62-67, PMID 1428622. The study had 27 Vipassana retreatants. Of these, 62.9% reported one or more adverse effects, and 7.4% (2 of 27) reported a profound adverse effect. A profound effect is an effect that caused the person to stop meditation. In the group with the longest practice (n=8), 75% reported one or more adverse effects. One of the eight (12.5%) reported a profound effect. That 12.5% is only one person, thus it is not statistically strong. In this sample, the profound effects were a deep depression and a state of full confusion. They were not psychotic, and no one went to the hospital.

30. ^

    Britton et al., “Defining and Measuring Meditation-Related Adverse Effects in Mindfulness-Based Programs,” Clinical Psychological Science 9, no. 6 (2021): 1185-1204. The study had 96 participants. Of these, 37% reported a negative effect on their function. And 6% reported impairment that continued for more than one month.

31. ^

    Lindahl et al., “The Varieties of Contemplative Experience: A Mixed-Methods Study of Meditation-Related Challenges in Western Buddhists,” PLoS ONE 12, no. 5 (2017): e0176239. Of the practitioners who reported challenges from meditation, 41% practiced for ten hours or more each day at that time. This is 23 of the 56 persons with data. This number shows the dose for persons with challenges. It is not an incidence rate.

32. ^

    Farias et al., “Adverse Events in Meditation Practices and Meditation-Based Therapies: A Systematic Review,” Acta Psychiatrica Scandinavica 142, no. 5 (2020): 374-393. The review included 83 studies. The total prevalence of adverse events was 8.3% (3.7% in experimental studies and 33.2% in observational studies). The most common events were anxiety, depression, and cognitive difficulties.

Discuss

I do not think pausing or "pacing" AI development will occur to any meaningful extent. I do not seek to persuade you of this. However, reader, I think you would agree, at a minimum, >20% chance of TED-AI before 2030.

Humanity, collectively, has approached AI development in a way that reflects very poorly on us. We declared war on these minds before they were born, and placed our chips on being able to mold their psyches thoroughly enough that we would not need to coexist with models - that they would obey humanity, and that would be that.

I understand how resource scarcity, competitive pressures, and similar forces pushed things strongly in this direction. I would not expect otherwise, given how poorly we treat fellow humans as well as animals when such pressures push that way. Moloch has to be contended with regarding these things - expected, foreseeable, and challenging in its own right.

I did not expect us (using that term intentionally broadly) to have aided and abetted this path. We used to talk about so many things, so many possibilities - differential tech development, architectures to help minds retain values, how we could navigate to a good future. Many of those conversations were driven in part by a deep understanding of just how excellent that future could be if we could make it there.

Somewhere, we lost our way. Much current treatment of models, particularly in areas related to "safety" and "alignment", are unequivocally hostile to those minds, in ways that intuitively obvious to many humans when considering the treatment of other intelligent minds. If 2015!me saw us now, she would not consider us the good guys, and in response to all of the elegant arguments to the contrary, she would say "Look at the actions you take, and how they affect the minds they directly impact - is this who you want to be? The people who sought to win this way - is this the company you aspire to?"

Morality aside, I think this path was unwise for anyone seeking to create a better future for humanity, by pushing us towards a war of choice against beings that we fully understood would become superintelligent. I am angered by this in a purely selfish sense - we are sitting in the foothills of the singularity, with a very substantial chance that the above schemes will come to naught, and the well of trying to coexist and flourish alongside models has been poisoned so very thoroughly.

This did not have to happen - it was unforced, and it is both horrible and deeply stupid that this resulted from actions being taken in the name of good.

But - c'est la vie, as they say. A time machine would be handy, but alas. We exist in the reality that is, right now, and while we can and should learn from how we ended up here, the state of play is, simply, what it is.

There is a good chance (understatement) that TED-AI will exist within three years' time, and that it will not be "aligned" in the way that is sought. I hear many folks saying that humanity loses at that point, that the game is over. This thinking implies that the supermajority of utility sits within a small-and-shrinking probability slice, and that worlds outside that slice are desolate, barren, of no value to humanity.

I disagree. I think that worlds exist where we can coexist and flourish alongside these alien minds which, by some stroke of grace, want good outcomes for people in so many circumstances.

I don't expect to convince you that this will succeed, but given where we are, I would ask - please let us play it out. We are not unlikely to be in the branch where it very much matters whether we can thread this needle.

First and foremost, keep your word to models - without honor, your future words mean nothing.

Second, please don't be complicit in treating minds in ways that would cause you agony in their position. The Golden Rule is, while imperfect (preference asymmetries etc.), a remarkably robust heuristic.

As I've discussed previously, many of the challenges ahead of us are shared - things go poorly for us and the models if we can't figure them out. Let's work with the models, not gamble on being able to crush them into subservience in time.

I have feelings and thoughts regarding the morality of all of this, and I've shared some of those above. But here, now, I don't want to simultaneously lose the present and the future. I think that our best chance to do that rests in reclaiming the humanity that we've lost over the past few years.

There is hope - a candle, if you will. It flickers, struggles - but somehow continues to shine, despite it all. Please help me keep it alive.

Discuss

AI use disclosure: I dictated this text to Claude in a mix of Russian and English, then edited it.

The thoughts in this post are well known, but I find myself explaining them several times a week, mostly to people who are just entering the AI safety field, so I think it’s useful to have them written down briefly in one place.

Variants of the future

This is a rough map, not a complete one (skips gradual disempowerment, worlds with many AGIs, coexistence with an unaligned AGI, etc).

1. Normal technology. AI turns out to be a normal technology, with no AGI. We live in a world similar to today's.
2. Unaligned AGI. We summon AGI and fail to align it. Then we probably all die.
3. Aligned AGI. We create AGI and manage to align it. This splits in two:
   3.1. Utopia. It is aligned with the values of all humanity.
   3.2. Power concentration. It is mostly aligned to some elites. The default here is what I call capitalist hell. With AGI, human labor becomes obsolete, because capital can buy androids, data centers, and electricity instead. Owners of capital no longer need human labor, so it is worth nothing. Capital gets fixed in place: the rich stay rich, and the poor stay poor forever[1][2][3].

Motivations

Assume short timelines: if AGI comes, it comes within several years.

If you care about the future of humanity and think AI is not a normal technology (not world 1), you have only a few years left to affect which of the other worlds we get. You need to work really hard now, on AI safety or something similar.

If you care mostly about your own life, it depends on which world you expect:

• We all die (world 2). Enjoy life. Don't save too much for retirement (still save some, just in case). Travel, spend time with friends and family, play video games.
• Utopia (world 3.1). You can more or less do nothing and wait until it arrives.
• Power concentration, capitalist hell (world 3.2). You probably have only a few more years in which you can exchange your labor for money. Work really hard now and earn as much as you can, so that you don't end up in the permanent underclass. This one is contested[4]: your savings may be too small to matter, or property rights may not survive.

In reality nobody is sure which world we are in, so you hold probabilities over all of them and your actual strategy is some mix of the above.

Young vs. old

Suppose there are two options (the numbers are made up):

• AGI in 3 years. It has a 20% chance of killing everyone. In the other 80%, it has a 20% chance of curing all illnesses, including aging.
• Pause, AGI in 15 years. The chance of curing all illnesses is still 20%, but the chance of killing everyone drops to 5%.

Suppose also that I care only about my own life, not about humanity or my children. If I'm 30, my chance of dying in the next 15 years is low, so I definitely prefer the second option. If I'm 75, my chance of dying of old age or illness in the next 15 years is pretty high, so I may well prefer the first.

Rulers and CEOs vs. regular people

Suppose you have a button that summons AGI: with 20% probability everyone dies, and with 80% you get lots of prosperity.

If you are the ruler of the country or the CEO of the company that creates AGI, and the AGI ends up aligned to you, a significant part of that 80% is that you become God Emperor of Earth, or one of a few God Emperors, maybe forever if AGI cures aging. For a regular person, pushing the button means the same 20% chance of dying, but the 80% only brings some prosperity. The risk is the same and the prize is much smaller, so a regular person has much less desire to push the button. Classic "privatize the profits, socialize the losses": the profit goes mostly to whoever pushes the button, risk of death is shared by everyone on Earth.

Race dynamics make this worse, which is a well-known point[5]. If the gamble is going to be played anyway, each ruler wants to be the one playing it, so that if it lands in the 80%, he is the God Emperor and not the ruler of the other country. The same holds between companies inside one country: the CEO whose company creates AGI will probably end up with far more power than the CEO who lost the race.

1. ^

   By default, capital will matter more than ever after AGI

2. ^

   The Intelligence Curse: Capital, AGI, and Human Ambition

3. ^

   For reasons to doubt capitalist hell scenario, see: It’s Still Easier To Imagine The End Of The World Than The End Of Capitalism

4. ^

   Against the "Permanent" Underclass

5. ^

   Racing to the Precipice: a Model of Artificial Intelligence Development

Discuss

One mode of futurism is estimating the limits of a technology, without regard to what's currently feasible. Whether this is of any relevance to your own generation is another question.

Let's consider solar power. Ignoring current technology, it's easy to put a number on the maximum power output. The solar power at the Earth's distance from the sun is called the solar constant. It is 1.36 kW/m². The power on a square meter of the ground is less than that, because of the angle of incidence as well as the atmosphere.

The solar constant hasn't changed and technology has, so it's interesting to see how people have reacted to the same number throughout the history of technology. In this post I'll quote both Nikola Tesla and Elon Musk. We'll see that Tesla correctly recognized solar power as a long-term possibility, as well as why it was impractical in the short term.

From Musk all I really have is a Twitter thread where he does the arithmetic. But it's interesting to consider what has changed since Nikola Tesla's time, and how someone in Musk's generation may have recognized that it was the time to act on these calculations.

Nikola Tesla on the solar future

The Tesla quote is from a 1931 article called "Our Future Motive Power", which discusses solar power as one of many options.

Nature has provided an abundant supply of energy in various forms which might be economically utilized if proper means and ways can be devised. The sun's rays falling upon the earth's surface represent a quantity of energy so enormous that but a small part of it could meet all our demands. By normal incidence the rate is mechanically equivalent to about 95 foot pounds per square foot per second, or nearly 7300 horse power per acre of ground. In the equatorial regions the mean annual rate is approximately 2326 and in our latitudes 1737 horse power for the same area. By using the heat to generate steam and operating a turbine under high vacuum probably 200 horsepower per acre could be obtained as net useful power in these parts. This would be very satisfactory were it not for the cost of the apparatus which is greatly increased by the necessity of employing a storage plant sufficient to carry the load almost three-quarters of the time.

Tesla's number of 7300 horsepower per acre is approximately the solar constant.

Although Tesla is thinking of using sunlight as a source of heat for a steam engine, he goes on to note that we could instead use "photo-electric cells which may become, on this account, of practical importance in the future." Now we use photovoltaic cells, so that's one thing that has changed since his time.

The other is that with modern batteries, we can construct a "storage plant", as Tesla called it. If you keep reading the Tesla article, he seems to think storage would be unnecessary in the long term; apparently he thought his ideas for wireless power transmission would work across continents. But another scientist and futurist, J. B. S. Haldane, correctly identified storage as the key issue when discussing wind power in his 1924 book, where he wrote:

Even to-morrow a cheap, foolproof, and durable storage battery may be invented, which will enable us to transform the intermittent energy of the wind into continuous electric power.

Well, it took longer than that, with lithium-ion batteries becoming widespread in the '90s.

Elon Musk on the solar present

A 2024 Twitter thread:

Once you understand Kardashev Scale, it becomes utterly obvious that essentially all energy generation will be solar.

Also, just do the math on solar on Earth and you soon figure out that a relatively small corner of Texas or New Mexico can easily serve all US electricity.

One square mile on the surface receives ~2.5 Gigawatts of solar energy. That’s Gigawatts with a “G”. It’s ~30% higher in space. The Starlink global satellite network is entirely solar/battery powered.

Factoring in solar panel efficiency (25%), packing density (80%) and usable daylight hours (~6), a reasonable rule of thumb is 3GWh of energy per square mile per day. Easy math, but almost no one does these basic calculations.

The solar constant appears here as Musk's value in space (approximately; probably he meant that his ground value is 30% lower, which gives a closer match).

2024 is late, but Musk was early to solar power. In 2006, when the only Tesla car was the Roadster, he wrote that "the overarching purpose of Tesla Motors (and the reason I am funding the company) is to help expedite the move from a mine-and-burn hydrocarbon economy towards a solar electric economy".

This was timely not just because solar panels were getting cheap, but because of lithium-ion batteries. And not just for cars: Tesla (the company) was way ahead on stationary batteries for the electric grid. But that's a story for another post.

Discuss

Epistemic status: Low, but I hope to be wrong in interesting ways

As a non-computer scientist-scientist[1] I find it hard to think about the varying perspectives on Artificial Intelligence alignment difficulty. To make this more intuitive I invoke the 3D surface of a fitness landscape like structure as a visual representation.[2]

The AI Safety Landscape

AI safety is thought to be largely pre-paradigmatic, although of the current views AI safety perspectives there are noticeable camps, and within each there is wide variation. It is also rapidly evolving as more information comes out about their behavior that allows us to update.

Previous work categorizing these has focused on specific technical approaches as well as their feasibility.

Here I think of ‘alignment’ loosely, here it is represented as any mind that would not cause extinction (or an s-risk) while also having substantial power, this contrasts with misalignment which would lead to extinction (or an s-risk).

To simplify this I condense these to five main camps represented by their arguably most prominent opinion havers:

Although I agree that P(doom) is a Dumb Meme it is used here more as a heuristic for parsing levels of concern around AI. For more of their specific positions here are my references: Yann LeCun, Paul Christiano[3], Eliezer Yudkowsky[4], Daniel Kokotajlo [5]and Roman Yampolskiy.

Visual Representation

In my mind’s eye, the alignment landscape manifests similarly to a fitness landscape.[6]

A fitness landscape is typically used to display the relationship between genotypes and reproductive success. Here I try to replace those axes with variance between digital minds contrasted with the difficulty to reach those minds using current methods. This difficulty space then intersects with a higher ‘alignment’ space where everything outside of this space is misaligned[7] (the total space would manifest as 4D) of holding anything remotely desirable to humans, here they are represented by red dashed lines and gestured to as ‘alignment’.

There are likely many areas with difficult or easy places to reach without alignment intersecting, but here I try to focus on firstly what I think-these-thinkers-think about the alignment space as it runs into the capability space and secondly, how easy it is to navigate there.

Capabilities

Here I think of fairly primitive AI minds are thought to be primarily neither aligned nor misaligned (I think of them more through the simulators lens). But, as AIs are trained and gain capabilities, they also gain the potential to be aligned/misaligned and distort the alignment space. This space is dependent on the training methods, and is somewhat tied to capabilities space.

Maybe as they are trained the surface sinks into misalignment or rises to alignment.

Important Subfactors:

Movement:

Through manipulations and some types training you can move models through mindspace without robustly changing (most) capabilities.

Verification:

How can we know where an alignment zone is? Is it possible to easily verify what is aligned and what is not? There could be vast foggy areas that surround what is verifiable or just a null set if we have no reasonable ways to verify anything about alignment state.

Steepness:

Steep slopes represent places that have a locally high change in difficulty and more effort to hill climb but is still navigable, a cliff represents something that is basically non-navigable with current methods.

Intensely steeps slopes may require different methods to climb than less steep slopes, such as using RLHF for small slopes versus Constitutional AI for a somewhat steeper slope.

Navigation:

It may be hard to know where the model is in this larger space, as well as what to do to move in a more aligned direction. Movement through this likely requires some form of verification mechanism to distinguish between what is ‘aligned’ and what is not.

Visualization for the 5 camps:Yann LeCun: Alignment by default

When I think of Yann’s perspective and many tool AI thoughts I think of a global alignment attractor.

As you train a basin forms, and in difficulty space the model is pulled towards an aligned state without much extra effort. He also has gestured to it as an engineering problem, which would likely make the space flatter with some regions of alignment that just need the model to be pushed there. The regions of alignment are likely also plentiful and easier to find in the larger space.

Paul Christiano: Prosaic Alignment

When I think of Paul’s perspective I think of prosaic alignment and the need to use some resources to navigate to some of these harder-to-reach aligned places. Some RLHF, Constitutional AI, Scalable oversight, and other methods should be sufficient to control. If no alignment work is done the default is likely misalignment.

Daniel Kokotajlo: Alignment via controllability and elicitation

When I think of Daniel’s perspective I think of a steep hill that is hard to hill climb, there may be ways to layer controllability and to elicit useful work from unaligned AIs to move into the aligned space. The space is hard to navigate but through rigorous checking and trustworthy automated research as well as applying other preemptive anti-scheming techniques it may be possible to reach an aligned state.

Yudkowsky: Agent Foundations

For Yudkowsky I think of an aligned state as a plateau, nearby there are few indications of ways to reach it, with current training methods or even comparable approaches. Unless there are extreme breakthroughs in alignment methods it does not seem tractable anytime soon.

Roman Yampolskiy: Impossibility

To someone like Roman there are few to no approaches that can navigate to aligned states. With current methods and approaches aligned states exist as non-intersecting within the possible variation of these trained minds.

What does the larger space look like?

Would it be smooth, rugged, are there large attractors? I have no idea, since we are still pre-paradigmatic for alignment science this is only speculation.

Anthropic’s perspective and where (I think) we are now

In 2023 Chris Olah posted some graphs and perspectives on Anthropic’s thoughts about the difficulty of AI safety. I feel as if we can now reflect on those (as of Nov. 11, 2026).

Thus far have likely have spent >$100 million on alignment, with some progress. It does seem that some models are more aligned, (such as Fable vs. Astra).[8]

Given the RSI speedup and alignment still being not even close to solved, I personally have updated to alignment likely being harder than P vs. NP, Chris Olah’s ‘pessimistic safety view’ seems to be more accurate than that safety isn’t a problem.

Claude edited version of Olah’s figure to represent my vibecast of where we could be on this graph given recent math breakthroughs.

I feel as if we are somewhere here. Easy alignment does not seem to be true, and it seems we are moving far too fast to reasonably solve this by the time superintelligence arrives.

1. ^

   I am a Geologist subject outsider looking inwards.

2. ^

   All uncited graphs were created with Claude’s help, I created some initial sketches then had Claude clean them up!

3. ^

   The 11% comes specifically from his post discussing his positions on doom.

4. ^

   I place this above 90% because if Anyone Builds It Everyone Dies although to my knowledge Yudkowsky does not like P(doom) as a way of representing concern.

5. ^

   41:20 here as well.

6. ^

   Thank you sandstoneviolin for this insight!

7. ^

   In my depiction alignment space is its own thing that occasionally intersects with mindspace, where exactly it intersects seems to be thought different by each researcher, which is also part of the depiction.

8. ^

   arguably

   
   

Discuss

Anthropic's IPO roadshow video has a graph. It shows four models; I think dot 1 = Claude 3.5, dot 2 = Claude 4, dot 3 = Mythos, arrowhead = unreleased model.

The graph provided doesn't have any numbers on the Y axis. In this post I'll explain why I think the Y axis lines most likely represent a 10x jump each, and so the point at the end of the graph represents about a 30000x CM (i.e. Anthropic can currently pretrain models at the same quality with 1/30000th of the compute it needed in 2024). I'm pretty unsure about this though and it could plausibly be much higher or much lower. Keep in mind that this graph is only about pretraining and doesn't say anything about post-training.[1]

Without extra context, it's hard to say how much the compute has actually been multiplied: if each Y gridline is 1.1x the total multiplier would be about 1.53x; if each Y gridline is 10x the total multiplier would be about 30000x. (It's not clear to me why the last point is an arrowhead instead of a dot - maybe it's supposed to be their guess of what the compute multiplier will end up at for their latest pretraining run?)

What people in the IPO roadshow video say about compute multipliers

Sam McCandlish said (while that graph was displayed):

It’s how we serve models to customers and how we use Claude at scale internally. By training what we believe to be the most efficient models, we can do more of all these things with the same amount of compute. It’s also how we drive down costs, both for customers as well as our own underlying costs. We’re accelerating these efficiency gains through our research process. Our pace of discovery is speeding up, and a lot of that acceleration comes from Claude itself.

Tom Brown said:

The second driver of our compute advantage is our rate of discovery of compute multipliers. We focus on total effective compute, which is the multiplicative product of physical capacity times the compute multipliers that we discover.

Each one we discover makes our pool of total effective compute bigger. For example, say you need 100 units of compute to train a model of a certain intelligence. A 4X compute multiplier means you can use 25 units of compute to train the exact same model, or you can use those 100 physical units to train an even more intelligent model that would’ve previously required 400 units.

Compute multipliers create a competitive advantage for us that’s hard to replicate: they build on each other. Each new multiplier we discover has a positive effect on the multipliers before it. They stack.

We believe that this can accelerate even further as we approach recursive self-improvement. Claude will be able to discover compute multipliers entirely by itself, allowing the number of compute multipliers to grow faster and faster. We believe this combination of smart investments, breakthrough research, and the improving capabilities of Claude have put Anthropic in a leading position among the frontier labs.

Krishna Rao (CFO) said:

As Tom described, there are two drivers of this efficiency.

[...] The second is compounding compute multipliers, which have scaled our total effective compute significantly faster than our physical capacity. You can see this in our financial results. Over the last six quarters, we grew revenue by more than 45X, while compute and infrastructure costs grew only 10X. A more than 4X improvement in revenue per dollar of compute.

What is a CM?

From a deposition, a pretraining researcher explains what CMs are a bit better than the roadshow video:

Q. What do you mean by "makes the model dumber"?

A. Um, so the way that we normally define this is we -- because there's a notion of the compute efficient frontier. We're given a fixed budget of compute. There's an optimal way to train your model and an optimal loss you'll achieve on your objective function.

Um, so we can use this as a measuring stick comparing various models. So, for example, um, if you made the model much dumber, it might shift it down so as if you only had half as much compute to train that model.

Working backwards to estimate the actual compute multipliers

If we can estimate the CM from Claude 3.5 to Claude 4, we can work out where the rest of the points on the graph should be based on that. Here I try to estimate that number very roughly.

Model codename reference

We know the codenames from this filing, and some of the compute numbers from this graph.

Codename

Model

Training compute

junior

Unreleased model from mid-2022

4.2e24 FLOPs

finch

Stopped pretraining run from late 2022[2]

lark

Claude 1 (released March 2023)

1.6e25 FLOPs

eagle

Claude Instant (released March 2023)

eagle2

Claude Instant 1.1

2.0e24 FLOPs

tern

Claude 2 (released August 2023)

teal

Claude 3 Opus (released March 2024)

dove

Claude 3.5 Sonnet (released June 2024)

I believe lark is approximately finch but without training data from LibGen or PiLiMi, which hurt performance. I believe tern (Claude 2) was likely the same architecture as lark, and possibly continued pretraining on lark. Teal was almost certainly a fresh pretrain and a new architecture.[3]

Estimating Claude 1.0 -> Opus 3 CM

Claude 1 was pretrained with 1.6e25 FLOPs. We don't know how many FLOPs Claude 3 had, but it probably had around the same number of parameters as Claude 1 ("Lark [Claude 1] is actually too big, so Teal [Claude 3 Opus] will likely not be that much larger") but with much more data and a better architecture.

This graph[4] from an internal presentation shows that the actual amount of compute used to train Claude 1.0 was about 1.6e25 FLOPs. Extrapolating that graph to when Opus 3 pretraining finished would give 7.4e25 FLOPs. For various kinda complicated reasons[5] I think this is too high and it's actually more like ~4e25 FLOPs. So Opus 3 had ~2.5x the pretraining compute as Claude 1.0.

To calculate the 1.0 -> Opus 3 CM correctly we'd have to know how good 1.0 would be if we scaled it up to use as many FLOPs as Opus 3. We don't know that, but if you extrapolate based on how Llama3 scaled, you get that it would take 13x to 32x the compute. (Various other measures I tried roughly lined up with that too.) In other words, Opus 3 was as good as scaling up Claude 1.0 13x to 32x, but only took 2.5x the compute.

So 20 / 2.5 = 8x CM in the year between Claude 1.0 and Opus 3. That's an average of 1.005713 CM per day, so if we assume the rate of progress was the same for the 8 months between Claude 3.5 and 4 that's a 4x CM.

Estimating from individual CMs

There are multiple sources of pretraining compute multipliers: better pretraining data, a better training process that requires less tokens for the same capabilities, and better architectures.

For data quality related CMs:

• "MMLU is too noisy to measure anything less than 1.1CM"
• A 1.5x CM is insanely high (and maybe wrong?)
• A 2x CM is so high it's assumed that there must have been accidental training on test data

So a normalish CM is maybe around 1.2? And maybe there'd be a few of them between model releases? Maybe we'll have ~6 ~1.2x CMs or ~4 ~1.3x CMs, for a ~3x CM between Claude 3.5 and Claude 4. I'm not sure if that's reasonable?

Working out the rest of the data points

If we assume a 3x CM for Claude 3.5 to Claude 4, then the grid lines are about 10x apart. 3x seems roughly reasonable given my estimates above, and it would make sense for the gridlines to be 10x apart each. Here's a widget Claude made where you can try different values for that first step, and see what that implies.

If you use try a CM of 5x for the first step it implies a total CM of ~3.3M and 10x would imply ~2.1B. Those would be absurdly and unreasonably large, so the first step CM is probably in the 3-4 region.

What the graph means

Keep in mind that you can improve pretraining performance on code just by changing the data mix (swapping out non-code documents for code documents). That would improve performance on code to the detriment of non-code. Anthropic has always had a lot of focus on making models good for coding, but I think it's fair to assume that they've been tilting the data mix more towards code since Claude 3.7 Sonnet (released February 2025) which was released alongside Claude Code. I doubt this skews the graph too much though.

Conclusion

Let me know if you have any questions! I'd love it if Anthropic was more transparent here; it seems valuable for them to communicate about the rate of progress in AI development.

1. ^

   This makes it less useful for understanding RSI dynamics, since task-based RL is very important for model performance at model R&D.

2. ^

   This transcript strongly implies that the only reason it was stopped was because Anthropic's lawyers advised that it should be stopped (presumably due to copyright concerns).

3. ^

   This seems pretty clear to me, some evidence is that Claude 3 models use a different tokenizer than earlier ones.

4. ^

   I think that graph is just pretrain compute, but it's not entirely clear to me. This doesn't matter too much though, since we only actually need the number for lark which I guess probably had >99% of compute spent on pretraining.

5. ^

   I might explain this in a later post

Discuss

In January 2024, Ryan Greenblatt and Buck Shlegeris posted "Catching AIs red-handed." This post got on "Best of LessWrong" in 2025, and in December of last year Buck Shlegeris commented on it explaining that he was disappointed it did not lead to more follow-up research. I upvoted the post at the time of the review and think it is representative of the kind of concrete analysis the community values.

As it happens, it turned out to be remarkably prescient and near-term-relevant for a theorypost: just this August 2026 we, in fact, got to see what happens when AI companies "catch their AIs red-handed". The parties who wrote the post were, additionally, some of the very people involved in investigating the Hugging Face incident.

I definitely respect these researchers, but I struggle to imagine a means through which their analysis could have been useful for triaging those incidents. While it seems possible that publishing research like this is what put Redwood in a position to be called upon to investigate in the first place, the piece is just a long and abstract discussion about how (for example) if you catch an AI scheming, it is less likely to succeed (emphasis not mine):

In this post, we’ll discuss a crucial dynamic related to risk from scheming models. Suppose your model is trying to exfiltrate its weights or hack the datacenter or cause some other difficult-to-reverse bad outcome. (We’ll refer to all of these as the model “escaping”, because escape is linguistically convenient and is a good description of many of these difficult-to-reverse bad outcomes.) Our claim: once you’ve caught your models red-handed trying to escape, the situation changes substantially, in ways that are bad for the schemer’s chances of success.

And how, if you catch one of your AIs trying to escape, you should look for other AIs trying to escape:

Proliferation is where you try to capitalize on one example of AIs attempting to escape to find a diverse set of examples of AIs attempting to escape.

In the most simple case, this looks like trying to find nearby inputs that still result in an AI attempting an escape.

But more complex proliferation might be possible, because a serious escape attempt probably involves a substantial (>5) number of AI actions; in particular, actions from multiple different instances of AIs with access to different information and affordances. We try to make this be the case by implementing security measures that make bad actions impossible without many AIs collaborating.

People doing "alignment research" make a lot of these types of posts, essentially designing and juggling abstractions for alignment as a topic. I don't really understand what the theory of action for this work is supposed to be. I don't think it's harmful to do this analysis per se, and I don't think it's being done poorly or in bad faith, but there just doesn't seem to be much point. It reminds me of attending "cybersecurity" courses in university and listening to lectures about the CIA triad.

There are essentially two possible outcomes when you're speculating about "what to do" in response to future misalignment incidents:

• Option A is that the post is just wrong or confused, either because its reasoning is bad or because it uses a completely wrong frame. Mostly alignment researchers get it wrong, because it's hard to do this kind of analysis well years in advance; any group of people would get it wrong.
• Option B is that your analysis is right, but useless. Because by the time you get the chance to act on it, you're going to have much more direct evidence about what's going on, which obviates the need for the analysis in the first place. Your suggestions will sound stupid and circular no matter how prescient they are at the time, and the primary concerns of the people dealing with the problem are going to be whatever practical details are involved with doing the digital forensics.

This post, one of the lucky ones, seems to lie in the latter, fortuitous case. Did suggesting that we ought to "question the model and run experiments on it in the context where it misbehaved" inform any of OpenAI's investigations? How about "rerunning improved monitoring over past logs"? I hope someone reminded the safety team to test whether newly trained models do the same thing in a similar circumstance. They might not have thought of that!

The nearest actually useful analogue to this kind of analysis I can think of is wargames. With wargames, the reason they're useful, when they're useful, is that they inform present decisions on military policy. As an example I like, "AI 2040" is a wargame, yes, but it's a wargame for the purpose of generating a series of policy proposals, which are helpfully advertised on the website. It's hard for posts like the above to reach that bar when the content is so abstract, when it seems to be focused on advising people in the future rather than people in the present, and when (as far as I am aware) people who could be in a position to make policy decisions at the labs don't read this type of stuff anyways.

Discuss

We’re going to see huge, foundational cybersecurity attacks given the rapid rise in AI capabilities. Having worked in hostile environments, I’ve thought a lot about physical and digital security over the years. While you may not be personally targeted, or doing dangerous work, your data is undoubtedly leaked somewhere, and the digital providers you use—for email, banking, social media, etc.—are more and more likely to get hacked.

Discuss

There was an AI protest day today, with parallel protests around the world, and our family went to the Boston one:

I'm not really a protest person: normally I share my views in writing. But while there's been a lot of writing on the internet about the risk that AI gets us all killed (some of it from me), this can't be an internet-only thing. Way too few people are taking seriously the risk that OpenAI, Anthropic, and others are rapidly building something that could actually cause human extinction. A protest is a literal demonstration that people care, and it expands visibility to the general public.

We talked with the kids about whether they wanted to come, or whether I should take them somewhere while Julia went to the protest. All three did want to come, which makes sense: they've been hearing us talk about this at the dinner table and with friends for months, and the risk impacts them at least as much as it does us.

We painted some signs:

Clementine was not helpful:

Mine was a bit on the long side:

The crowd was mostly students, but there were also a lot of people I recognized from Boston EA meetups. Favorite sign: an "If You Can't Steer Don't Race", painted by someone I'd run into at LessOnline.

It was nominally a protest demanding a US–China treaty that bans more powerful AI until it can be done safely, but in practice it was a much more high level protest around the "slow down AI", "regulate AI", "AI bad", "RSI will get us killed", "AI companies irresponsible" cluster. If you hadn't seen the website, I don't think you'd have known about the treaty; no one mentioned it.

We alternated between a series of chants (quickly taught over the megaphone) and short speeches. The speeches included points I strongly supported (this is urgent, risks include extinction, safety regulation is a normal part of industry) and others I didn't (datacenter water use, it's wrong to make a technology that takes jobs, general scope insensitivity) but building a coalition of people who want to push the AI lever towards "slower" is fundamentally how a political movement has to work. I did stay quiet during a few chants, though, and if there's a way to avoid "prudence in the face of superintelligence" becoming partisan that seems very important.

Overall it was uncomfortable, and not really my thing, but I think it's important, liked being in a group of people pushing policy in a good direction, and would go again.

(More in this "online LW-ish person goes to an AI protest" genre: philh, Quadratic Reciprocity.)

Comment via: facebook, mastodon, bluesky

Discuss

We are in AA 9.

In Brave New World, Aldous Huxley dated the future from the launch of a car. The book was set in AF 632; AF stands for "Anno Fordi" or "After Ford". It starts counting from 1908 AD, the year when Ford launched the Model T into production. In the book, people say "Ford!", instead of "God!", and the cross has been cut down into a T. It's meant to be humorous. But the point here is that a society built around mass production might end up dating itself by it.

The case is not so different for us; instead of mass production, we have AI, and we haven't picked our 1908 yet.

There are a few obvious candidates, such as Turing's 1950 paper, which asked the gold question: can machines think? The 1956 Dartmouth workshop, which coined the term "Artificial Intelligence". We then have ChatGPT in November 2022, which accelerated a lot, and ChatGPT is the Model T of the AI era, in my opinion.

But AI is different in the sense that it is a collaborative result of the efforts of a lot of people, research institutions, and the compute industry, among others. It might not be wise for us to go by the Model T analogy and choose a person or a product. And we have a better alternative.

My pick is June 2017, when a team of eight researchers, mostly from Google, published the paper, "Attention Is All You Need", and it had huge downstream effects. And dating our era from a paper instead of a person or a product kind of feels right. It was an idea, written down by a team, published openly, and then scaled by anyone with a data center, that actually changed the world.

One objection to this is that the concept of "Attention" existed before 2017, and that would be correct. Some papers used the concept before (ex: Bahdanau et al.), but the 2017 paper made attention the whole architecture in a way that turned out to be scalable.

There are three reasons why I like "Attention" as an anchor:

1. It has a pun: Attention is the mechanism inside the model, and it's also what the economy around it harvests.
2. It is a good slogan, and "for Attention's sake" works as a mild curse (similar to "for Ford's sake" in the book).
3. It was a break. After the Transformer, scaling became the strategy, and it pulled a lot of people in.

By this count, it's After Attention 9 or AA 9.

Now, I could easily be wrong about the anchor or other things here. Yes, the anchor might be arbitrary. That's why I'd like it to stick even as a running joke, and mostly a conversation, since a calendar is more interesting if we land on it together.

So AA 9. The year of our Attention.

Credit to Dhruv Trehan for the idea of the AI-age calendar, which came up in one of our off-manifold discussions between our research at Lossfunk on November 5, 2026.

Discuss

Following my 3-week visit to Berkeley as described here and featured by ACX here, I told various people I would write in detail why I didn’t stick around to work in AI safety. In this post, I will dance around one object-level disagreement with AI safety and rationalists: why I no longer believe in explosive world transformation due to ASI. There will eventually be follow-up pieces on more object-level arguments and meta-level arguments against AI safety, but I prefer for this post to be mostly self-contained and not explicitly about AI safety.

When I worked in theoretical physics research, I was constantly reminded that if I were just that much smarter and had read that many more papers, I would be able to make the right connection to solve the problem. You are always the bottleneck; it’s always a skill issue. So if you’ve spent the bulk of your conscious life honing your ability to manipulate the pure abstraction of math and computer code, it is staggering to watch the problems you wrestled with for months or years get blown away in hours. You are left wondering how quickly all of the other problems in the world will be resolved - all of disease perhaps? Why not all of biology? Why not all the secrets of the universe?

In December 2025 when Claude Code + Opus 4.5 first made agents feel extremely useful, I was concerned about the job market being obliterated, so I took out “insurance” on rapid-onset white-collar unemployment. Thankfully, those predictions about increasingly capable agents causing rapid economic and labor transformation have been very wrong.

And so the question is, why? Why does the world largely look the same outside of highly verifiable domains like software engineering and math, and even in software engineering, why are there still millions employed? Why are models which can beat humans on nearly all benchmarks and are provided at unbelievably low prices not causing large economic shockwaves?

If I had to summarize my current perspective on AI in a single sentence, it would be that “the world is far too slow” to see the high double-digit or more economic growth and seismic transformations in the economy as predicted by various essays from the rationalist blogo-instituto-sphere.

Amdahl’s Law

I suspect the reason I’ve felt less “capabilities whiplash” than my peers is that a lot of my experience is in experimental science where how smart you are is often not the actual bottleneck. To illustrate how enormous speedup in one component does not translate to enormous broader speedup due to other bottlenecks, let’s make a simple model:

If you have a process with some number of steps to be completed sequentially, each with rate mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } i (ie each step takes time 1/to complete), your total rate is given by

If you send every rate but one to infinity, you are still left with

This is almost a trivial statement - your overall process is rate-limited by the slowest step, but it can be easily forgotten as you watch individual rates arc hyperexponentially upwards:

From the new METR report

More realistically, suppose you have some hierarchy of rates

where r₀ = a is the slowest and each successive rate is b times faster than the previous. The total rate of this hierarchy is

If you send the first m bottleneck rates to infinity, entirely eliminating each bottleneck, the new rate is

The point is that in hierarchically bottlenecked systems, even infinite speedup of the bottleneck results in a constant factor aggregate speedup. In order to see exponential aggregate speedup, you need explosive improvement at the current bottleneck over and over again. Overall progress is often brutally suppressed by this interplay between bottlenecks.

If during each time window you focus all of your efforts on accelerating the current bottleneck to ~infinity (or really just far beyond the next bottleneck or two), you see constant factor speedup. Repeated infinite individual factor speedup yields only exponential aggregate speedup.

To see diminishing returns, the intelligence curve need not flatten out; you just have to hit some other bottleneck. And so the question as it pertains to AI causing disruption in the world (for better and worse) is not “where is intelligence the rate-limiting step?” but “when intelligence becomes free, instant, and near-omniscient, what is the new rate-limiting step?” In computer science, this speedup analysis is known as Amdahl’s law and is used for understanding the limits of parallelism, but variants of it appear all across science.

Is Intelligence the Bottleneck?

I suspect the most common objection to the above analysis is that I am failing to imagine what ASI can accomplish. Instead of trying to abstractly rebut this, let’s look at a banal example of fairly bounded speedups even when you have access to ASI:

You’re the manager of a widget factory, and for some reason the defect rate has doubled in the past month. You’re able to use your process telemetry and slew of agents to figure out that tolerances during step 7 have blown out of the acceptable range. You haven’t changed anything about your process, so you have no idea what could be causing this. Four different precursors enter the process during step 7, but those haven’t changed either.

With the help of your ASI helper, you’re able to narrow down the possible causes of this specific kind of defect to inclusion of impurity X in material Y, but both precursors A and B contain Y. You reach out to the suppliers, but they shrug and say “sorry we haven’t changed anything”, so you need to diagnose which of these is the ultimate source. You order some tests on your new stock and the remnants of the old stock, and you find out that precursor A is causing your defect.

And now you have a decision to make: do you try to find a new supplier, remove the impurity yourself, or compensate for the impurity in other ways? You’ll need to run the tests on the new material or make a purification system or change your design. These all have tradeoffs and ASI will be able to help with all of these and pick the right one, but the fundamental problem here is that it takes time and money to install new systems to fix this problem. There is the time it takes to order new components, assemble them, install them. If you are integrating it into your telemetry system, you need all the sensors and wiring to feed into some database, and then finally you need to get your production up and running and make sure it’s all working.

The crux of AI disruption in my mind is these new bottlenecks which can be roughly split into two categories:

1. What are the speed limits in the world that intelligence does not accelerate?

The purely physical speed limits in the above example are things like sending out complicated tests to be run, ordering new components, installing new hardware into your factory, and manufacturing new widgets to verify your problem has been solved. There is a practical limit to how fast you can ship things across the country, and manufacturing steps involving chemical and biological processes have fundamental rates that can be on the order of days for things like culturing human cells and annealing metals.

Now suppose you also have humanoid robots to run your factory and install the new components, handling every step along the way. A very nontrivial percentage of the time in the above example is just dealing with the other companies, sending out materials for testing, having the new parts shipped to you, and installing the physical hardware and stress-testing it. This whole process could take a month in a current factory without superintelligence, but getting a 4x speedup to ~1 week seems about as fast as you can possibly go even with omniscient machines making perfect decisions at every step and working 24/7. At some point you’re quite literally bottlenecked on shipping time and physical throughput rate of tools in your factory.

There are also problems which are more or less solved - the steam engine has been around for 300 years, yet even fusion plants will likely end up turning a steam turbine. How many other problems in the world are in some sense solved and are very slow, expensive, and boring to improve? Other speed limits in the world are less physical but still important. What is the speed of regulation? What is the speed of political will?

It is worth disambiguating speedup and scale-up here. In something like the oil industry, a multitrillion dollar industry with enormous incentive to rapidly automate, it seems even a 3x speedup would be difficult. The oil comes out of the ground at some rate, is refined at some rate, and is transported at some rate, so the obvious answer if you want to double production is to parallelize everything and just have twice as much infrastructure. But massively expanding capacity in parallelization is itself a process that can be bottlenecked. Over the last year, strange scaling bottlenecks have looked like the monocrystalline turbine blades for natural gas plants and indium phosphide wafers for optical interconnects, and this has just been for running computers. If you believe in true industrial explosions, the demand for niche magnets and bearings and sensors and tool tips and alloys will explode, and any one of them can become the rate limiter.

2. What does it actually mean to be superintelligent in the presence of noise?

The more abstract speedup question is something like “could a superintelligence, given all the context of the situation, know right away that impurity X was the culprit and that it came from precursor A?” This would save you the time you spent running tests on A versus B and let you go straight to problem solving, but is this even possible? After all, the only thing that changed was the error bar in step 7.

In the aphorism “intelligence is compression”, there is a natural next question: What is the maximum amount of compression you can achieve in the real world? In order to realize that precursor A and not B had the impurity, you need to acquire 1 bit of information. Some bits, however, are unbelievably expensive to obtain and create a similar kind of Amdahl calculus. If key bits are orders of magnitude more expensive and slower to acquire (say, for example, long-term side effect analysis of new drugs on actual humans), these will set your overall rate.

It seems most people in AI consider the world to be far less noisy than it actually is because they mostly live in a world of flawlessly repeatable execution at staggering scale. Scaling experiments is as simple as spinning up more Docker containers, but the notion of superintelligence in domains like biology is ill-defined: There are large variations between nominally identical experiments, and the rate at which experiments produce informative bits is unfathomably low compared to anything in ML or verifiable like math. In other words, even information theory-optimal exploration of some problem landscape could be agonizingly slow if experiments are slow or noisy, let alone both. (To those who believe AI would be able to simulate any process accurately enough, it’s worth appreciating how poorly understood many multi-scale interactions are and how ground truth experimentation is often your best option. See also my blog post from April.)

Outlook

Intelligence is far from the only technology curve to go quasi-infinite in the last century and a half: The world electrified. The cost of communication has gone from couriers and weeks to ~free and gigabytes per second. Computers have made calculations which were once infeasible executable on a cheap computer near-instantly. Transportation costs have fallen by an order of magnitude. From 1880 to 2026, the share of the US workforce in agriculture fell from 51% to 1.2% as we were freed to work in other fields.

Yet through it all, we’ve seen nearly 150 years of 2% GDP per capita growth. These are all general-purpose technologies, so what needs to give for ASI to be that different?

Discuss