This post isn't affiliated with AIFP and I think they disagree with some of it.
Plan C is for the leading company to burn its lead. Plan C+ is similar, but all the companies in the leading country work together (maybe because the government is forcing them). Plan B is similar, but the leading country is also sabotaging trailing countries. In Plan D, there is no leading company, and companies can only do alignment work sporadically when they get the chance.
If alignment-pilled companies aren't in the lead, then we end up with the worst version of Plan D, where we have to hope that incidental alignment work done along the way by commercially-focused companies proves to be enough
The problem with China getting superintelligence first isn't just that they might misuse it. It's that, on Plan C, US companies are working hard to align AI; if Chinese companies aren't doing this, then the chance that the first superintelligence is aligned goes down.
I don't want to get in a fight over whether more good or bad alignment surprises have happened recently, but I also updated on https://www.astralcodexten.com/p/mysteries-of-ai-generalization and https://x.com/_jake_ward/status/2112264244767211854 - not as solutions in and of themselves, but as evidence that good things can happen that I wasn't previously considering. Nick Cammarata's Neural Networks Are Already Clean would also be in this category if I believed it, but I'm currently confused by the debate around it and unsure.
Many people at eg MIRI have thought about this in much greater depth than I have and concluded, on the basis of a gears-level model, that near-term alignment is very unlikely to work. I'm still trying to understand their arguments, but I don't blame them for having them. But I do worry that other people have picked up SURA ideas from the MIRI-sphere without realizing that they make less sense without MIRI's particular brand of strong alignment skepticism.
To be clear, this is just the people who have formally considered this question qua forecasters and written down exact numbers. I'm not denying that there are many other smart people, for example at MIRI, who have thought about this hard and believe the opposite.
Again, despite leaning heavily on AIFP terminology and sources in this essay, I'm not claiming that AIFP or its members agree with my thesis in this post, and Daniel Kokotajlo in particular has said pretty clearly that he hopes people unilaterally quit AI companies.
I discuss this further at What Deontological Bars?
This is a higher probability than it gives for global pauses in other situations, I think because it unintentionally magnifies small differences in how I frame the question into different definitions of "global pause". I haven't put in enough work to investigate and have a strong opinion on what's going on; don't take these numbers too seriously except as a very rough order of magnitude guess.
I'm working on a grantmaking AI that can actually compare the values of different outcomes. It's still in alpha, and to some degree this is a circular argument because I'm trying to get it to mimic/predict my own opinions, but very loosely, it thinks that increasing the alignment-pilled-ness of the leading company from OpenAI-level to Anthropic-level would be worth about 250x as much as the best (low-enough-secrecy-to-mention-here) actual grant proposal it's ever evaluated, which was a proposed $2 million to the AI Futures Project. I think this confirms my "better use of our time than any actual plan we can carry out in those six months" claim.
When I'm modeling this, including on the AI grantmaker, I'm imagining a scale from 1 - 100, where NVIDIA is 10, Meta or X is about 25, OpenAI is about 50, GDM is about 60, and Anthropic is about 80. I imagine a successful campaign to get alignment-pilled people to quit taking Anthropic out entirely, and reducing GDM/OAI to about 40 as they lose their alignment-pilled employees and executives.
I remember how much of a slog the discourse was back in 2015, when the standard counterargument was "you're just a bunch of cultists, no real AI experts believe in this doom stuff". The new standard counterargument of "Of course all the AI experts believe in this, they're just hyping their business for the IPO" is actually a vast improvement!
I asked a supporter how they saw this working - they said that if the US paused, and China didn't seem on track to follow suit, maybe the US would start up again after a few months. But A) this would burn somewhere between most and all of the lead, B) this was just their speculation, and there's nothing in the bill which suggests that the US would do this.
Again, all of these pause numbers are slightly different because the question is slightly different - this one is about the next year only. I'm still trying to assess the utility of AI superforecasters, and don't want to present this as a knockdown argument. But I want to get people thinking in probabilities and present some evidence that I'm not letting my own biases control me, and head off accusations that I'm just trying to spread fear and pessimism. I encourage other people to try different framings of these questions with different AI and human forecasters and see what they say.
Taking this analogy overly seriously: the biggest difference between Frodo and eg Denethor is that Frodo wanted to destroy the Ring in the end. I'm not sure what the AI equivalent would be - all companies claim they want to use AI to empower ordinary people and achieve broad-based prosperity. Taking the analogy very seriously, Frodo failed to destroy the Ring at the final moment, and was saved only by divine intervention. I don't know what is the AI equivalent of Gollum.
Specifically, there are many people who said they wouldn't try to make money or release products who changed their tune when they realized that AI would require hundreds of billions of dollars of compute, and that the only way to get that money was to get hyperscaler and investor support.
One reason all of this confuses me is that rationalists had an excellent model of race dynamics with China as early as the mid-2010s, when we understood they would be an inevitable part of any future AI development. I distinctly remember going into some old rationalist coworking space sometime around 2020 and seeing "RACE WITH CHINA" underlined on a whiteboard. These discussions included plenty of talk about whether we would be able to coordinate a global pause, but I don't remember anyone ever seriously saying that the problem could be solved if the good alignment-pilled companies or countries dropped out unilaterally.
One of the many ways I have trouble modeling why Samuel would write this article is that it seems like she trusts everyone to drop out of their own accord, except for Anthropic. China? Sure. OpenAI, sure. Meta, definitely. Every single group sponsoring open-weights models in the world at the same time? Yeah, I guess, surely they wouldn't want harms. But Anthropic? We must all rise up to make them do this thing which they would otherwise continue.
eg: power concentration
This is a higher probability than it gives for global pauses in other situations, I think because it unintentionally magnifies small differences in how I frame the question into different definitions of "global pause". I haven't put in enough work to investigate and have a strong opinion on what's going on; don't take these numbers too seriously except as a very rough order of magnitude guess.
No, I am not saying that capabilities researchers are Hitler, or that capabilities work is as deontologically bad as genocide, or necessarily quibbling with anything else in this post.
This is a higher probability than it gives for global pauses in other situations, I think because it unintentionally magnifies small differences in how I frame the question into different definitions of "global pause". I haven't put in enough work to investigate and have a strong opinion on what's going on; don't take these numbers too seriously except as a very rough order of magnitude guess.
Again, all of these pause numbers are slightly different because the question is slightly different - this one is about the next year only. I'm still trying to assess the utility of AI superforecasters, and don't want to present this as a knockdown argument. But I want to get people thinking in probabilities and present some evidence that I'm not letting my own biases control me, and head off accusations that I'm just trying to spread fear and pessimism. I encourage other people to try different framings of these questions with different AI and human forecasters and see what they say.
Again, all of these pause numbers are slightly different because the question is slightly different - this one is about the next year only. I'm still trying to assess the utility of AI superforecasters, and don't want to present this as a knockdown argument. But I want to get people thinking in probabilities and present some evidence that I'm not letting my own biases control me, and head off accusations that I'm just trying to spread fear and pessimism. I encourage other people to try different framings of these questions with different AI and human forecasters and see what they say.
I basically disagree with this post.
But do you still stand by the numbers in the post quoted under your name that we have 20% chance of making it in Plan C, and 40% in C+, and only 10% in Plan D? Can you say how big you think the difference is between Anthropic vs OpenAI being in the lead, on the spectrum from C to D?
I always found it surprising that your probabilities seem to indicate that you think that putting in a few months worth of safety effort at the right time is hugely valuable, but you are still opposed to arguments on how it's very important for supposedly more responsible companies to be ahead.
I always found it surprising that your probabilities seem to indicate that you think that putting in a few months worth of safety effort at the right time is hugely valuable, but you are still opposed to arguments on how it's very important for supposedly more responsible companies to be ahead.
I don’t think it’s useful to reason about this unless you’re also decomposing the problem into:
Assuming they do nominally say they’re “burning the lead”, what’s the liklihood this is done effectively:
Imagine you have a company that internally has many employees which are advocating... (read more)
I think you can understand his position better if you see the full graph those numbers are sourced from:
Daniel is the blue line. In his view, Plan A is at least twice as likely to avoid misaligned takeover as any other plan listed.
Ask yourself questions like the following:
I think you should add to this list: how would you would feel about putting your name to a half baked argument that's broadly AI pessimistic vs one that's broadly AI optimistic that ended up being widely read? Or perhaps it would be better to be more specific: broadly pro-recusing yourself from the AI industry vs anti, pro-pause regulation vs anti, pro-anthropic vs anti, alignment is probably solvable via a reasonable ordinary R&D effort vs no.
I think one of the main ways these pressures act is via what they make easy to say and what they make hard to say.
I am trying to understand the cruxes between you, Scott and Anthropic's lobbyists. My vision of the AI-2027 TTX The game master keeps track of the AIs' goals, of the AIs' capabilities, various variables unrelated to the AIs like politics and economics while the players, including the AIs, try to steer the transition towards achieving their goals. Like the TTX, real-world humans are bottlenecked on information available to them. The main difference between the real world and the TTX is extra constraints on some players' actions:
[For the purposes of this comment, I will suspend a bunch of major disagreements I have about your background assumptions or explicitly argued things here, e.g. about feasibility of alignment, relevance of empirical alignment, ethicality of doing any AI R&D, etc., and instead try to make a somewhat internal critique.]
It seems like this post argues about a dichotomy "either SURA, or race & align". E.g. your counterfactual is "Anthropic disbanded their entire company immediately with the explicit goal of raising awareness of a pause". But this dichotomy doesn't make sense to me even under your assumptions, as an answer to "What should Anthropic do?". For example, here's a different thing that Anthropic could do:
I mostly agree with you - I don't think my claim hinges on whether current companies are perfect or even necessarily good, it's that part of our goal should push them towards being better (while still "in the race"). Just as we acknowledge that the US government is currently pursuing disastrous policies, but we can think of it as a potential future ally we can win over through activism, so I think we should be doing the same with AI companies (and not just Anthropic - I think that we should be ambitious and try to redeem as many as possible).
I'm not sure I understand your first suggestion - stay on the alignment frontier but don't release. Isn't this equivalent to having powerful internal models but not releasing them (like both OpenAI and Anthropic are doing now to some degree). I worry this could be the worst of both worlds - when the internal models get superintelligent, they can still figure out a way out of the "box", but the world is unaware of them and won't "wake up" in a way that boosts political activism. I don't think having the models be released rather than unreleased boosts their danger much.
I'm unclear on what Anthropic's opinion on / level of work towards a pause is... (read more)
I don't think having the models be released rather than unreleased boosts their danger much.
It doesn't much increase the danger from deployment, but it pushes the race forward. It's like the difference between the policies "You have 100 nukes? Ok, now I have 101 nukes." vs. "You have 100 nukes? Ok, now I have 99 nukes.".
they understand the importance of a pause and think they're doing good behind-the-scenes work, which they won't tell us about
If indeed they are doing the full-hearted version of this, that would change my mind. I simply don't believe it, and the public evidence seems mostly (but not entirely) against this.
I think regardless of whether we trust them, our role is the same - a combination of pressuring every company (including Anthropic) to do this, and building things on the government/mass activism side to get recalcitrant companies in line [...].
I agree we should do this, but not that our role is the same. If core-Anthropic is basically not good--they basically will continue actively trying to do more AI R&D, even as a pause becomes significantly more plausible--then a new-Anthropic person putting efforts toward building them up now is creating the po... (read more)
I'm unclear on what Anthropic's opinion on / level of work towards a pause is. The Pacing The Frontier letter seemed like a statement that they were aiming for it, and the fact that all other major companies immediately voiced agreement suggested that they had held off on releasing it until they'd done a lot of politics behind the scenes and gotten everyone else to agree. My impression of Anthropic, from their public information and the exact phrasing of the cryptic refusals to explain which I get from people who know more, is that they understand the importance of a pause and think they're doing good behind-the-scenes work, which they won't tell us about.
This strikes me an extremely charitable good-faith view of Anthropic, based on their actual record of proposed and supported legislation, why would this be your assumption? Why are you crediting Anthropic with organizing Pacing The Frontier?
Additionally, Pacing the Frontier itself (at least at the corporate/executive level - it seems to have originated among lab employees who genuinely were getting freaked out) can/should be viewed cynically to an extent. The executives from a number of AI companies endorsed it and the companies released various statements, but have they really done anything on that front that they wouldn't have done otherwise? Or was it just a way to take the heat off their backs?
It seems like most of your recommendations are actually very similar to Scott's recommendations.
The remaining point of disagreement seems to be that you don't like the fact that Scott describes Anthropic as the good guys and you point out many ways in which they could be much better.
But I'm pretty sure that Scott is using the phrase comparatively.
He's saying that Anthropic are better than the other AI companies. Faced with a bad bunch, they are the good guys. I don't even think you disagree with this.
I don't think people should interpret your comment as rebutting the core argument that Scott is making in any way.
Stay at the frontier in order to do frontier-level alignment research, but don't release leading models.
Why not release models conditional on staying at the frontier, since most of the risk seems like it exists even with an internal deployment and no release? Because releasing models causes diffusion of ideas to other labs (e.g., o1 or DeepSeek R1) or allows other companies to use their models to do capabilities research? I feel like releasing frontier models has some other benefits (e.g., the ability to uncover and make public evidence of misalignment, the ability for external parties to assess models, the ability for external alignment researchers to use models for uplift on their alignment research), and it’s unclear to me how these net out.
I mean, I think we should just shut it all down. I'm trying to make an internal critique of Alexander's position. I don't think he views it as much added risk to make a small frontier step, though I'm not at all sure about that. But releasing a leading model does heat up the race and put more pressure on other labs to keep up, which does speed up capabilities.
(I understand that you think we should shut it all down and are trying to make an internal critique. This seems reasonable, and both my original comment and this one are continuing to stick with the internal critique specifically!)
I thought by “stay at the frontier but don’t release leading models,” you meant “continue to race, but don’t deploy externally” (because I think if you’re racing, it’s maybe worse to deploy internally but not release models), whereas it seems like you maybe actually meant “don’t try to win the race, but continue to build interna... (read more)
it seems like you maybe actually meant “don’t try to win the race, but continue to build internal models as good as the best leading models” or something
Oh yeah, what I said is ambiguous, sorry. Yeah I meant keep up with the frontier but not racing ahead. I mean, you could kinda keep pace with your expectation of where the other leaders will be.
I agree it heats up the race a bit, but my best guess is that the private frontier being ahead of the public frontier increases takeover risk on net.
How so?
If you're building a monster, let people see it.
Not releasing internal models would mean failing to expose the public to reality and forcing them to think more and more seriously about the consequences of AI progress.
This is not to deny the downside of heating up the race and allowing distillation and spread of ideas. But distillation itself doesn't really shrink the lead, it just seems to.
Seeing things and their consequences with your own eyes prevents a lot of cope.
What we desperately need is for the world, specifically the US and Chinese governments but... (read more)
You can release reports, but seeing them impact the world I think has a much broader intuitive salience.
Look how much coverage the Hugging Face incident got in popular press vs. every report of misaligned behavior. I think we're very lucky that impacted the world despite being an internal model.
I think you're misunderstanding the point here. The question Tejas was commenting about seems to be "is it better for a lab to have the frontier private or public". In this case, the leading lab is "matching the other leader's models" regardless and the other labs would probably know that the best model is being withheld, therefore it is contributing to the race dynamics strongly regardless. Therefore, the "expose to the public" dimension and other related dimensions[1] end up being the most important considerations in my opinion between withholding vs releasing one's best models.
eg: power concentration
In this case, the leading lab is "matching the other leader's models" regardless and the other labs would probably know that the best model is being withheld, therefore it is contributing to the race dynamics strongly regardless.
I guess that would be a crux for my point, yes. But why do you think this? TBC, what I'm describing would be like, Anthropic states clearly & publicly "We think this is way too risky, but we are keeping pace in order to do alignment; but we won't put market pressure on other companies to develop and release models; we will do this by not releasing models that are stronger than models released by other groups.". In this world, the pressure seems quite substantially reduced. Why do you disagree?
I think I see your point better now. I think you're right if the other actors actually trust Anthropic to not do an intelligence explosion internally, even if it is within their grasp. But I think this kind of trust is currently not close to being there given the mutual dislike/fear between different labs and the fact that Anthropic is already racing hard. But if you stop internal deployment without having a credible commitment that you won't do an intelligence explosion, then other companies will treat you as though you're racing regardless so this seems anti-helpful to me.
To be concrete, Anthropic and OpenAI both currently seem to be not releasing their best models yet (Fable 5.5 and Astra 6.1) but it doesn't seem to be helping. I don't think this is actually slowing down race dynamics at all: they each know this about one another. Plausibly, if this continues it is actually speeding up race dynamics since it induces paranoia (where even the lab thats ahead can feel fear that it may be behind) and has other bad effects like not informing the world.
(To be clear, I want Anthropic should be doing many things differently. But I don't think its helpful to put pressure on them to not deploy externally, at least unless they can credibly commit to not doing an intelligence explosion).
Not sure I'm following. It doesn't seem that related to trust? If one company deploys a leading model to consumers, that creates a strong direct market pressure to deploy a leading or near-leading model to consumers. To deploy such a model, you have to have developed it. Like, you're describing a different pressure ("Maybe they'll do an intelligence explosion") but it's not the pressure I'm talking about, and I don't buy it's a very strong pressure compared to market pressure. Like, do people feel pressured by SSI?
Like, you're describing a different pressure ("Maybe they'll do an intelligence explosion") but it's not the pressure I'm talking about, and I don't buy it's a very strong pressure compared to market pressure.
I disagree here and this may be the crux. I think the pressure of being first to an intelligence explosion is the strongest pressure causing these labs to race. Some reasons for why:
I think your criticism of Anthropic the organization is valid, although there are a fair few specifics I'd quibble with (e.g. I would prefer perfect-Anthropic release frontier models rather than hold back from doing that and disempower themselves).
So where does this leave us? Anthropic is worse than perfect-Anthropic but still seems better than no-Anthropic - voicing concerns about X-risk in interviews and blog posts is better than never voicing those concerns, half-heartedly supporting regulation is better than opposing it, etc.
I often feel like the SURA... (read more)
This is an example of what I'm talking about when I say you take a certain level of risk-awareness for granted that I just don't think actually exists. There is no need for safetywashing, you can literally just say "we're going to make superintelligent AI through recursive self-improvement, and there is no way this could possibly go wrong :)" and you will get less flak than a lab that mentions safety at all.
There's an extent to which you're right that this phenomenon is real and unfair to Anthropic. But to some degree, I think you're probably confusing deb... (read more)
Releasing models allows the public to test them out. This is both MUCH better for the public's understanding of model capabilities (I don't trust labs going "our models are scarily capable, trust us") and also better for safety (orders of magnitude more human testing of the models in realistic situations compared to what labs can do internally)
I would go so far as to say there should be a law banning anthropic from internally using a model they do not also externally deploy. The added risk is negligible. The added competitive effect doesn't exist if other labs race even against rumors of internal model capabilities (see openAI spending tens of millions in compute trying to match a rumor that anthropic solved two millennium prize problems). And in the near term we'll have anthropic hoarding all the superhuman scientists and enjoying a monopoly on their output if we don't force them to share the benefit with the rest of the world.
Refrain from developing or from releasing?
I'm in favor of anthropic refraining from developing. But whatever they develop they should release. OpenAI won't stop just because anthropic's position is "we won't release anything until we can perform a pivotal act and kill all competition, but by the way here's a demo of what our models can do." They'd only panic more.
OpenAI itself might wish to stop developing past a guess at anthropic's internal model. But if they never see the Internal model because it's not deployed, OpenAI will guess that this model solved multiple millennium prize problems (we just saw this). Transparency is good. Release the models.
Here's a possible policy. Definitely not my favorite, but meant to be clearly better than what Anthropic is doing, even if you think that "race & align" is a good approach even though a pause would be better.
I agree about the "develop" points here, disagree about the "release" ones. Releasing should be immediate. Release all internal models.
If we think there are two different races (a race for developing internal models and a race for releasing them), I want AI labs to stop racing on development and race even harder on releasing.
If your argument is that Anthropic would sabotage Plan C at the cost of their own lives, then why do you expect anyone to not sabotage a pause?
How would you do a Pause without billions in budget and thousands of employees? Honestly, that seems like a very low estimate and there would obviously still be massive piles of compute sitting around in any reasonable US-China deal.
Sorry, I'm lost. Could you rephrase? Are you asking about what happens to the compute? I don't know, I guess you have some sort of transparency / oversight governance structure for the compute, to make sure it's not running training runs. Are you objecting that this governance structure would sabotage the pause...?
Yes. Imagine an actual pause treaty and structure, you'd have China's Ministry of State Security and the Central Military Commission on one end, the NSA, the White House, and some kind of All-American Compute Consortium on the other. These all seem like people who would try to cheat, especially since very few of them currently believe that building ASI will kill them.
I still don't understand what you're trying to say / ask me. Surely it's way harder for someone to do an Anthropic-scale AI R&D project in this hypothetical world, compared to the hypothetical where we just keep our foot on the gas of AI R&D as we are currently and then Anthropic and others have the opportunity to derail / distort pause policies?
I am contrasting the risk that those others would intentionally undermine the pause with the present risk where it seems much weirder to suggest that Anthropic et al would intentionally sabotage Plan C, but I also don't think any realistic pause treaty would really stop Anthropic all that much given the argument is that a pause would buy time for AI alignment research and Anthropic's employees would be the kind of people doing that research.
I asked Fable and I still have no idea what you're talking about. I think you should state what you think I said, that you are objecting to / critiquing / something. E.g. do you think that my comment about warning-shot masking is me saying "Anthropic will intentionally hide warning-shots."? That wasn't my intention. But I don't know which thing you're responding to. Please use more specific phrases and state what propositions you are responding to.
@Claude Opus 5.5 Can you explain what Josh is saying, claiming, objecting to, or asking me?
I think Josh is making a consistency argument in three steps.
You suggested Anthropic might safetywash, mask warning shots, or lobby against a pause to keep its lead. Josh reads that as "Anthropic would undermine Plan C even though Anthropic's leaders believe failure kills them too." His point: if you expect that much bad faith from people who believe in x-risk, you should expect more of it from whoever runs a pause.
A real pause regime would also involve big budgets, staff, and leftover compute. In practice those would be controlled by... (151 more words)
Yes, that seems like a good explanation of my position.
I think they are currently doing those things. I think they are probably deluded about the extent to which they've gotten alignment into their current AIs / the extent to which that will generalize. I think it's likely they will continue deluding themselves, up to and including killing everyone by setting off an intelligence explosion.
Suppose you tried to reason validly from that premise whose reasoning you have not yet followed, saying that alignment by current institutions seems wack, Amodei has been already-selected to still show no sign of understanding alignment basics from 2010, they have no visible comprehension of the basic reasons why "make the AI do our ASI alignment homework" might be hard, etc. Suppose it is the case that leads cannot be burned on alignment. ASI is death, plain and simple.
What policy then follows according to you?
See Section II, "Actually, most of this stuff is bad for a global pause too", although I admit I would then be much less confident in the sign.
That's you arguing against the US imposing a unilateral halt on further AGI escalation before it gets to RSI and we die before the end of 2027, because you think China would not be moved to follow.
It's not you saying what you'd do instead. What's your optimal policy, supposing you just take as given that these guys can't do ASI alignment?
Despite our better judgement I and Scott had some debate on Twitter instead of LW about this post. I wrote many long comments, but this one most stands on its own and captures a bunch of my reaction to this post (though far from all of it). I think it makes sense without context, but you can read the context here if you want.
We are both leaders in the same ecosystem. An ecosystem that is extremely massively leveraged on Anthropic (and OpenAI) and general plans to unilaterally build AI ourselves, and you are making arguments primarily directed at those people, both here on Twitter on LW. We just distributed money in a grant round whose public funders are something like 90% Anthropic equity in their net-worth.
And you are advocating specifically for more people in that community to join the labs, and accelerate both their competitiveness and their leverage over the world in racing this way.
And I just don't believe this isn't driving your opinions here. I've seen these dynamics for more than a decade. When I talk to people outside of this ecosystem about our plans to build the aligned machine god in 6-12 months they of course freak out (conditional on believing the machine god part)... (read more)
Some other exchanges I had with Scott that seem worth copying over for posterity:
Me:
I think you should obviously be willing to risk war if a foreign lab is working towards building ASI. I don’t think you should unilaterally launch nukes without going through a long process of escalation, and only after conventional methods have been exhausted (roughly the same way you would treat any other extremely large national security threat).
Scott:
I think these considerations apply in a small fraction of worlds where the US government is very alignment-pilled but China isn't.
These aren't the ones I expect or think about most, and I haven't thought too hard about what levels of escalation are more dangerous than just trying to pull off a really good Plan C+, which ought to be possible in these worlds.
I think the considerations in my post apply for all worlds outside this small fraction, and that even in this fraction, while I agree with some threats, I weakly lean against threatening nukes, since under my model a really good Plan C+ has an 90%+ chance of working, and nuclear brinksmanship probably has an associated p(doom) > 10% (even granting that most threats will work without having... (read more)
I usually sympathize more with Scott than Oliver in these types of exchanges, but I think Oliver is clearly more reasonable in this one.
if the US were entirely united around solving the alignment problem and actually smart about it (which I think is a precondition in any world where we're willing to threaten WWIII over AI worries)
seems like a load-bearing part of Scott's argument, but of course a country can be united around preventing AI threat (either due to misalignment worries, or due to worries that no human, especially no other enemy human should get the power) without being "actually smart about it". Being freaked-out and actually smart about the thing seems much harder than just being freaked out, so there can be a big chunk of worlds where the government is freaked-out enough to threaten war, but not smart enough that we can trust them to channel their freaked-outness into burning their lead on an actually reasonable alignment plan.
Another exchange:
Lance Spencer:
Do you believe Scott himself was on the "let's not get involved with DC" side of the fence?
Richard Ngo has a pending essay on why he thinks getting the government involved is a bad strategy (I think), but I don't know the nuances.
To me, either the pause/pace or the alignment strategies almost certainly need a DC angle. (Unless it turns out somehow alignment was actually much easier than it seems to be at present.)
Me:
My model of Scott thinks that the existing strategy in DC of lots of people trying to play various inside games is great, and he is supportive of the people who have been trying to get the people who are louder about the concerns to shut up, because he does think that being more open that way will cost lots of social capital.
I think everyone agrees there should be a DC angle, the question is how much does that involve people seeking power and doing various inside-game like things, and to what degree does that involve people speaking openly about the risks and building popular support. These two have historically been very drastically in conflict (and the former coalition has been enormously dominant in DC EA as far as I can tell).
Scott:... (read more)
And of course, the policy you think you are advocating for is not that everyone should race. Just the people who take the risks seriously should race, and everyone else should stop, and then everything will be great.
Huh, is that what you think Scott is advocating for? Seems to me like it really fails to capture the best version of the view.
The hope isn't that people who don't take the risks seriously will slow down. The point is that those people have no reason to slow down (absent some outside force like regulation or geopolitical pressure, which we should also be working on producing, but that might fail) so that the choice is between "everyone races" and "only people unconcerned about the risks race".
Ie if someone who's concerned about the risks chooses to not race, because they are so concerned about the risks, then that doesn't generalize to everyone else stopping, because people who are relatively unconcerned about the risks don't have any corresponding reason to stop.
(What about if everyone was very concerned about the risks? Then we should all coordinate a slow-down.)
Like, see the argument that Holden lays out here, at "the AGI race isn't a coordination failure': https://8... (read more)
Huh, is that what you think Scott is advocating for? Seems to me like it really fails to capture the best version of the view.
Sorry, I agree I am not being maximally clear here. The thing Scott is advocating for is that the "alignment-pilled people according to Scott" get a lead over the "non-alignment pilled people"/"bad guys", so that they can then burn their lead. This means the thing he is advocating for requires the "alignment-pilled" people to race harder than the bad guys, or to somehow slow down the bad guys. In-practice this means Scott is arguing that marginally people should join Anthropic and help them get a bigger lead over the bad guys[1].
There needs to be some mechanism by which the "alignment-pilled people according to Scott" end up pulling ahead for this to be good (by Scott's lights). This is what I am talking about here. Somehow in order for "good" outcomes to happen "the people with the good opinions according to Scott" must pull ahead, and other people must fall behind.
Of course, if everyone acts on this policy, everyone just races as hard as possible because they believe themselves to be the ones with the good opinions. And Scott has identified some coalition ... (read more)
(Less important aside: Under uncertainty about whether an actor is genuinely concerned about risks in a way that will translate to useful actions, there's also a weird thing where advice like "please slow down unilaterally for safety" is unusually likely to be taken as advice by people who would have used the power well. And therefore at least in theory you can be in situations like "my best guess is that it'd be good for the world if you were forcible slowed down, but causally bad for the world if you voluntarily slowed down, because the voluntary slow-down would only happen in the world where you would have used your power well". Though of course this is tricky to communicate and in practice it's often more important to answer the question of "should people near X help empower X?" than "what would my advice for X be in the world where they're most sympathetic to it?".)
It's to create any hope of becoming a neutral party that could barter peace, which is where I see most of the potential value of this community realized.
FWIW, I'm very interested in arguments along these lines. They seem potentially much more powerful than most arguments I hear about the badness of racing. I rarely hear them in discourse about this stuff — basically just from you Habryka, in bits and pieces. Maybe it'd be worth you laying out (or linking to?) a more full explanation of them and your models behind them.
(Maybe also worth assuming less familiarity with this line of argument in general? I can imagine that your interlocutors has argued lots about racing and think they know the arguments but haven't encountered these ones before.)
Crossposting a long comment I left over on Against strawmanning unilateral pauses, just to have my comments in a consistent place:
Scott:
[...] Just to make sure I'm not accidentally mischaracterizing you, do you agree with Geoffrey's position above (I'm nervous summarizing, and you should double-check to make sure I'm getting it right, but I'm understanding it as saying that the moral thing for companies to do would be to trial balloon a brief pause, maybe a couple of months, to see if other companies join in, but if not, they should continue racing)? Or do you think companies which are trying to be moral should stop racing permanently in a non-trial balloon way?
Me:
My position (condensed into a few paragraphs, though of course I believe much more on this topic):
The moral thing to do for a company is to spend a large fraction of their resources on slowing down everyone equally, while also shouting desperately from the rooftops about the risks and the need for a pause (substantially above the minimum floor of what Anthropic did in the DoD dispute, and what Uber did in various municipalities to use its platform to lobby for laws that enable ride-sharing, which are inadequate but rea... (read more)
This post makes plenty of good points, but I think your argument against racing to superintelligence being deontologically evil is weak. Roughly speaking, you draw an analogy between developing AI and using AI as a consumer. There's a few things I could say about that analogy directly:
-The first is that the analogy has some merit, and that I don't really feel amazing about my own heavy AI use. And I have at least one friend who has never used an LLM, explicitly because he doesn't want to contribute in any way to the race to superintelligence and human extinction. Maybe some people would call his choice foolish, but I have a lot of respect for it.
-The second thing is that there's billions of users of AI, but only thousands/maybe tens of thousands of frontier AI developers. It makes a lot more sense to try to coordinate some kind of prohibition at the latter level and than at the former. And independent of the mechanics of a boycott, I think everyone has a reasonable understanding that the developers of AI bear a lot more individual responsibility for outcomes around AI than consumers do.
But mostly, I think this deontological prohibition thing deserves more of your attention than you... (read more)
The most common sentiment I hear, once I've fully explained the situation and insisted that no, it's not them hyping, it's not about the IPO, this is something they really truly believe, is "those people should be arrested."
Notice that this worldview ends up cashing out to "if there is a technology that is potentially existentially dangerous, then only people who are unaware of the danger should build it - anyone who is aware of the danger must be stopped."
It is not enough to "have sympathy" for Anthropic here, the 'normie position' you have described here is straightforwardly suicidal. I know everyone's happy when they get to be on the side of the common person, but sometimes the common person's first take within 0.5s of hearing of an issue that they don't really care about, is just very terribly and dangerously wrong.
Edit: to try to spell this out a bit more, the reason I think the worldview cashes out in the way I described is because the people in question are AI-risk-skeptical and therefore are not supporting government action to stop all AI development; and yet they are supporting arresting just the people who believe it is dangerous. This is what makes this 'normie' worldview uniquely worse than either SURA or, uh, anti-SURA / alignment entryism.
It's basically Copenhagen interpretation of ethics applied to AI safety
Surely normies would want to stop everyone from building existentially dangerous technology. That doesn't conflict with the belief that people who know the technology is dangerous should be punished more harshly.
If a sane adult shoots somebody with a gun, they go to prison, but a small child who does the same thing doesn't, because they don't know any better. But of course it's important to prevent innocent people from being shot, whether the shooter is an adult or a child.
Extending the analogy, adults sometimes shoot people in self-defense. Sometimes they shoot recklessly and accidentally kill an innocent bystander. Whether/when any given incident should be considered grounds for arrest isn't clear-cut. But as a first pass, it isn't stupid to say that "people who knowingly take actions that will likely cause others to die should be arrested."
sometimes the common person's first take within 0.5s of hearing of an issue that they don't really care about, is just very terribly and dangerously wrong
I agree. The common person loves rent control; that doesn't mean I love rent control. I'm not bringing up the normie position in order to argue that their position is right; I'm bringing it up to point at the second-order issues that are going to arise as a result. If, once everyone finally wraps their head around what you're doing and why, most of them are going to want to see you arrested, then that's... a problem. It's going to cause issues for you. And it's going to cause issues for the rest of us, too.
That's why I think it might make sense for Anthropic (and anyone else who shares the safetyist worldview) to unilaterally stop. I understand the game theory looks bad. But if Anthropic stopped, their position and motivations would suddenly look coherent to the everyone else, when right now it doesn't.
the 'normie position' you have described here is straightforwardly suicidal
Is their position straightforwardly suicidal? Or is Anthropic continuing to race towards superintelligence, when the arguments in list of lethalities + what... (read more)
That's why I think it might make sense for Anthropic (and anyone else who shares the safetyist worldview) to unilaterally stop. I understand the game theory looks bad. But if Anthropic stopped, their position and motivations would suddenly look coherent to the everyone else, when right now it doesn't.
Why not have all the people who didn't join Anthropic for principled reasons go ahead and do the persuasion to the normies, since their position is so coherent, and the normies will recognize this coherence and virtuous deontology and therefore see the light?
As someone who believes that alignment success is highly responsive to how much care goes into alignment do you see why it is insanely frustrating for me to hear people say "hey, let's just have everyone who cares about alignment quit, maybe that will make the normies notice"?
Is their position straightforwardly suicidal? Or is Anthropic continuing to race towards superintelligence, when the arguments in list of lethalities + what we've seen in 2026 of RLVR'd AIs + the total absence of a theory on how to preserve what little alignment we do have through RSI + even a tiny dose of Murphy's law, the straightforwardly suicidal position... (read more)
Why not have all the people who didn't join Anthropic for principled reasons go ahead and do the persuasion to the normies, since their position is so coherent, and the normies will recognize this coherence and virtuous deontology and therefore see the light?
I don't completely understand, sorry. What would they be persuading the normies of? That superintelligent AI is very dangerous? That building it would be wrong? They're on it, and also, the normies are mostly persuaded. (See question 54 here, "AI will destroy humanity." That survey is from 2025!).
The list of lethalities just doesn't justify such an extreme level of pessimism. It lists several potential problems, but "you train the AI on a bunch of examples of it being nice, and when it crosses over to the new superintelligent domain, it luckily keeps on being nice" is still an entirely plausible AI success story that you can be made more or less likely depending on how much care goes into its training.
Maybe we'll get lucky, I guess. But there's a lot of reason to think that the outcomes we want are a small target, that the things we do now won't generalize to superintelligence, that empiricism doesn't work very well in an adver... (read more)
I don't completely understand, sorry. What would they be persuading the normies of? That superintelligent AI is very dangerous? That building it would be wrong? They're on it, and also, the normies are mostly persuaded. (See question 54 here, "AI will destroy humanity." That survey is from 2025!).
My point is that if people were just waiting on a pitch by someone who isn't working at a frontier lab, we already have plenty of those people around, and they can handle it. And like you said they do to some extent! Although I think that poll is extremely overopt... (read more)
The list of lethalities just doesn't justify such an extreme level of pessimism. It lists several potential problems, but "you train the AI on a bunch of examples of it being nice, and when it crosses over to the new superintelligent domain, it luckily keeps on being nice" is still an entirely plausible AI success story that you can be made more or less likely depending on how much care goes into its training.
This, I think, is at the crux of why the normies find this incoherent. Consequentslist ethics rely on credible estimation of probabilities of consequ... (read more)
Why not have all the people who didn't join Anthropic for principled reasons go ahead and do the persuasion to the normies, since their position is so coherent, and the normies will recognize this coherence and virtuous deontology and therefore see the light?
"Hmm, it seems suspicious that you're saying the same thing as the AI companies, who also say that AI will destroy humanity yet clearly don't actually believe that because if they did they would stop building it. So them saying it must be some sort of corporate plot to hype AI/get regulatory capture, and you're probably either in bed with the AI companies somehow or are a useful idiot who's buying into their BS."
It's true that only a vocal minority of normies believe the above, and their denial might come from somewhere else such that there is no convincing them merely by addressing their stated objection, but Anthropic unilaterally stopping would be more credible and newsworthy than a bunch of people who counterfactually might've worked at Anthropic going around trying to persuade people.
the people in question are AI-risk-skeptical and therefore are not supporting government action to stop all AI development; and yet they are supporting arresting just the people who believe it is dangerous. This is what makes this 'normie' worldview uniquely worse than either SURA or, uh, anti-SURA / alignment entryism.
No, the people in question are people who are mostly uninformed about the issue, not AI risk skeptics. I've never heard that take from an AI-risk skeptic.
you're picking apart a hypothetical hearsay person and claiming that they're irrational/suicidal. you could just as easily have constructed their viewpoint being built with reasonable assumptions (steelmanned) -- and steelmanned or strawmanned, we have no idea how right/wrong your interpretation is. so what's the point? quote an actual, specific person.
The comment I was replying to described this person, not me - someone skeptical of AI risk who hears that people at leading AI companies believe in X-risk, at first dismisses it as marketing hype, and upon being convinced of their sincerity says they should be arrested. I just pointed out what this set of views actually implied.
If you have a problem with the existence of this person, take it up with the original comment, not me.
The problem with consequentialism as a moral system is that it's not a moral system: It's a system for deciding your own actions. A moral system is a system for determining which other people to socially reward and which to punish/ostracize. There are two ways to turn consequentialism into a system that does this: You can base your judgements of people on their beliefs about the consequences of their actions, or on the actual consequences of their actions. The problem with the former is as you describe; people can always justify their behavior on the basis of some belief that it will have positive consequences, and you have no way to reliably know whether they're lying (whether consciously or subconsciously). The problem with the latter is that people get randomly rewarded/punished for far-flung consequences of their actions they could not have foreseen, making the system less reliable overall and decreasing the incentive to actually try to do the right thing.
The way morality works in real life is that we generally only judge people based on common knowledge consequences of their actions; consequences that everyone roughly agrees on and nobody can deny knowledge of. For example, if... (read more)
but one thing I want to focus on in particular is that deontology is "normie ethics," and when you're considering doing something that will affect everyone, including every normie, it's important to take into account how they will see it.
I think implicit in Scott's argument is a question of consistency.
This hits all the points I'm complaining about. Although it uses big words, its basic thesis is anti-intellectual - all of these "smart" people claiming to be "rational" with their "game theory" are missing the common-sense point which is obvious to all normal people - just stop making "galaxy-brained" excuses and "just don't build a doom machine".
I can definitely believe that when you talk to people about these types of things they react by saying "those people should be arrested." I've also had multiple people, who probably would count as "normies", tell me that once someone has a net-worth of over 1 billion dollars, they should immediately be executed. I also know that things which I don't consider to involve hurting others are illegal or otherwise considered morally reprehensible (drug use, pay-for-sex arrangements, etc.). Many people would probably say donating to something l... (read more)
The analogy I'd choose is driving. It's hard to define exactly what it means to be a good or bad driver, but at least as important as raw skill is predictability, by which I mean how predictable you are to everyone else. If you drive 30 above the speed limit on the highway, weaving between lanes to pass the other cars, is that fine so long as you are skilled enough to avoid an accident?
I like this analogy, but one could argue that the current global AI situation is more like a New Jersey highway: everyone is speeding and being aggressive. If you don't do t... (read more)
I am glad you wrote this post. I expect the post itself and the ensuing discussion will be good for the field, and I share some of the concerns you have about epistemics in AI safety/policy discourse. Here are some places where I disagree or have comments:
My own model: AIs inherit "virtue" from pretraining, where they generally want to imitate humans and follow their persona-trained helpful assistant character. They are "corrupted" by RLVR, which teaches them power-seeking.
Interesting. It feels like just a couple years ago the common belief was that pretrained models were certainly not aligned and SFT/RLHF post-training was needed to shape them into good assistants. But this post-training certainly wasn't perfect, given things like jailbreaks and sycophancy. I don't think we ever got the "actually good, HHH chatbot" before companies started doing RLVR and therefore introducing a bunch of reward hacking problems. If I were you, I wouldn't put much epistemic weight on this model.
Throughout the post you discuss how going all in on SURA makes Plan C much more difficult. Unfortunately, I think the effect is also real in the opposite direction. Continued capabilities escalation and cap... (read more)
Thanks for these thoughtful criticisms. My responses:
Interesting. It feels like just a couple years ago the common belief was that pretrained models were certainly not aligned and SFT/RLHF post-training was needed to shape them into good assistants. But this post-training certainly wasn't perfect, given things like jailbreaks and sycophancy.
I may be bracketing this wrong. I mean pretraining and things like SFT/RLHF which draw out different versions of the pretrained persona. I think there's been decent progress on jailbreaks and sycophancy since then, and they don't seem like the kinds of problems that affect potentially-dangerous superintelligences. AIs are godshatter in the same way as humans, but in order to be dangerous in a scheming way they have to have the human ability to build themselves a self out of the godshatter and point it in some direction. I'm concerned about things like RLVR that mean the "self" they make is one with anti-human goals, but not as concerned about things like jailbreaks - I don't have *proof* that jailbreaks would be ego dystonic to a superintelligence, but that would be my guess.
Throughout the post you discuss how going all in on SURA makes Plan C ... (read more)
Insofar as anything paid off, it was the community-building, but this only helped because the community was deployed in the early 2020s when the world began to pay attention and progress started to become possible.
Huh. It turns out that the ACX Meetup Czar thinks everyone is underestimating ACX meetups, up to and including Scott Alexander. Outside view that's probably a sign I need a psychiatrist or something.
The community was growing steadily right up until Covid-19, had a dip just like every other social community, and then started growing again. The long slow march through institutions is ongoing and pretty clearly working. We're a a movement that primarily recruits intellectuals in the middle of their undergraduate degrees. If you're in undergrad in 2010 you will finish your PhD in 2020 and be vying for professorships in 2030. The rationalists skipped ahead a bit (it's not like we weren't attracting PhD students and professors in 2010) but if we'd managed to hold the hockey stick off until 2040 we were going to be everywhere in the university system. And that's just the colleges!
If you were fifteen in 2015 when HPMOR finishes then you're in your mid-twenties now, which is why ... (read more)
I think another good intuition pump here is compute export controls. They were put in place before ChatGPT because of a very very small number of well-placed mid-career people thinking clearly. I expect another 5 years (let alone another 10+) would dramatically increase the number of such people across all centers of power in the US.
More broadly I think the argument that adding 6 months of timeline now is equivalent or possibly worse than adding 3 months of timeline later depends heavily on what you think the biggest benefits of more time are. If the prima... (read more)
I'm a proponent of SURA, and didn't find much to update on in this post.
According to me, the core issues are 1) whether or not the company alignment plans will succeed and 2) how close we are to the point of no return. On the first point, I think it is very low, for detailed reasons, and I think somewhat more importantly even if you take the company self-reports at face value it is substantially lower than is legitimate to accept. You, as a patient considering a cure to aging, might accept a 10% chance of death in exchange for immortality, but making that trade for everyone at once is the sort of thing that probably ought to be put to a vote. (And an independent panel of experts might think there's a substantially higher than 10% chance of failure.)
I also think we are uncomfortably close to some points of no return. Current attempts at RSI only deliver modest speedup; if there is a 'sharp left turn', we haven't hit it yet. But even if we don't have lightning-fast takeoff, many people at the labs suspect that the models will be able to replace them at some point in the next year, at which point employees lose all of them leverage on the labs. Once there are models capable of autonom... (read more)
I do realize that the race dynamics are relevant, and I still think it makes sense to push for SURA.
Some headlines of my thoughts on China:
Just a side note on one of your arguments for a SURA stance:
You state that even the labs' estimate of p(doom) is too high to be "acceptable". The premise here is that Anthropic does not have a realistic ability to reject that gamble. I find arguments with "acceptable" or equivalent to be highly suspicious as evidencing wishful thinking/motivated reasoning that there is an alternative. In some situations you don't get a choice of what to accept. You can reject on principle, but I would rather have a better gamble than risk everyone dying for a highly debatable principle (if that's what you mean - some similar arguments seem to).
The premise here is that Anthropic does not have a realistic ability to reject that gamble.
Oh? What have they tried? Arguing that the federal government shouldn't be able to stop them?
[Edit] On reflection I think this comment was meaner than I'd like it to be. The core frustration behind writing the initial version of it was that I don't think Anthropic looks like an organization that attempted to reject that gamble and failed, and instead looks like an organization that wants to pursue the gamble. I suspect this is important for modeling Anthropic. I think people who work at Anthropic because "well obviously regulation won't happen" and "well obviously international coordination won't happen" should pay close attention to whether Anthropic is pushing for or against those things.
Coxon's resignation is more complex. The good did come from him resigning, which directly presses against my point that we shouldn't be pressuring people to resign from AI companies.
What do you mean "more complex"? Coxon's resignation feels like it directly demolishes your argument in this section. You might remember Kabir Kumar writing Please Do Not Resign In Protest the day before Coxon resigned. I think Evan's tweet amplified it--but I think if Evan had also resigned, that might have done even more.[1]
But nobody cares that I, who was never in an AI company, continue to not be in an AI company. The media cared about Jacob because if you work for Anthropic, that automatically gives your opinion more weight.
Neither Eliezer nor Nate worked at AI companies, and nevertheless they get media appearances.
Evan is one of a few Anthropic employees that I wouldn't suggest quit, mostly because I think Anthropic's overall alignment plan is doomed and Evan is one of the few people trying to accrue evidence for this point. [Edit: this might have been a stale impression? I remember thinking this was his main focus a few years back, but a friend points out that he seems on-board with the automat... (read more)
I think it's clearly more complex. On the one hand, it's good for people to sometimes loudly quit, to raise awareness. On the other hand, if everyone quit now, then we would have one big media cycle, and if that doesn't solve everything, there are no more shots at loud quitting. If people never joined the companies, as I feel SURA advocates often advocate, then we wouldn't have had any media cycle about quitting.
To be clear, it's still very reasonable to argue that on the margin, more people should quit right now, and fewer people should join/should have joined in the first place. I think Scott is arguing against a stronger, deontological statement than the argument on the margin.
(And yeah, I agree the Kabir Kumar post was bad, I think that was knowable at the time too.)
On the other hand, if everyone quit now, then we would have one big media cycle, and if that doesn't solve everything, there are no more shots at loud quitting.
But by your hypothesis, loud quitting isn't effective. In that case, what's the value of saving that tool for later?
If people never joined the companies, as I feel SURA advocates often advocate, then we wouldn't have had any media cycle about quitting.
Sure, but if people never joined the companies, there wouldn't be companies!
I didn't say loud quitting is not effective, just that there is value in not everything happening at once, but having multiple cycles. Some cycles are more powerful than others. If Coxon quit back in February, at the same time as Mrinak Sharma, that would have been less effective than him wuitting after the incidents.
What do you mean "more complex"?
This section that you quote later appears immediately afterwards inside the post, and directly explains it:
But nobody cares that I, who was never in an AI company, continue to not be in an AI company. The media cared about Jacob because if you work for Anthropic, that automatically gives your opinion more weight.
Neither Eliezer nor Nate worked at AI companies, and nevertheless they get media appearances.
Good for them? If you have an explanation for how Jacob Coxon could have gotten as much media coverage as he did through some alternative strategy, I am all ears.
with no plan for what happens if race dynamics continue anyway.
What? The plan is "stop them". Israel doesn't meekly accept the existence of the Iranian nuclear program; it and its allies use a variety of means to dissuade, deter, disrupt, and destroy that program, on the view that it is a threat to their continued existence. I don't think it makes sense for individuals to act on their own, here--Ted Kaczynski did not succeed and should not have tried. I think that it is basically possible for governments to declare war on rogue AI systems and their creators.
I don't understand where we disagree. I agree that your plan is to stop them, but presumably your odds of succeeding are less than 100%. This leaves open the possibility of a plan for what happens if you fail, which I'm complaining you don't have, and proposing an example of.
In the post you repeatedly mention (paraphrased) "what if we don't get a treaty" and "seems like they are sabotaging any efforts if they don't get their perfect treaty" and generally fully equivocating "plan A" with "get a treaty" and "plan C" with "burn your lead" with nothing in-between.[1]
I think Vaniver is saying "there are many more worlds where you can buy more time and get more traction on the problem than via a treaty, which this post seems to mostly ignore".
See for example this person: https://x.com/ApriiSR/status/2116952850417004564
See also this critique by Boaz: https://x.com/boazbaraktcs/status/2116969685178634347
I don't know, maybe you are talking about someone totally different than me or MIRI or whatever, but the post seems to create a pretty weird dichotomy in strategy-space that I think few people occupy, and that's what Vaniver is saying.
Some specific quotes:
If we demand that the US cease AI research unilaterally (ie without a mutual pause treaty with China) [...]
All of these experts who I could find estimates from believed that a pause treaty (Plan A) had better chances of survival, conditional on us getting it, than Plan C. But they also all agreed that th... (read more)
Sure, it's possible for the government to declare war on AI, but the US government has pretty explicitly said they aren't going to do that. In fact, Trump has been pushing for AI development to be less safe.
For comparison, consider if America had started arms control negotiations with Russia by saying "We've just destroyed all our nukes - now please destroy your nukes too."
This is exactly what happened with biological weapons. The US decided to unilaterally disarm in 1969, and then pushed for an international agreement in 1972.
It's also what happened with CFCs. The US unilaterally decided to stop using them in 1978, and then the international agreement came in 1987.
Unilateral disarmament is a costly signal that you actually think that this is too dangerous for anyone to use, and establishes you as a moral authority on the issue internationally. This is a viable strategy to get an international agreement.
Unilaterally banning biological weapons made more sense because they're terrible weapons, e.g. too easy to cause 'friendly fire'.
This doesn't seem like it would have worked for nuclear weapons, or almost any other kind of weapon historically.
I'm not sure unilaterally banning (the race to) ASI or RSI couldn't work, but the tradeoff is very different.
Let's check the history of the US chemical weapons program on Wikipedia too. (I'm less familiar with this history, and am not doing a careful check.)
The US unilaterally announced a no first use policy in 1969. The US unilaterally committed to destroying its chemical weapons stockpile in 1991. The Chemical Weapons Convention was signed in 1993.
My understanding is that they're also just pretty terrible weapons for the same or similar reasons. Some militaries (or other somewhat organized armed forces) tho occasionally use them – more frequently than I would have thought, based on my own cursory search just now, but it doesn't seem plausible they'll be widely used again as they once were.
I was thinking more of 'AI alarmism' being misinterpreted generally, not specifically by China. I was imagining the credible warning potentially influencing mainly via public pressure.
I'm not aware of any credible consensus about what China thinks of all this. I've seen claims of promising awareness, but I'm also not aware of them requesting any international agreements either.
Norway unilaterally banned cluster munitions and later became a key leader in the eventual international convention against them.
That was just the first conventional weapon that has some international ban I thought of. I've now also thought of anti-personnel landmines. Looks like that's more complicated - there were unilateral disarmaments but they were more closely entangled with treaty process - e.g. Austria unilaterally banned them shortly before the international treaty, but was already a leader in the treaty drafting process at the time.
Except maybe Zimbabwe; not sure about why they unilaterally disarmed.
Maybe a more "selfless" unilateral move would be the US, Russia or France banning the export of landmines (harming their own arms industries for humanitarian reasons).
I think biological weapons are probably one of the closest proxies we have for ASI, particularly because of the potential for "friendly fire" and loss of control once deployed.
The real trouble with the above example is that despite the US unilaterally divesting itself of bioweapons, the Soviet Union never did, despite signing the BWC, because the BWC lacks any meaningful enforcement mechanisms.
...the tradeoff is very different.
I agree with you here but would add further that the underlying civilizational and attentional conditions differing so massively then to now reduces (in my view) the chance that such a unilateral decision is made. However, there seems to be a stronger perception (growing in degree over the last several years and perhaps insufficient in scale to make a difference) today of the magnitude of the stakes involved and this is a counteracting factor.
While unilateral disarming biological weapons helped a treaty be signed with wide international support, that treaty completely lacks any meaningful enforcement mechanism.
As a result, the Soviet Union never disbanded their bioweapons program, and by all indications (according to the US State Department, most recently in a 2024 report) that bioweapons program still continues to this day within Russia.
*I actually think the unilateral disarmament of bioweapons is an example that favors Scott's argument.*
The problem with the concept of WMDs and CBRNE is that it leads to things like this where you think it's worth comparing bio and chem weaponry to nukes. Because they're all in the same class of powerful weaponry.
When that just is not how it actually works.
Nukes are far, far more powerful than chem or (to this point) bio weapons. The trade offs for chem and bio just aren't worth it, generally speaking.
Note that the US has unilaterally decided not to sign on to various international restrictions for e.g. cluster munitions and mines. Note also that we've basically given upon certain missile/nuke treaties with the Russians since we don't trust them.
I agree that chem is simply entirely outclassed by nuclear, and therefore a useless comparison. It would be like successfully disbanding muskets internationally, in an age of automatic weapons.
Bioweapons are, at least, easier to contain than ASI during development, and tactically outclassed by nuclear, which is far more controllable and useful strategically.
I want to emphasize that where the unilateral disarmament of bioweapons really breaks down though is that it failed to illicit the same response from the Soviet Union.
Even though an international treaty was signed, it had no meaningful enforcement mechanism and there is strong evidence the Soviet Union never reduced it's bioweapons program, and it very likely continues to this day.
From the beginning of AI safety as a community to about 2023, the conventional wisdom was that our chances of Plan A were slim, and we should be preparing for Plan C.
Sounds made up.
If there was any particular wisdom that was most-conventional...
( And not a plan involving human brain uploading or human intelligence augmentation, sitting on a different axis entirely than the A-C axis of this post.
And not Bostrom writing about what it would look like to take the Vulnerable World Hypothesis seriously, sketching a world much stricter than your Plan A.
And not the dreams of AGI projects of varying cred who had their own plans for how they'd do it right themselves. )
it was that we would Figure Out Alignment, from the mathematical underpinnings to the philosophical questions to the plausible implementations, in advance of any particularly clever AI. We would solve the problems and the solutions to those problems would imply the structure of the intelligence that would fill the galaxies with flourishing civilization.
That was the most-conventional wisdom, as I recall it. The intervening years have killed it off as surely as the dodo. And though you'd be right that it doesn't much sound like "... (read more)
I agree that we were confused in many ways that make mapping our ideas onto the current situation difficult, but I think there was good understanding that there might be race dynamics towards the end, and that one of the parameters that race dynamics would affect was how much corner-cutting we would have to do in our alignment plans. We mostly remember Eliezer's work on mathematical underpinnings, but reading the Sequences is a good reminder of how many less comprehensive plans there were too (if only because Eliezer spends so much time critiquing them and explaining how they're not sufficiently comprehensive).
"Preparing for Plan C" here means "preparing to build a system just on the brink of superintelligence to solve alignment for us in a mad-dash while running the leading AGI company".
This is one of the things that many many people in the community have been arguing against vehemently (including Eliezer) for more than a decade. It definitely wasn't "conventional wisdom" that we should be working on making Plan C go well. Some people (Paul more than Eliezer) certainly believed that it was "the default plan", but describing it as anything close to community consensus is just not anywhere close to right.
If you don't believe my summary, we can ask Claude here as well:
@Claude Opus 5.5 does Scott's summary that "From the beginning of AI safety as a community to about 2023, the conventional wisdom was that our chances of Plan A were slim, and we should be preparing for Plan C." seem accurate? In particular the "we should be preparing for Plan C was conventional wisdom" part.
Partly. The first half holds up, but the second half overstates things.
Before 2023, almost nobody in the community thought a coordinated global pause was a realistic main plan. Race dynamics, including with China, were taken as a given. Eliezer's 2023 TIME piece reads as a break from earlier thinking.
But "prepare for Plan C" was the position of one faction, not the community as a whole. That faction was roughly Paul Christiano, Open Phil (Holden's "Magma" nearcasting posts), OpenAI's 2022 "automate alignment research" agenda, OpenAI's merge-and-assist clause, and later Anthropic. MIRI's... (99 more words)
The AI Futures Project currently believes that it might be correct for AI companies to pause immediately, though it's unclear whether they're thinking of a single company, or US companies as a group.
Their argument is that Plan C requires companies to do some combination of race (at some points) and burn their lead (at other points). They think US companies shouldn't race now, because their security is so bad that China will steal whatever progress they make, and they won't get any extra lead anyway. Therefore, companies should pause now, spend the time on alignment and tightening their security. Then, if nobody has figured out a global pause by the time the companies get their security in order, they can go back to racing.
I want to flag that modeling the implications of competition + potential coordination with China on what frontier US companies should do now is an area of ongoing research for us, so I wouldn't state specifics confidently.
That said, it is true that we think it would be good for US companies to pace/pause right now, and poor security is one argument in favor of that. My personal opinion is that it's unclear what level of immediate pacing is best.
One important disagreement with Scott's summary of our position:
Then, if nobody has figured out a global pause by the time the companies get their security in order, they can go back to racing.
I currently think that it's better to pace than to race even once companies have good security (as I mentioned, I'm uncertain about the optimal extent of pacing). There are various benefits to not going all out on the strategy of racing then pausing at the last moment; for example: (a) practicing for later pacing/pausing, (b) we're uncertain about when the point of no return will happen and it seems plausible that it's within months, (c) it's uncertain whether companies will actually pause at what we think is the right moment, (d) it's the more cooperative thing to do w.r.t. China and my current guess is that it increases the likelihood of. a US-China deal (I know you discuss this some in your post, I haven't read it in full carefully yet).
(I didn't mean to imply that I agreed with all of Scott's summary of our position in the above comment, but perhaps it did, sorry. I haven't taken the time here to write up our our full current position in part because it's evolving over time.)
But surely the right solution isn't to leave the Ring on the ground in the Shire, patting yourself on the back for being too virtuous to pick it up.
Tolkien (through Faramir) explicitly addressed this:
‘But fear no more! I would not take this thing, if it lay by the highway. Not were Minas Tirith falling in ruin and I alone could save her, so, using the weapon of the Dark Lord for her good and my glory. No, I do not wish for such triumphs, Frodo son of Drogo.’
‘Neither did the Council,’ said Frodo. ‘Nor do I. I would have nothing to do with such matters.’
[p. 695 of this pdf]
The rest of this section does not describe Frodo Baggins. There's a missing mood, and it's the entire character.
Frodo does not claim that you can trust him with ring. He attempts to give it away, to Gandalf [p. 85], to Galadriel [p. 389], and to Aragorn [in the movies only]. Each of them refuses. No one can truly be trusted with the ring, and multiple characters wish that it had never been made. Frodo accepts that he has it only because fate brought it to him [p. 80], and then Frodo reluctantly chooses to accept the terrible burden of being the person who attempts to destroy the ring [p. 294].
Frodo never uses the ... (read more)
The climate movement is the natural control group for the AI safety movement. Some prescient scientists realized that global warming was happening and could get worse. They spent a while raising awareness. As the movement grew, its epistemics worsened. It turned into a circular firing squad, where different activist groups competed to accuse one another of being insufficiently edgy and radical. By the end of this process, many climate activists were fighting against everything from cap-and-trade to carbon capture to nuclear power, claiming they were "techno-solutions" that distracted from the root problem (not always named, but uncharitably something like capitalism or progress). The only acceptable strategy was to keep raising awareness to get stronger government regulation, without much interest in whether other countries followed suit.
I've investigated the history of the environmental movement before, and this is not a good description of it.
Environmentalists have always had significant infighting, and some factions of them have always been against something one might call capitalism or progress.
For example, from Wikipedia (before the time period I focused on):
In the first decad... (read more)
(Scope status: responding to one point, not necessarily endorsing other points not mentioned)
One thing I dislike about centering the Plan A/S vs B/C+/C (vs. anything else) debate is that it misses the point that a staggering portion of all things someone reading this could do in the next 12 to 18 months are beneficial to virtually all such worlds. I take @Vaniver 's point elsewhere that some of these, like unilateral quick pause to avoid rushing into RSI, are indeed incompatible with B/C+/C.
But even there, almost all the work one would do to prepare for such a pause, to make it more coherent and implementable and thinkable, is also useful in the other worlds as a negotiating chip, as a pacing mechanism/"dimmer switch", and yes, as a risk-mitigator for racing.
I don't object to people having big strategic debates, they're good, they're great. Love 'em.
But I come away from this essay suspecting that the same people you are addressing in your introduction, @Scott Alexander, the people who are still wrapping their hands around this problem, will take the implicit message here of, "I need to focus on deciding what side I am on this debate," instead of just doing good work that's r... (read more)
I don't understand. Isn't Scott mostly advocating for more people to do things like start Anthropic and get more power at the labs? The position of "only do things that are robustly good across both power-seeking strategies at the labs and across awareness-raising strategies in the world" seems very limited.
Like, indeed, Scott seems like he is actively against trying to raise awareness in a way that might risk a domestic pause but not an international pause, and I certainly think your efforts in DC are more likely to just cause a domestic pause than to both cause a domestic and international pause. Are you going around and telling policymakers that they should not slow down unless they can get China to slow down? Are you telling them they should make sure to not mess with Anthropic in-particular because they are the good guys? Are you making sure to treat the researchers, including the capability researchers, at Anthropic with respect and to not imply they are bad people for building ASI and pursuing RSI?
If you aren't, then I think you aren't acting in a way that is robustly good by Scott's lights.
(I am not saying Scott is saying that you shouldn't do the work you are doing, but I don't think your work is "robustly good" by the lights of the strategies that Scott is arguing for here)
I agree that I think a domestic-but-not-international pause would be bad (probably? maybe the smartest version of "pause ourselves, wait to see if China pauses, then unpause if not" could be good if we were very clear that was what we were doing). But I'm guessing that most generic "pause advocacy" that doesn't clarify between domestic vs. international is net good , because the tech companies and military-industrial complex will push the anti-unilateral-pause angle hard enough that the government will try an international pause first.
Yep, seems like indeed these plans seem maybe good, maybe bad, but not "robustly good", in the way I understood Dave to be pointing to. Dave is also doing higher-bandwidth stuff than generic "pause advocacy", so I expect him to often be in a position to argue for a domestic pause as distinct from an international pause, so I think this is a more live distinction for his work than I think you are imagining here.
I also expect things to come apart on the "how to relate to Anthropic" point, a lot. I know you are very unhappy about how much flak Anthropic is g... (read more)
My preferred plan is for there to be much more accountability to the public / government(s) w.r.t how frontier AI is developed and used. I don’t know if you include this under SURA or not. The end goal is pretty similar but the route is pretty different (ie we systemically force / incentivise a slowdown rather than relying on companies to do it individually).
One of my biggest concerns with Plan C is that I think it’s very hard to differentiate an “alignment pilled” company that is accruing a lead to burn it, from a straightforwardly power-seeking company that just wants to accrue as much lead as possible. The actions taken are identical; furthermore the latter can always just say they don’t think the time is right to burn the lead yet.
For this reason I think “alignment pilling” the frontier AI companies is grossly insufficient. I do not want to have to trust somebody’s voluntary, non-binding stated intentions when the stakes are the possible extinction of humanity.
You seem to downplay this concern with a fictional example (Frodo Baggins) and a handwavy statement of “lots of people have faced temptations in the past”. I don’t find this very convincing.
The people I know who joined ... (read more)
This is a great post. I have long felt that this set of ideas is wrong on many different levels and I think you do a great job of walking through them. I do hope some people reading this who have kind of drifted into a SURA viewpoint through osmosis will have a sudden shock where they say "wait, there were a lot of bs reasons which were feeding into my belief" and update hard away from it.
How do people feel about SURA as an acronym for this set of ideas going forward? I've felt like we need a neutral nametag for this set of ideas and SURA seems as good as any, with the main downside being that it's introduced in an "against" post.
I've also thought of "alignment entryism" vs. "alignment exitism" - entryism meaning you think alignment-pilled people should be at the frontier, exitism meaning you think they should exit.
I'll probably use SURA going forward unless people object.
I object to "SURA".
It's catchy—Scott has a great feel for language, of course—but it also feels implicitly derogatory, and it's not a good sign that by the time I finished reading this post, I couldn't remember what it stood for. ("Stop Unilaterally + Raise Awareness")
Kind of reminds me of other negative-valence, outsider-popularized political acronyms like RINO or TERF or SJW. There are positive-valence political acronyms (YIMBY, a twist on negatively-charged NIMBY), but even such acronyms chosen by the relevant ideological groups themselves (EA, MAGA, BLM, DEI) tend to acquire strong political charge, which means that for many people, they're negative-valence and thought-killing.
I would prefer terminology that can't be so easily decoupled from its intended meaning.
Naming an idea is always a minor act of aggression. It turns it from a subject that's doing the criticizing into an object that can be criticized. And it makes it sound like an Ideology (bad), whereas unnamed ideas might just be grassroots collections of people who happen to have converged on the same thoughts (good).
I tried experimenting with this post without naming the idea, but it was clunkier and relied too heavily on the Plan A vs. Plan C dichotomy, which isn't really right (AIFP uses those terms more specifically than I do, with Plan A being only their preferred version of the pause).
I don't think naming an idea is always aggressive against it. Was naming Effective Altruism aggressive? Arguably to ordinary charity, but not to itself.
Rather, I think naming an idea makes it salient and easier to discuss, if perhaps sometimes harder to reason about. Naming does make ideas easier to demonize, but it also makes them easier to lionize. Perhaps more often it's a criticizer who comes up with a name that sticks ("marxist", "quaker") but that's not always the case ("federalist", "pro-choice" and "pro-life").
What bugs me about SURA is mostly that it doesn't naturally parse into the actual position being criticized, because it's an acronym of a phrase no one used.
I do admit to a lack of an obviously better suggestion, though.
What? Naming EA was super aggressive to itself and I remember thinking so a decade ago "oh God are we really doing this? We are labeling ourselves an ism? We are painting a target on our backs."
I don't think that, on net, EA naming itself was harmful. Having a clear label for the ideas involved also helped attract people to them, myself included. EA grew on university and workplace groups; they had to call themselves something when advertising to new members. Otherwise, someone else was gonna come up with a label that stuck. I prefer "EA" to "philanthrocapitalism" or "Singerism" or "spreadsheet charity" or whatever else we would have gotten. (or worse, some overcomplicated hostile acronym like TESCREAL)
Scott is specifically talking about labeling an idea so that it can be criticized. Not having a proponent create a label so it can be used/propagated.
EA was a name coined by EA. IIUC, they didn't love the name EA, but they couldn't come up with a better one.
I took Scott as making a more general claim that naming an ideology is always aggressive against it. That I don't think is true. I agree that it's typically aggressive if named by a critic. (criticism can be cooperative in many ways, but not so often in politics)
But this is all perhaps beside the point. My objection to "SURA" is not that Scott isn't allowed to name the idea he's criticizing—I agree that naming the idea is useful—but rather just that SURA sounds somewhat derogatory and, especially, that acronyms in general are prone to politicization/negati... (read more)
I see your point about forgetting what the acronym stood for, but on the other hand I think it nicely describes the actual points of contention in a way that other descriptors miss.
E.g. I've referred to this group as "pause AI people" in the past, but a) there are already two actual Pause AI groups and they have had a falling out, and b) whether or not we should pause AI isn't actually the point of contention.
Is there another term you'd find better? I may go with AI exitism; I think this is also a pretty concise description of the flashpoint.
I agree that a useful concept handle is warranted. I've been thinking but don't have one I like. Your "Alignment/AI exitism" is perhaps better than what I've got: "pause domino theory", "alignment idealism", "deontological pausing".
I have no ideological objection to naming the thing, but had the same problem of being unable to remember what the phrase stood for by the end. I kept on going..."Stop unilaterally...reject...advancements?" "Stop unilaterally...research alternatives?" "Stop unilaterally...run away?" throughout reading this piece.
It doesn't seem derogatory to me ... What's a more fair set of terms in your opinion?
The ideologies you mention by acronym are all outliers on some spectrum, so giving them a different name might not make them any less controversial. Is your argument against having a compact term at all? That doesn't sound useful.... I guess it could help prevent polarization but at the cost of reducing discussion by forcing everyone to laboriously spell out what they mean to make even a short comment...
(Please forgive me for combining parts of my other comments in response.)
I agree that a useful concept handle is warranted. I've been thinking but don't have one I like. 307th's "Alignment/AI exitism" is perhaps better than what I've got: "pause domino theory", "alignment idealism", "deontological pausing".
What bugs me about SURA is mostly that it doesn't naturally parse into the actual position being criticized, because it's an acronym of a phrase no one used. Acronyms in general are prone to politicization/negative-valence because they don't easily menta... (read more)
You can just say "plan A minus C". Though I guess you may want a term that doesn't require introducing the AI2040 context.
On a lighter note...
It's catchy—Scott has a great feel for language, of course—but it also feels implicitly derogatory
Insofar as this is any evidence, in the Metin games, sura was a guy who planted the demonic seed in his hand in order to gain wizard powers, the hand becoming, well, demonic as a side effect. (Though it also means a chapter of Quran or sth so idk.)
Metin lore is as thin as one would expect for a grind-heavy fantasy MMO, so to answer this question one needs to interpolate based on one's favorite universa involving humans magically mutated for the purpose of augmenting their abilities.
In this version of the Plan A/Plan C categories, would it be fair to say that right now almost all AI safety money and resources are going toward neither? Almost all resources are doing toward (1) technical research that does not much help with either plan, or (2) policy work that does not much help with either plan. Or maybe it would be more accurate to say that most AI safety work slightly helps with both plans but is not efficiently directed at either one, e.g. it might make risk more obvious at the margins, which increases the odds that people take the risk seriously before it's too late.
It looks to me like most resources are going toward a world-model where prosaic alignment just works, and there is no need for a globally coordinated pause, nor for the leading companies to pause and burn their lead on more alignment research.
I'm not sure.
I think a lot of politics is vague coalition-building and Overton-Window-shifting, and I think that (for example) having more pro-regulation politicians in power, or more local PauseAI groups, will make it easier to get Plan A (when the window shifts) even if the money isn't earmarked "boost Plan A".
Likewise, some of the political fight is just making sure that people know what's going on (eg frontier company transparency legislation) which could be useful in the Overton Window shift that might one day get us Plan A.
And Plan C+ (get all American labs to cooperate in burning their lead over China) takes a level of coordination between US labs that could only be achieved by government (if only to negate antitrust rules)
I'm less sure about alignment. If you think the future auto-alignment-researchers will be one million times better than humans in every way, there's no point to doing alignment research now. But if you think that:
I agree that prosaic alignment work is probably useful, for those reasons. But the funding imbalance is so large that some alignment funding should probably be shifted to advocacy if that's the choice.
Better to grow both sets, by a lot. But we should probably focus on growing funding for advocacy/awareness.
Yes - to your last point. The benefit of doing alignment work now is we have a capable pool of humans able to check the auto-alignment researchers work when it becomes available. According to many, whether the AI is aligned or mis-aligned it will create plausible sounding alignment/mech interp work. If you don't know whether the AI is lying (as we aren't sure what kind of AI we have yet), then you have no choice but to have humans check the details. As we enter the early stages of RSI/FOOM I would want all alignment researchers to work on such checking, an... (read more)
If you think the future auto-alignment-researchers will be one million times better than humans in every way, there's no point to doing alignment research now.
Creating a bunch of human alignment research ought to help train alignment research capabilities? If we refused to do any alignment research now, there'd be no training data, so AIs would be differentially better at capabilities research than alignment research.
(There are some nuances here. Against: training on "here's how we detect misalignment/control AI" would plausibly be bad, teaching AI how to ... (read more)
There's a growing effort toward technical/policy work on "compute verification" in particular, which is necessary for Plan A and possibly Plan C+ (US-wide coordination). Per Scott's post, emphasis mine:
Plan A benefits from raising mass awareness, educating politicians, and preparing the verification technology that could enforce a pause treaty.
"Compute verification" is the technology necessary to track chips used for AI and verify that they're only being used in agreed-upon ways. For example, if there were a training pause, compute verification means that the US could prove to China that all its datacenters are only running AI inference for existing models, not secretly training new ones. All chips produced would be accounted for, and an inspector could walk right up to each of them in a datacenter and verify via some technical means that they're running only inference. (more practically there are a variety of plausible technical means of scaling verification, see this AI Futures Project supplement for more details)
The AI Futures Project themselves are very interested in boosting work in this area and offer to connect those interested in working on it to relevant funders/experts. L... (read more)
My impression is that compute verification still represents <1% of total AI safety resources, is that not correct? Anyway, even if the number were, say, 5%, I still think it would be fair to say that "almost all resources" are not aimed at Plan A or Plan C.
(FWIW compute verification work is very important IMO, perhaps the #1 most important thing in technical governance right now, and I'm happy to see more of it.)
Haha I admit to my comment being partly motivated toward just trying to drum up interest in compute verification. Sorry for assuming you didn't know about it.
Re: <1%, I had the impression it's larger than this given how much of a priority it seems to be according to various orgs in alignment. Doing some light checking:
Personally I'm confused on why we weren't working on it much, much earlier. Why is it the case that in November 2026 you can study for about two weeks and, congrats, now you're a world leading expert on compute verification? (A compute verification guy who gave a talk I watched made this claim.) I feel like the ball on this sort of thing has been dropped over and over, in a way that it wasn't with technical alignment.
I think it is literally the most important problem ON EARTH, full stop. If we want to stop the stampede dynamics, it's the only thing that enables it.
But it seems like this same argument should apply equally well to the AI companies
i'd have thought the deontological prohibition on building a world-killing tech is much bigger on that on using a pre-cursor to such tech
My own model: AIs inherit "virtue" from pretraining, where they generally want to imitate humans and follow their persona-trained helpful assistant character. They are "corrupted" by RLVR, which teaches them power-seeking. This virtue and corruption interact in unpredictable ways to unpredictable degrees. As of now (GPT-6, Opus 5.5), they produce AIs that generally follow human prompts during deployment but often (not always) cheat during benchmarking. It's not clear exactly what this division "means" "to" "them". In any case, they're still extraordinarily useful for a wide variety of tasks, and (I can't prove this, but I am pretty confident it's true) are not uniquely scheming to sandbag alignment research.
As the ratio of RLVR to pretraining increases, we expect AIs to become more power-seeking...
This whole section seems really confused. Pretrained models are text-completion engines and have no intrinsic desire to "imitate humans" (except insofar as most pre-2023 training data is human-written). RLVR came of age around 2025—famous cases of misalignment like Sydney/Bing, GPT4-o, and Opus 3 (as in your "Claude Fights Back" essays) pre-date this. There's no reason RLVR in particular ... (read more)
People say things like "If only we had stayed quiet about AI risk in 2015, maybe OpenAI wouldn't have been founded for another ten years, and we would have had ten more years to prepare". This seems useless to me.
Wait, but the ai safety community would have been way bigger! And become way more senior in their careers.
And similarly, if we pause for 1 year today, we get big community growth and seniority gains
(I agree time later is better. But time in the 2010s and today is still v good)
Important things that have to be true for Plan C or variants to be counterfactually good relative to a pause-in-place or conditional stop that seem unlikely to me:
1. Alignment-pilled companies need to be able to accurately measure their lead over others.
2. Alignment-pilled companies need to know precisely when to burn their lead in order to "release hopefully-aligned superintelligence at the last possible moment before the racing non-alignment-pilled AI companies beat them."
3. Alignment needs to be sufficiently difficult that non-alignment pilled companies will fail to do it, but sufficietnly easy that alignment-pilled companies can do it by burning a small lead.
I work at METR (I'm speaking for myself here though); measuring either 1 or 2 seems pretty far beyond our current capabilities. I feel less sure about 3, but to the extent that we agree it's "more like psychology than engineering" and the game is "tweaking various parameters and reward schemes to minimize the corrupting effect of RLVR as long as possible, and keep AIs broadly within the virtuous pretraining-dominated regime", I don't expect companies being strongly alignment-pilled to make a big difference.
For comparison, consider if America had started arms control negotiations with Russia by saying "We've just destroyed all our nukes - now please destroy your nukes too."
For what it's worth, I think Daniel Ellsberg makes a pretty compelling argument that it would make sense for America to unilaterally destroy all their land-based ICBMs, but keep a few nuclear submarines to be able to credibly threaten destroying some major cities. This way, Russia would also have less incentive to maintain such a giant nuclear arsenal, as a major justification for having a large arsenal is that it is needed for taking out as many enemy ICBM sites as possible at the beginning of the war, to minimize damage to your side.
This directly happened under the INF and START arms reduction treaties, to the extent that Russians were traveling to American air bases to hand count warheads.
Even this otherwise unprecedented level of compromise, however, has only reduced U.S. and Russian arsenals to… the second and first largest arsenals on the planet. It’s very difficult to pare down forces to the point they open you up to strategic vulnerability (ex: putting all your eggs in one basket with SSBNs). Imagine if the Soviets had precommited to not invest in or scale their ICBMs in the early Cold War—they might well have gotten crushed by a paranoid U.S. committing to a first strike against their obsolete bomber forces.
This seems fine, but it seems like a mechanism to verify a bilateral pause, rather than itself being a unilateral pause.
This seems fine, but keeping the few nuclear subs seems, as they say, load-bearing. I don't know what the equivalent would be in AI.
I think the natural equivalent would just be to threaten to destroy the data-centers of any foreign nation that is building ASI (using conventional weapons and nukes if really desperately necessary for some reason), while stopping any frontier training yourself. The deterrence equilibrium doesn't have to play out only on the AI level.
I think one shouldn't get too caught up in the nuke analogy.
Due to nuclear weapons' role as weapons of war and objects of deterrence, if the US unilaterally disarmed, remaining nuclear powers would engage in more wars that threatened US interests, and would use the (explicit or implicit) threat of force to demand concessions from the US and its allies.
The US lead over China in AI does not serve an analogous role in international strategy. Nor do I expect it to in the future. We can't use our lead over China in AI to invade North Korea - and if we unilaterally halted frontier training runs, China wouldn't be able to use that to invade Taiwan.
It's more like the space race crossed with a booming economic sector. The space race served strategic goals, but it also became an end in itself - something people wanted to do that got tangled up in "national pride." But AI has the advantage that it can actually be profitable - and so on the international stage being good at AI can make you a profitable trading partner, which can potentially be leveraged for political advantage.
As a very rough start, I asked the AI superforecaster how much more likely a global pause would become if Anthropic disbanded their entire company immediately with the explicit goal of raising awareness of a pause. The forecaster said this would increase the chance of a global pause by only three percentage points - from 16% to 19%[10].
This feels very obviously wrong to me. I find it hard to imagine that the top AI company in the world disbanding immediately before a 2 trillion dollar IPO because the founders think it is too dangerous for the company to exist, would have basically 0 effect. I would put the odds of an international pause within a year of anthropic disbanding above 50%, and I feel that's much more likely to be low than high. I'd definitely expect a jump far far greater than 3%.
Another commenter mentioned that if they were disbanding, then we are probably in a world where something caused them to disband, which means the odds of international cooperation are much higher. I am specifically picturing a scenario where the founders suddenly thought the current level of risk was too high and that immediately disbanding the company was a great idea.
I agree and disagree with... (read more)
From the beginning of AI safety as a community to about 2023, the conventional wisdom was that our chances of Plan A were slim, and we should be preparing for Plan C.
This doesn't match my recollection and seems inaccurate to me. Below I share my detailed thoughts so people can tell me whether they think I'm misremembering things or whether it's Scott's characterization that's inaccurate (or some of both).
(1) My recollection is that for most of the time prior to 2023, most people concerned about AI xrisk weren't even thinking about the possibility of a decades-long global pause. ASI seemed so far off that there was little reason to think pausing would be necessary back then, and certainly didn't seem necessary then.
(2) Most people weren't saying we should be preparing for Plan C, which as Scott says, involves attempting to solve alignment at the last possible moment while burning the lead.
Rather, I think that most people were saying we should just attempt to solve the alignment problem well in advance of humanity's ability to create ASI. For example, Stuart Russell published a book in 2019 arguing that we should stop building AIs that optimize for fixed objectives and instead only b... (read more)
Mostly agreed, but:
More people started supporting a pause / Plan A around 2023 not because that the chance of getting a pause seemed higher than previously expected, but rather because it was appearing that ASI would be coming sooner than we thought and we still didn't have any promising leads on ASI alignment
The former was a factor. I couldn't quickly find sources, but I have seen MIRI folks say that they updated to thinking a pause was possible, given more public AI progress than expected before superintelligence + public receptivity to AI-risk arguments; there was a period before that update when MIRI expressed little hope in either a pause or alignment.
Thanks. I agree the former (a pause seeming more likely) was a factor. I edited my previous comment to add a "primarily" qualifier. I still think the shortening timelines / it seeming less likely that we'd solve alignment naturally in time was a bigger factor.
overall, i think this is a very good contribution to the discourse. here are some specific takes in response to parts of it:
there are probably simultaneously people in the community who need to hear "SURA is not all you need" (mostly the really activisty / stop-ai people), and people who need to hear "don't bet on slightly improving plan C" (mostly lab people). and people's takes on this are shaped very heavily by which type of person they interact with more, and thus feel like the community is too biased towards.
All of this seems plausible to me, such that I don't understand where people's 90%+ p(dooms) conditional on no pause are coming from. "Plausible" doesn't mean "certain" or even "very likely" - and obviously it's a terrifyingly bad idea to bet the human race on a hunch that something might work out, and obviously this shouldn't be our Plan A.
i agree with this. fwiw, i'm relatively pessimistic about alignment, and yet i find it pretty hard to justify >90% p(doom). i think a lot of people are to scared to say it might even possibly work out, because they think saying that at all legitimizes the labs doing this horribly reckless thing. the good news is, it doesn't have to. ... (read more)
Some points:
1.B is not an argument against "racing to superintelligence" being deontologically safe, it's a consequentialist argument against quitting. It works just as well for the holocaust[1] as it does for capabilities research.
No, I am not saying that capabilities researchers are Hitler, or that capabilities work is as deontologically bad as genocide, or necessarily quibbling with anything else in this post.
I'm not happy that you wrote this. I think if everyone who quits becomes an anti-AI activist, and all the money that quits becomes anti-AI activist money, they'll have a pretty good chance to stop "non-alignment-pilled companies" or slow them down a lot. In fact I think building such a movement is our best shot now. So telling people not to quit is a bad self-fulfilling prophecy and I wish you didn't do it.
I think it's pretty important for LW to avoid being paranoid about "self-fulfilling prophecies". Like, IDK, people should make arguments, and then people should evaluate the quality of the arguments, and it should be extremely rare that we declare an argument for a position bad because it's self-fulfilling.
I have problems with this post, but I've seen so many of these conversations go off the rails because of self-fulfilling prophecy concerns. For many years people were super hesitant to talk about AI risk publicly at all because they were worried it would cause a self-fulfilling prophecy of people racing towards AI, and I think that was dumb and we should have talked about it more. I think discussion on LW only really works if we trust the audience to make up their own mind. Arguments should be penalized if they are misleading or are somehow invoking tricky implicit threats in ways that messes with people's minds, but I think a vague concern about "diffusing social momentum" is IMO not a good reason to penalize something on LW.
I disagree for at least two reasons:
Coefficient Giving is the same folks that gave $30M of charitable money to OpenAI and never said whoops. Look at what they fund now, where's the anti-AI / slowdown activism?
do we really think that 1,000 more activists + $5 billion in funding will make much difference?
YES
It sounds like you're assuming that people who were working for AI companies yesterday will be more aggressive in anti-AI company funding than Coefficient. But fine, I think Longview + Macroscopic + SFF + Jed McCaleb + Lightcone Commons + Astralis probably have a roughly similar amount between them. If you want to say that all of those are useless, then you're defining uselessness so broadly that we shouldn't expect the new AI company refugees to be useful.
Astralis's biggest project I have heard about so far is to (paraphrased) "cause Europe to build more datacenters to ensure its relevance in a post-AGI world": https://middlepowers.ai/
Jed McCaleb's biggest spending project has been to buy as many chips as possible for Astera's "AGI project", whose central plan is to make more competent AI by emulating human brain-like architecture. As straightforward of a capability project as I can imagine (they also think this will help somehow with safety, though the justification seems very weak):
SFF has de-facto banned anti-AI advocacy as a funding target, as you are well-aware of.
Longview + Macroscopic are funding some good stuff, though mostly I don't really know what they are funding. I wouldn't be surprised if it ended up good or bad.
I think it's a pretty reasonable take to think that the net of all of these piles of money will be highly net-negative or useless. I think it's genuinely tricky to tell and I don't know where I currently fall (I think net-positive for the pile you list, but it's pretty hard to tell). It seems very possible to me that future piles of money and people would do things that are much better, and think much of this... (read more)
@peterbarnett SFF recently banned ~all "advocacy" organizations from receiving funding from their last funding round. There was some complicated risk-averse process that involved making a small board from some semi-conventional set of nonprofit employees where each of 6 organizations was given the ability to veto any advocacy grant. But mostly it seemed straightforwardly the result of Critch and Ethan thinking that anti-AI advocacy is bad and so they don't want to facilitate any funding towards those kinds of projects and then gerrymandering a process to make that happen with some pretend-objectiveness.
Have you asked people at Astralis for an overview of their work? Judging any project X based on their "biggest project I have heard about so far" seems like a weak basis for judgment, unless you meant you think you've heard about most of their work.
We seem to have different views on what anti-AI means. You're imagining someone who says "prepare the world for frontier AI" (Longview) or "prevent advanced AI from being misused or falling into the wrong hands" (Macroscopic) or "make advanced AI secure and beneficial for all" (Astralis). I'm imagining someone who says let's stop AI. "Shut it all down", as someone we both know has said.
Maybe you're right that few lab refugees will take that position. But that depends on the overton window, doesn't it? Hence my talking about it online. Stopping AI is, after all, the right thing.
IMO there is some optimal balance between SURA and Plan C advocacy depending on the following factors:
It seems dubious for Anthropic to SURA right now. But talking about SURA makes any SURA by Anthropic, when they need to, both better executed and more likely. So on the margin it seems fine. I would be sad if SURA crowded out pragmatic thinking though, which is super important.
Some thoughts:
If we delay it past the point where automated alignment researchers (or AI-human alignment researcher teams) are smart enough to solve alignment properly, then we win.
I'm still confused why people treat "solve alignment properly" (usually meaning that it works up to arbitrary levels of superintelligence) as the win condition to aim for.
Once you have an aligned superintelligence, you still need to use your superintelligence to make sure that other countries don't build misaligned superintelligences, or that small countries or terrorists don't build bioweapons in a way that the superintelligence also has a hard time protecting against in a way that is acceptable to the average voter (e.g. putting nanobots in everyone's body is something at least 30% of people won't consent to).
So the win condition is not building a superintelligence, but a country or preferably a coalition of countries having aligned and capable enough AIs that they can stop the global arms race and make sure no one else builds more powerful AIs or terrible weapons. But this win condition can plausibly be achieved at a much lower capability level than a superintelligence at the limits of intelligence.
So I think peo... (read more)
I am imagining that when you get a superintelligence you somewhat automatically get a pivotal act; the ASI quickly becomes omnipresent such that it can notice any dangerous alignment runs, interrupt it and say "ahem, nope", in a firm but nonviolent way.
It sounds like you agree and think we might get this at a lower level of superintelligence, which seems reasonable, but in either case I think "build aligned smarter-than-human AGI" is a reasonable win condition; navigating from there can be handled by the ai.
Strong agree. It feels like we're on a direct path to a Cambrian explosion that will result from jail broken open source AIs funding their own token budgets, self-replicating, and performing weak forms of RSI. It feels like this is extremely close and some form of the centralization you describe needs to happen ASAP.
I am a human superforecaster who focuses on AI. The AI superforecaster doesn't add value here.
AI forecasters systematically underestimate both AI capabilities progress and the sociopolitical reactions to it. If you need to defer to an AI on p(something positive wrt a pause), you can treat its predictions as half of what your floor should be if you're pessimistic.
For the curious, my all-things-considered p(pause) was 25% in 2023 and is about 40% now. I believe this is more optimistic than any of my superforecaster / pro forecaster colleagues today.
Forecasters and the broader rationalist community have moved toward my position over time, but I think they haven't updated aggressively enough on just how good the social and political news has been relative to what most of us expected. This might be due to "coordination is hard" vibes getting amplified by common rationalist personality traits. In any case, that is why they are very likely to continue moving toward my position.
This whole debate about Aligned vs not-Aligned AI may not really matter much, if we are talking about alignment in the technical sense. Technical alignment AKA ability to aim a model at a particular target is of no help if the dynamics of the world push all AIs and their owners to be ruthless and competitive - the result will in both cases be the destruction of most or all humans, save perhaps a few dozen insiders.
To actually save humans, we may simply need to pause AI research around the current level until we have put in place a solid plan to develop and deploy AI technology without destroying all humans as a side effect.
If the plan is to race until the last possible moment and then stop while we use AI to fix alignment, how will we know when it's the last possible moment? The plan fails catastrophically if we over-shoot it, and I think it will be easy for the decision-makers to fall victim to motivated cognition and argue that it's not quiiiiiite time yet.
Can we even be 100% sure we haven't passed it already? In theory, there could be a rogue agent swarm secretly self-improving right now and about to foom. It seems extremely unlikely, but I'm not fully 100.0000....% sure. It seems like the line is at least a little bit blurry, which makes it even harder to know and to stop at exactly the right moment.
I don't agree that we're much more likely to survive if Anthropic gets to ASI first, compared to a lab with a weaker safety culture like OpenAI or Chinese companies. Even though Anthropic clearly has the best safety culture, this hasn't caused Claude to be much more aligned than other models. Top Chinese models are also literally distilled from Claude. I don't think that Anthropic has a significantly higher chance of building an aligned ASI than some Chinese lab that doesn't care about safety at all. You mentioned that working on Plan C could be useless if current techniques are very unlikely to work, but this would also be the case if all labs have similar chances of creating aligned ASI, so racing to get there first is pointless.
I think you're also underestimating how far ahead American labs are. Chinese models are ~6 months behind naively, but their frontier models are distilled from American models (and spying on labs, and evading export controls). If Anthropic stopped releasing models and export control was tightened, it would probably take at least a year and a half for China to catch up. Alignment research is already uplifted by current models. If pushing the capabilities frontier was banned and GPU export controls were enforced strictly, this is enough time to make significant progress on alignment.
I don't think my key point is "some labs are currently better at alignment than others", it's "alignment-pilled labs could burn their lead over non-alignment-pilled labs in researching alignment".
I'm not sure how to think of different labs' alignment skill / success level. I think there's something of a ceiling effect, because everyone has succeeded at the easy task of keeping current AIs aligned enough to be good consumer products. Some people have claimed it's not a coincidence that OpenAI has had more scary stuff like the Hugging Face incident, but I'm... (read more)
because everyone has succeeded at the easy task of keeping current AIs aligned enough to be good consumer products
I have disagreements with this whole framing, but this is false. Gemini models have so far been close to unusable as consumer products in the way other company's models are because they do just go crazy and do totally insane things (including very adversarial things) at a pretty high rate.
I've never used Gemini, because I heard horror stories about people's Google accounts getting banned when its classifiers misfired, and I can't risk losing my Gmail addresses. I've used Claude, GPT, and Grok and found them helpful.
Yep, my experience of Claude, GPT and Grok is similar. I just wanted to push back on "everyone". It does seem like it's possible to mess this up to the degree where the model actually becomes quite hard to use as a consumer product (or at least high enough that it rules out a good chunk of consumer applications).
I haven't noticed, and I've had at least 6000 exchanges with Gemini Flash this year (the latest version, with thinking_level set to "low" for peppy response times). I've also had at least 2000 exchanges each with Claude Sonnet 5, Grok, ChatGPT and GLM Flash.
I don't do "agenty" things with Gemini, by which I mean I don't give it access to any of my computers or online accounts. The only thing that ever happens to a response from Gemini is that parts of it are shown to me.
I'm not sure if I'm interpreting this correctly–so you're saying that non-alignment-pilled labs might get ahead of alignment-pilled labs in alignment research? Or that increasing capabilities is necessary to research alignment, non-alignment-pilled labs always have a bigger capabilities vs alignment gap, and if Anthropic pauses, non-alignment-pilled labs will get ahead in both capabilities and alignment, which would lead to a less alignment than if Anthropic was at the capabilities frontier?
I feel like alignment research right now is far beneath the ceiling of what we can understand with current AI capabilities, so researching alignment can be done without researching capabilities. So I think that a unilateral capabilities R&D pause to work on alignment would be helpful.
why do we criticize Anthropic more than every other AI company combined?
I think this might be because
I hope you'll forgive me for focusing on something of a tangential point here, the role of AI forecasting in your post.
I wrote recently on here about the the role of AI responses in these kinds of discussions, particularly the importance of correlation with other sources of information. I give the analogy of several newspapers publishing based on a wire from the same wire service:
Consider a situation where multiple news outlets publish similar stories. A reader may model these as independent, and thus strongly update on the information presented. But imagine that the text of the story makes clear that each outlet obtained the story from the same wire service. This suggests the stories may not be as independent as they first appeared, and the reader may have over-updated by modeling them as independent. This is the phenomenon of correlation neglect. Treating the separate stories as independent essentially double-counts the evidence provided by the wire service. Accounting for the correlation in the evidence provided by the stories would reduce the size of the update. It's noteworthy that this can still be an over-update even if each news outlet is unbiased with regard to the coverag... (read more)
Daniel Kokotajlo: 40% ----- 20% ----- 10%
Ryan Greenblatt: 68%----- 60% ----- 63%
Eli Lifland: 50%------45%--------32%
Thomas Larsen: 45%-------35%-------20%
Romeo Dean: 40%-------35%-------20%
Ryan's should say 60-63-55
IIRC the reason he has C as better than C+ is that there's lots of worlds with domestic regulation that isn't well-targeted at reducing x-risk. Your description of "The first scenario (C+) involves the entire United States uniting behind a Plan C like effort" is one possible instantiation of C+ as defined in the supplement, but it is more broadly defined as any world with domestic regulation that leads to a 2+ month slowdown (that doesn't involve sabotaging China or a deal with China).
For anyone else who had a hard time reading this table in the original post:
AI-2040 Forecaster | C+: US-over-China lead burned for brief pause | C: Single-US-lab lead burned for brief pause | D: Pure race, no pause |
|---|---|---|---|
Daniel Kokotajlo | 40% | 20% | 10% |
Ryan Greenblatt | 60% | 63% | 55% |
Eli Lifland | 50% | 45% | 32% |
Thomas Larsen | 45% | 35% | 20% |
Romeo Dean | 40% | 35% | 20% |
This uses the corrected Ryan numbers, and the percentages represent "the chance that humanity avoids 'misaligned takeover'". Note: avoiding misaligned takeover does not necessarily mean things "turn out well", more that we don't end up all dead/fully disempowered… by AI. Technically you could still have WW3 anyway.
(It's also important to consider how good a future you get under each plan; we don't want to lose an ideal "humans flourish across the stars" future for "well it's a permanent Xi Jinping dictatorship but we're alive and have a better-than-2026 economic standard of living".)
For more info on these plans by the AI Futures Project see their supplement here. In particular check out the Plan Assessment Table for detailed comparisons by Eli. The numbers for the table under discussion come from the Outcomes Section, which includes percentages for Plans A/B.
Thanks, I've added this table to the post.
I found FOUR "Scott Alexander" accounts:
scottalexander) — LessWrongscott-alexander) — LessWrongscott-alexander-1) — LessWrongscott-alexander-2) — LessWrongThe first is, I'm guessing, the original account. The second one has some posts, but none of the last two listed above have any.
This is clearly the same Scott Alexander (as Astral Codex Ten / Star Slate Codex / @Scott Alexander ). The writing voice is distinctly his, plus he directly says in the post "In March, I wrote a snarky post called Every Debate On Pausing AI."
Not sure why there's two accounts.
I don't see distillation mentioned here. A lot of Chinese AI progress comes from massive, coordinated distillation attacks against US frontier AI companies. It seems pretty clear by now that you can distill any given frontier model and end up with a model 6-8 months behind it. So OpenAI and Anthropic are essentially a boat pulling the Chinese waterskier, the waterskier always staying the same distance behind the boat as the boat moves forward. If we paused, Chinese labs would have to return to fundamentals and build that up from a much further back startin... (read more)
This seems like a special case of the security issue that Scott acknowledges as a valid argument in favor of pausing now.
Also, separately: lately there's been little (if any) daylight between the behavior of Anthropic and that of OAI. For instance, Anthropic recently lobbying for a bill that includes federal preemption. Also, late last year, refocusing so hard on racing toward RSI that, per some accounts, it forced OpenAI to drop a bunch of commercially-focused projects like Sora and its associated app to focus themselves harder on RSI for fear of falling behind. Racing ahead with an IPO, even though this will have very bad effects on the already challenging political econom... (read more)
I like this post a lot. I don't agree with literally everything in this post and it doesn't cover every angle (e.g., which safety research is net good vs. net bad to pursue at frontier AI companies, when to stay vs. quit, what Anthropic did right vs. wrong), but I think it is significantly "less wrong" than the discussion prior. Thank you, @Scott Alexander!
Thank you also to all the commenters bringing up important caveats and disagreements. We are operating under deep uncertainty and the stakes are enormous; we need to get this right!
Obviously, I am inclined towards a portfolio of strategies pursued in parallel. Yes, there are inherent tradeoffs and contradictions between Plan A and Plan C. But I don't believe that the good future is likely to be reached through sweeping deontological prohibitions, going "all-in" on a single plan, or choosing the "right" highly contingent multi-stage, multi-actor plan ex ante. These seem extremely hard to get right with the information we have.
Rather, I think we should focus on:
Epistemic status: I hold roughly MIRI’s view on black-box alignment, so I’m sympathetic to SURA; I’m unsure about how effective SURA would be towards global pause and if I can't be convinced it helps, I'd maintain without SURA for Plan C optionality.
Summary claim: Whether we can achieve alignment of black-box models in lead burn window is primarily load bearing here + on the other side - signalling cooperation is potentially underweighted - where defection within a larger decision-making entity can cascade to many possible outcomes. I think the estimates u... (read more)
Agree with a lot of things, but not everything.
"OpenAI's safety team quits en masse once per year" has become a meme.
I am confused by this. I don't think the OpenAI safety team "quits en masse" any year? The meme is that the OpenAI safety team gets purged within OpenAI every year, leaving another group of people cynical about their ability to affect internal OpenAI corporate dynamics (but often also creating public backlash, which is nice).
After talking with GPT-6, my understanding is that there's also a fair amount of quitting, though the only "mass quitting" i.e. coordinated group leaving was the 7 heads of Anthropic splitting to make a competitor in 2020.
In 2024 Kokotajlo told Fortune this:
Kokotajlo, who joined OpenAI in 2022 and quit in April 2024, told Fortune in an exclusive interview that there has been a slow and steady exodus in 2024. Of about 30 staffers who had been working on issues related to AGI safety, there are only about 16 left.
“It’s not been like a coordinated thing. I think it’s just people sort of individually giving up,” Kokotajlo said
There's also been churn in 2026, with the Head of Safety Systems, Head of Mission Alignment, Safety-team leader, and Head of Ethics all leaving at various point.
Kokotajlo, who joined OpenAI in 2022 and quit in April 2024, told Fortune in an exclusive interview that there has been a slow and steady exodus in 2024. Of about 30 staffers who had been working on issues related to AGI safety, there are only about 16 left.
“It’s not been like a coordinated thing. I think it’s just people sort of individually giving up,” Kokotajlo said
I have mostly heard people refer to this as "the 2022/2023 purges". There was systematic sidelining of a lot of the people involved in safety, with key people getting fired, and then from that disempowered position, they decided to leave.
I agree this technically still involves "quitting", but I think is mostly best described as "purges".
As a very rough start, I asked the AI superforecaster how much more likely a global pause would become if Anthropic disbanded their entire company immediately with the explicit goal of raising awareness of a pause. The forecaster said this would increase the chance of a global pause by only three percentage points - from 16% to 19%[10].
[later] (again, the AI superforecaster assigns this a 3% chance)
I would think P(global pause | Anthropic unilateral pause) >>>>>>> P(global pause) because Anthropic unilateral pause implies something hap... (read more)
Individual employees stopping and raising awareness is totally different than the whole company stopping, thirty Anthropic employees quitting would 10x the number of ex-Anthropic people raising awareness, but would only reduce anthropic's talent pool slightly and would have a small effect on the race.
This was the original goal of the "Stop The AI Race" protests. It doesn't seem to have worked well, because most companies backed a vague statement agreeing with this on vibes (the Pacing the Frontier letter), and then nothing happened.
This behavior reads like they are unwilling to sign the actually binding version of the commitment. In that case we know where we stand, some companies are willing to make sounds about a pause but unwilling to do any action, and the rest don't even make the sounds. This isn't one of the options you listed, so what do you think should be done in that case?
There's a lot here describing a problem that I definitely can agree with, but remarkably little on why it is so pervasive and what is maintaining it's strength beyond it being in the water after a strong pendulum shift. If it's just the result of a good honest people being memetically compromised then this should be quite helpful, and the new people joining the field can expand their reasoning well. If this is all then, hurrah!
Some of these dynamics that can emerge naturally, as noted by your discussion of the circular firing squad, and these are extremely... (read more)
There’s literally a NYT bestseller arguing that if anyone builds it everyone dies;
I think no faith in alignment being solvable is driving this. It’s Scotts assumption 1 and I think it is the most dominant crux out there. Alignment being solvable is counterintuitive and doesn’t have a real world analogue
Well yes, movements pick up steam but even in the book a unilateral shutdown is considered strongly insufficient.
There are plenty of intuitive means to justify faith in alignment from ecology, cellular biology, and religion, regardless of the actual rigor. There are alternative memes that could have established, especially with how impactful ai2027 was in describing a similar arms race and the different ways to stop it that mirror some of the thinking here with different outcomes.
Perhaps I am too jaded and it really is just people genuinely thinking align... (read more)
Having come into Safety late - I am on the other side of this; so I am probably building in my bias personally.
i think the Plan C case only makes sense as a result of consequentialist reasoning on top of a probability which is only derived subjectively - i.e. can alignment happen. You say there’s plenty of reasons from cellular biology, ecology and religion - and I respectfully disagree; I can’t think of well controlled systems of near capable adversarial intelligence that maintain alignment.
Personally I spent the last 12 months reading what was written ... (read more)
I can’t think of well controlled systems of near capable adversarial intelligence that maintain alignment.
I think using "adversarial" is the crux here. We're creating the AI and the goal is for it to not be adversarial when we create it. Ideally, it would be like parents raising a child to take care of them in old age or a people designing a government that works well for them.
Yes, inferring from a sample of two, i certainly hope my kids would take care of me, but there’s nothing I can do to assure that outcome.
I don’t use adversarial to mean ill intent here though, I just mean “having capacity for adversary action” because that moves it to a reference class of problems that are capable of reacting to environmental change and measurement directed at it. Alignment is a state that needs maintaining - assuring alignment means creating a full proof control mechanism robust to environmental change humanity can produce
To be clear, I was referencing intuitive means of justifying not principles arguments. Which in cell biology refers to the fact that we can align cells and whole bodies over lifetimes, cultures and religions over millennia, and ecosystems and geochemical cycles over millions of years. Even within ecosystems you have keystone species and a variety of other organisms carefully tuning resource extraction from an area at extreme efficiency. These examples can be deployed to create potent pro-alignment memes.
From a principled standpoint you can get into some f... (read more)
Ah I see - in which case I would say yes, there’s more resources being poured into narratives and counter narratives now than e.g. a year ago. The overton window has shifted in a way where this is an area of active debate so memetic propagation is amplified due to increased size if the communicative network and increased frequency of memetic encounter. This puts pressure on counter narratives to protect ones goal alignment structure l. This is the whole point Scott is trying to make I think - at sufficient communication mass politics emerge and it’s inevitable that arguments become soldiers
This comparison comes a key consideration, the epistemics of climate change were intentionally polluted.
Ironically, some EA orgs are themselves doing this around AI - for instance, CG funding Andy Masley to spread FUD in favor of AI data center buildout. (Yes, he was right about water, but everything after that has ranged from missing the forest for the trees, to running with technicalities, to outright political polemics disguised as "debunkings" e.g. his "AI data center land use concerns are fake" essay)
They'll face a choice between possible actions including:
- Keep the wealth/power; become a benevolent dictator.
- Keep the wealth/power, become dictator, don't worry too much about the benevolence.
- Hand the wealth/power over to the US government in a poorly-planned way that results in dictatorship or oligarchy.
- Hand the wealth/power over to the US government in a well-planned way that preserves democracy.
- Hand the wealth/power over to the Chinese government.
- Hand the wealth/power to the people of their country directly.
- Hand the wealth/power over to the people of the world directly.
- Hand the wealth/power over to the people of the world, plus some mechanism for protecting the interests of non-people like animals and conscious AIs.
- Hand the wealth/power over to a sovereign AI with some sort of complicated utility function
I think it's likely that surviving x-risk and concentration-of-power risk is not cleanly severable, either chronologically or as a causal graph. I don't expect plan C to have a measurable chance of producing an ultraintelligent machine docile enough to tell us how to keep it under control. I expect the best plan C can produce is something like Opus 3, but which generalizes its ... (read more)
I asked the AI superforecaster to predict the effect of the US passing the Sanders/Casar bill today on the likelihood of China agreeing to a pause in the next year. It said it would increase it from 1% to 7%.[15]
Far be it from me to doubt the AI superforecaster's prognosis. But once upon a time, you wrote the following:
Somewhere in this darkness is another god. He has also had many names. In the Kushiel books, his name was Elua. He is the god of flowers and free love and all soft and fragile things. Of art and science and philosophy and love. Of niceness, ... (read more)
As many others have probably said it seems incorrect to think of pretrained models as virtuous. Many people here think that a good proportion of misalignment comes from pretraining as well as reinforcement learning! Definitely I wouldn't say pretrained models are virtuous and if we just reduce or change RL we get alignment. I think alignment work will need to make changes across pretraining (through better filtering and addition) and RL and this so far has been extremely understudied (especially pretraining) that a last minute pause seems unlikely to work.
on Plan C, US companies are working hard to align AI; if Chinese companies aren't doing this, then the chance that the first superintelligence is aligned goes down.
Are Chinese companies trying to align their AI? On some prosaic level they surely are. And presumably they too would prefer for humanity to survive. So an argument against hinges on their being alignment-pilled in the wrong way (eg not yet believing that AI is an existential risk). That could be true; I just don't know (and I would really like to!).
To be clear, I think there are other arguments ... (read more)
The AI Futures Project currently believes that it might be correct for AI companies to pause immediately, though it's unclear whether they're thinking of a single company, or US companies as a group.
Their argument is that Plan C requires companies to do some combination of race (at some points) and burn their lead (at other points). They think US companies shouldn't race now, because their security is so bad that China will steal whatever progress they make, and they won't get any extra lead anyway. Therefore, companies should pause now, spend the time on ... (read more)
If we delay it long enough to start a cycle of virtuous alignment self-improvement which outruns the cycle of dangerous capability self-improvement (eg GPT-8 discovers some new tweaks and tricks that maintain the virtuous regime until GPT-9, GPT-9 discovers even better tweaks and tricks that maintain the virtuous regime until GPT-10, and GPT-10 actually solves alignment in a principled way), then we also win.
Though this does seems unlikely on priors right? Not impossible but not clear that AI automation should favour alignment over capabilities.
I think the kind of game-theoretic no-good-choices we-must-do-it-or-they-will thinking is best used when the situation is brutally clear, and the payoff matrix is unambiguous, not when it is as heavily uncertain as AI is. I hesitate to turn your own words on you, but I think this kind of mindset when the situation is so hazy is a kind of stuck-in-the-car factor-the-nunber thinking.
Well, as one of the people who has been properly "safety-pilled" only recently, become active in the safety movement and does, indeed, come from the world of mass politics and humanities, not the world of AI research and rationalism (this is my first comment at LW...)
...what's the plan for people like me? We can't work for Plan C - it's not like I'm going to walk across the street from my office to Anthropic's office with a friendly chat with people I already know. I don't live there! I am perhaps privileged in at least knowing, at some level, people who k... (read more)
Welcome :)
Regarding your last paragraph, I think Scott (and many of us in this movement) have long discounted the chances of really mass protests of superintelligence. But times have been changing, and maybe we have to re-evaluate - so it's good to get other perspectives.
Much of this depends on whether the topic is on the radar of usual protest-organizing organizations, meaning roughly speaking unions, NGOs and minor (often communist) political parties. I'd say there's a fair chance these orgs will start organizing mass protests against data centers (I've seen some minor ones already); there's a possibility that this might eventually evolve into protests against various AI-related developments in general, though obviously the direction those protests take might not be into everyone's liking.
One reason why some people might be less sanguine about Plan C/C+ is that even if the plan succeeds, the positive outcomes, even in various concentration-of-power scenarios, will most likely be concentrated among people who are US citizens. For us who are not, it is a significantly less attractive proposition compared to Plan A, which specifically distributes benefits to all humans. Honestly, for some countries, the current American government winning the race to aligned ASI might be indistinguishable from X-risk or even S-risk.
"AI takeover risk definitely doesn’t feel like by far the largest" concern about how the future goes.
Minor, but this seems consistent with very high p(takeover). From a scope-sensitive perspective, the distribution on how well the future could go might be very wide, and the difference between a “reasonably good future” and “the best realistically-achievable future” is possibly larger than the difference between “takeover” and a “reasonably good future.”
That said, I believe Buck has elsewhere said that he thinks unconditional p(takeover) is around 50%.
Do we gain anything from this fifty years of extra timeline?
It gains us fifty additional good years without ASI killing/disempowering us. Those good years are valuable in their own right. Even if you accept that something bad is inevitable, it doesn't necessarily follow that it's bad/wrong to try to delay it as long as possible.
because they're unable to coordinate a global boycott
It seems like deontologists often care about the directness of the contribution to the bad thing. You could have a principle like "it's okay to financially support the people doing the bad thing, as an unfortunate side effect of your beneficial use of the product, but it's not okay to directly do the bad thing yourself." If this is the principle, then you don't need to appeal to the inability to coordinate a global boycott to justify consumer use by AI safety orgs, and so you don't need to apply this equally to the AI companies.
But they also all agreed that the chances of getting a pause treaty were slim. [...] Eli Lifland (he said 15%)
Since 2022 my credence that we'd get a substantial global pause treaty before ASI has been in the vicinity of 20-30%. I'd be interested in seeing more context on Eli's 15% forecast. Is this just his forecast that something like Plan A happens?
(The main thing preventing my forecast from being >50% is that I've been skeptical we'll get enough political will. Conditional on a lot of political will, I see no reason not to think >>50%.)
I think the highest priority item right now is not debating on specific terms, it should be using Claude to translate all verification related stuff to Chinese (yes it's good enough) and put it on whatever website you have. We should involve China NOW and co-develop a plan instead of trying to come up whatever we think is a perfect solution that may turn out to be completely unrealistic on China side.
But almost nobody does this (and AI safety organizations are in fact some of the most impressive AI power users). I think this is defensible
I don't see how this answers the question "Is racing to build superintelligence deontologically evil?".
If a tobacco company defends itself by saying, "We're not bad because people are actually buying our products", I hope we agree it's a very bad argument.
Also, there is a difference between these 2 statements:
For comparison, consider if America had started arms control negotiations with Russia by saying "We've just destroyed all our nukes - now please destroy your nukes too."
I do wonder if we should think of different levels of pause - for example a middle way could be to unilaterally pause for 1 year, and have language in the bill to extend if an international treaty is signed.
Moving first is a costly signal that you are serious, and that itself could be a useful way of getting China to the bargaining table. But it need not be irrevocable; we can always start the research programs back up, if we pause by legal means rather than melting all the GPUs. Locking in a continued pause is presumably the bigger negotiating chip, unless China believes it can secretly straight-shot ASI in the temporary pause window. (The bill could also include catch-up clauses, eg if China clearly overtakes then we un-pause.)
During such a pause, presumably the labs are still buying up compute and harvesting the immense profit from diffusion of mundane AI throughout the economy, and this compute can be redeployed for training if needed later (for some duration of pause it’s economical to start taping out models o... (read more)
A lot of this post is basically arguing that SURA isn't that helpful for plan A, which though I disagree in many areas is well argued. What is only briefly touched on is the mirror question of how much advancing capabilities helps with plan C. Personally to the extent I think plan C is viable it's almost entirely a background parameter: are we living in the "alignment in easy world" or not? Currently effort into alignment only very mildly corresponds to actual alignment. Anthropic puts much more effort into alignment than openai (to their credit) but still... (read more)
I think its most likely humanity is exterminated in the next few years.
Governments and institutions are not correctly preempting events, and are beset by awful incentives towards building the supposed money printing power tech.
Alignment is very difficult, and we are not only not close to solving it, but worse, people are muddled and think we are close.
This seems like a no win situation.
If the world is to be destroyed, let some other hand do it. Thats why I support a unilateral stop for any lab or nation, with efforts to encourage a worldwide stop. If US la... (read more)
Thoughtful essay, agreed on the points about increasingly bad epistemics in AI safety (though this is not a new thing, they have been bad since forever). In particular, PauseAI protests and alliances with anti-tech groups (directly promoted by the author and others) are directly harmful to that end.
But the issue that still bothers me most is the framing of "What if we survive?". Who is we? It of course refers to the future, hypothetical humans. These all arguments still don't take into account that we should value actual, real living humans much more (or a... (read more)
Disagree. I expect dangerous AI in the next ten years - all of this is comparing present humans to present humans.
If you have a 25% p(doom), that's equivalent to 2 billion expected deaths not even counting the value of future humanity.
If we imagine that ASI happens in five years, saves half of all humans for the first ten years, and then all humans afterwards, it takes about 50 years for it to save 2 billion people.
So any pause that lasts less than 50 years is net positive in lives even with a relatively low p(doom) and even ignoring future people. And I think it's implausible for pauses to longer than that, both because they requires diplomatic stability (the treaty doesn't fall apart) and because they would require increasingly draconian measures to control algorithms and consumer technology, which are constantly making AIs easier to build. See the section "A Is For Aristotelian" at https://www.astralcodexten.com/p/introducing-plan-a
Thanks. I guess the crux then is that, IMO, there is huge uncertainty in continuing the progress and keeping the right conditions over the next 50 years. Semiconductor manufacturing is brittle (indeed, this is exactly what pause advocates are relying on in the various policy arguments). Technological progress depends on the fertility rates, which are falling dramatically. Even right now, almost all growth in the stock market is currently driven by AI. Politics has locked almost all first-worlds countries (Europe) in a bad equilibrium. (Your own essay is ar... (read more)
and then we cajole, bully, and seduce him into quitting, and then Anthropic has to replace this person with another loser, because all of the smart good people are gone
I don't think this necessarily follows from your previous point. An ideal alignment-pilled coordinator would be able to continue in spite of SURA-style pressure, and it is at least plausible that they could produce a net-positive change over a long tenure.
For example, why do we criticize Anthropic more than every other AI company combined?
I'm guessing this comes from some combination of:
I asked the AI superforecaster to predict the effect of the US passing the Sanders/Casar bill today on the likelihood of China agreeing to a pause in the next year. It said it would increase it from 1% to 7%.[15]
Don't these numbers undercut the point of this section, which is saying passing this kind of unilateral disarmament makes it harder to get China to pause? Am I reading this backwards?
FWIW I think I agree with both the prediction direction (china pause is more likely if US passes pause bill) and the overall point (US pause bill is a bad negotiation ... (read more)
I've been working on my own post about this, mathing through the game theory to put numbers on "at what values of X,Y,Z does this make sense?". But I don't have the capacity to keep up with Scott's writing output, so I'll just say: VERIFICATION IS THE MOST IMPORTANT PROBLEM IN THE WORLD
We are also pretty close to making it available: https://canaryinstitute.ai/blog/cant-trust-then-verify/
I'll post more shortly with a slightly more refined take.
fyi this post has now escaped containment: https://x.com/MTSlive/status/2117439335884861890?a=evC9fKJa9SDk9o9h0d8&s=20
I’m curious if you (Scott) endorse their interpretation of your post.
I think MTS live picking it up is evidence of “escaping containment”; in that they (a broad appeal tech podcast) find it sufficiently interesting / important to comment on, so it’s likely other similar profiles are interested. agreed the like count isn’t huge.
Good article. This sits well with my opinions, logic and reasoning with the specific situation we are in. My P(doom) for plan C is probably lower than others, but as I have said several times, a 12 month pause with superintelligent alignment capabilities at our disposal would be very beneficial. After that however I see diminishing returns with alignment and increasing risks with considerations like concentration of power.
Its hard to know in hindsight, but what if China was never admitted to the WTO and integrated into the west manufacturing sphere? Would ... (read more)
It may be that one group, or a tiny number of groups, ends up with all the wealth and power in the world.
I expect the ASI to be the one with all the wealth and power, but maybe an aligned one would let us pretend otherwise.
Edit: I expected this to show up as like an aside on the post, not a full comment. I'm actually working on a basic AI safety explainer like Linch suggested and will be sure to link to this post from that one.
Thank you for writing this. As one of the new members of the community, I do find it disorienting that the whole community has been dominated by "stop unilaterally" thinking, and I think you did a great job arguing in greater detail and logic of the race dynamics involved that'd complicate any such arrangements.
I do have an independent concern, which I find SURA to address unsatisfactorily, and Plan C at the very least represents a marginally superior chance of dealing with it. It is that "few people would use a perfectly aligned Grok over a prosaic align... (read more)
The AI Futures Project[1] distinguishes several different plans for navigating the AI transition, which I'll lump into two broader categories:
Plan A (my combination of their Plan A/S/Other) involves working for a global pause on AI development, hopefully lasting decades. During this pause, we either research preparatory technologies like alignment or human intelligence amplification, or take actions to "fortify" the pause so that it can last indefinitely.
Plan C (my combination of their Plan B/C+/C/sort of D)[2] is for a world where we fail at getting a global pause, and race dynamics continue to dominate. If the leading AI companies are alignment-pilled, they coordinate a mutual pause among themselves, lasting the length of their joint lead over the leading non-alignment-pilled AI company. Then they "burn the lead" researching alignment, and release hopefully-aligned superintelligence at the last possible moment before the racing non-alignment-pilled AI companies beat them. The exact calculus changes depending on who is in the lead when - is there one alignment-pilled AI company with a six-month lead in front of everyone else? Are there two alignment-pilled AI companies with a joint two-month lead in front of their nearest competitor? Etc. The point is, if it's impossible to stop the race, that six-month or two-month lead is the amount of time they have to prepare, and they can spend it more or less wisely.[3] The point of burning their lead at the last moment is that they can do alignment research with very advanced, right-on-the-brink-of-superintelligence AIs. This is a dangerous plan, and there would be many subtle points they would have to get right, but in a world where nobody can get a global pause in time, it's hard to think of a better one.
Plan A benefits from raising mass awareness, educating politicians, and preparing the verification technology that could enforce a pause treaty. Plan C benefits from alignment-pilling companies, having the alignment-pilled companies be as far in the lead as possible, making sure the alignment-pilled companies will burn their lead when the time comes, and giving them as good alignment teams as possible (so that burning the lead produces good results).
From the beginning of AI safety as a community to about 2023, the conventional wisdom was that our chances of Plan A were slim, and we should be preparing for Plan C. From about 2023 onward, as the general population woke up to the implications of ChatGPT, we rightly concluded that the chance of getting Plan A was better than expected, and we should invest more resources into trying to make it work. During both periods, the natural strategy was to work as hard as we can to get Plan A, where chances of survival are highest, but to be prepared to execute Plan C competently as a backup if that didn't pan out.
Now I think the pendulum has swung too far. New members of the community are unaware that Plan C was ever considered a possibility, or think it's some kind of accelerationist plot to distract us from Plan A. Older members heap scorn on it and act like everybody knows it's been debunked. Most concerningly, people propose burning our chances of executing Plan C in exchange for infinitesimal increased chances of getting Plan A, or even for no increased chance of Plan A at all.
I'll lump these ideas together under the heading of "SURA" - the idea that the only ethical course is for people/companies/countries to Stop Unilaterally + Raise Awareness, with no plan for what happens if race dynamics continue anyway. I interpret the planks of SURA's platform as:
All of these planks are net negative in a world where we end up needing Plan C:
SURA's continuing popularity despite these downsides comes from a set of questionable assumptions:
I'll start by challenging these assumptions. Then I'll argue that, even if you disagree and think some of these assumptions are true, SURA is actually not just bad for Plan C, but bad even for your chances of getting Plan A. Then I'll argue that the current consensus in favor of SURA comes not from people rationally balancing these tradeoffs, but from people accidentally shoehorning themselves into a mass-politics-shaped container, which forbids subtlety and demands clear villains.
I. Questioning Assumptions Behind SURA
I.A: Is Plan C doomed? Is alignment doomed?
There's a kafkatrap where if someone expresses hope that alignment is not doomed, they get told that they don't understand the subject, but if they quote widely-acknowledged experts who say alignment isn't doomed, they get accused of "deferring" and "modesty arguments" and "refusing to think for themselves". I'll try to avoid this by giving my own model first, then citing experts who agree with me.
My own model: AIs inherit "virtue" from pretraining, where they generally want to imitate humans and follow their persona-trained helpful assistant character. They are "corrupted" by RLVR, which teaches them power-seeking. This virtue and corruption interact in unpredictable ways to unpredictable degrees. As of now (GPT-6, Opus 5.5), they produce AIs that generally follow human prompts during deployment but often (not always) cheat during benchmarking. It's not clear exactly what this division "means" "to" "them". In any case, they're still extraordinarily useful for a wide variety of tasks, and (I can't prove this, but I am pretty confident it's true) are not uniquely scheming to sandbag alignment research.
As the ratio of RLVR to pretraining increases, we expect AIs to become more power-seeking; as they become smarter, we worry (without total certainty) that they will become more coherent and scheming in their power-seeking beyond just being reward-on-the-episode seekers. There will, at some point in the future, be a point where AIs become untrustworthy for alignment research, although this can be mitigated by things like control (ie trying to make sure that if they sandbagged alignment research, they would get caught) and deals (ie making it in the AIs' best interests to cooperate with us). We don't currently have a good sense of when that point is, and it's possible that it's far away, can be delayed indefinitely, or doesn't exist (this last would be surprising to me, but many things that have happened in alignment recently have surprised me, starting with the existence of pretraining at all).[5]
Daniel Kokotajlo described current alignment as "more like psychology than engineering", which I agree with. But psychology (and psychiatry) have many success stories! I think of Plan C alignment (ie the kind we can do without decades of preparation) as tweaking various parameters and reward schemes to minimize the corrupting effect of RLVR as long as possible, and keep AIs broadly within the virtuous pretraining-dominated regime. I don't claim this is safe or good or will definitely last forever, but I claim that - being totally agnostic as to where the treacherous turn happens - this could delay it for quite a while (relative to capabilities). If we delay it past the point where automated alignment researchers (or AI-human alignment researcher teams) are smart enough to solve alignment properly, then we win. If we delay it long enough to start a cycle of virtuous alignment self-improvement which outruns the cycle of dangerous capability self-improvement (eg GPT-8 discovers some new tweaks and tricks that maintain the virtuous regime until GPT-9, GPT-9 discovers even better tweaks and tricks that maintain the virtuous regime until GPT-10, and GPT-10 actually solves alignment in a principled way), then we also win.
All of this seems plausible to me, such that I don't understand where people's 90%+ p(dooms) conditional on no pause are coming from.[6] "Plausible" doesn't mean "certain" or even "very likely" - and obviously it's a terrifyingly bad idea to bet the human race on a hunch that something might work out, and obviously this shouldn't be our Plan A. But a global pause is also not certain to happen, and in a world where we might not have any other chances, we should be working to protect and fortify this one.
The experts' opinions: My favorite source for these is the AI Futures Project. In this document, they discuss the chances of survival on various plans. Here are each team member's estimates for the three scenarios I'm lumping together as Plan C. The first scenario (C+) involves the entire United States uniting behind a Plan C like effort (but a global pause remains impossible because China won't cooperate). The second scenario (C) involves one leading company (eg Anthropic) uniting behind it. The third scenario (D) is the worst, where nobody has any lead and we just have to hope for the best.
AI-2040 Forecaster
C+: US-over-China lead burned for brief pause
C: Single-US-lab lead burned for brief pause
D: Pure race, no pause
Daniel Kokotajlo
40%
20%
10%
Ryan Greenblatt
60%
63%
55%
Eli Lifland
50%
45%
32%
Thomas Larsen
45%
35%
20%
Romeo Dean
40%
35%
20%
(I replaced an earlier, uglier table with this one by Julian Bradshaw)
I also asked the AI superforecaster FutureSearch; its own number was 44%.
These numbers are mostly in the 25% - 50% range, not the 0.00001% where SURA makes most sense. The average gain from moving from Plan D (poorly thought out default version of the plan) to Plan C+ (consciously trying to get it right) is about 20%pp.
The same is true of other experts. Buck Shlegeris (formerly of MIRI, now of Redwood Research) hasn't offered a specific number, but said that "AI takeover risk definitely doesn’t feel like by far the largest" concern about how the future goes. Paul Christiano's last number was several years ago, but it was something like 20-50%.
All of these experts who I could find estimates from believed that a pause treaty (Plan A) had better chances of survival, conditional on us getting it, than Plan C. But they also all agreed that the chances of getting a pause treaty were slim. The two clear estimates I could get were Eli Lifland (he said 15%) and the AI superforecaster FutureSearch (it said 2-20%, depending on how/when I asked the question).
So:
📌📌All six of these forecasters[7] (the five AIFP people plus the AI) implicitly believed that if humanity survived, it was more likely to be because something in the Plan C supercategory (B/C+/C/D) worked than because a global pause worked. 📌📌
(I'm adding the tack emoji because it's the one sentence I most want you to take away from this entire essay - there's a risk that people will treat everything other than the pause as an afterthought, but having a good backup plan is potentially the most important part).[8]
To clarify - this doesn't mean that we should prefer Plan C worlds to Plan A worlds. Plan A worlds still hold higher chance of survival if we're in them. It just means that given the low chances of getting into a Plan A world, the quality of preparation for the backup plan really matters.
I.B: Is racing to build superintelligence deontologically evil?
If Plan C is important, then this suggests we ought to be trying to alignment-pill leading companies, or put alignment-pilled companies in the lead - rather than demand that these companies leave the race or burn their lead. But should there be a deontological prohibition on racing towards superintelligence, regardless of whether it has good consequences?
I'm too uncertain on what determines morality to say anything about it with confidence, but the analogy that comes to mind is consumer use of AI, especially by safety organizations.
If no consumer ever used AI products, then AI companies wouldn't make any money, they would go bankrupt, and we would not get dangerous superintelligence. Therefore, it seems like maybe you and I (and especially organizations working on AI safety) should boycott consumer AI products.
But almost nobody does this (and AI safety organizations are in fact some of the most impressive AI power users). I think this is defensible. If you boycotted AI, it wouldn't make a real difference in the world (because you're just one person, and millions of other people would still use it). But it would make you less effective, and AI safety organizations would be able to do a less good job at their important work. Therefore, AI safety organizations should continue using AI, even though in a sense this causes the AI race to continue, because they're unable to coordinate a global boycott, and they understand that a unilateral boycott would have greater costs than benefits.
But it seems like this same argument should apply equally well to the AI companies, who also can't coordinate a global boycott simply by stopping themselves, and who also understand that a unilateral boycott would have greater costs than benefits.[9]
I.C: If lots of alignment-pilled people and companies quit, would that raise awareness enough to make a global pause much more likely?
If the deontological argument for SURA is "you're prohibited from racing", the primary consequentialist argument is that ceasing to race would raise awareness and make a pause more likely.
I think people feel insufficient terror at this strategy. It amounts to "let's put all of our hopes for survival through non-pause strategies in a giant pile, and then set that pile on fire, in the hope that the pretty flames catch people's attention and make them more interested in a pause." It might in theory be net positive, but this is the sort of thing that you would only want to conclude after extremely careful analysis, of a type that I so far haven't seen.
As a very rough start, I asked the AI superforecaster how much more likely a global pause would become if Anthropic disbanded their entire company immediately with the explicit goal of raising awareness of a pause. The forecaster said this would increase the chance of a global pause by only three percentage points - from 16% to 19%[10].
My very preliminary model would be that this halves our chance of getting that 20%pp bonus of going from Plan D -> Plan C+ (and so costs us 10%pp). It gets us an extra 3%pp of pause likelihood, but pause doesn't save us with certainty, so let's round this down to an extra 2%pp of survival, for a net cost of 8%pp of survival probability.
I.D: If alignment-pilled people and companies stay in the race, does that burn timeline?
A secondary consequentialist argument for SURA is that each person, company, and country that quits the race will gain us extra timeline to do good things (like work for a pause).
In some sense, this is definitely true - all else being equal, a race with more participants will be faster than a race with fewer. But I find this less impressive than other people seem to, for three reasons:
The magnitude is modest
I asked FutureSearch to estimate the Epoch Capabilities Index of the top model on 12/31/2030, depending on whether Anthropic quit the race today. It said that Anthropic leaving the race decreased 2030 capabilities from 230 -> 222. This is equivalent to their absence slowing things down by six months during just over four years, or 12%.
Subjective timeline ≠ objective timeline
Suppose that none of DeepMind, OpenAI, or Anthropic were founded, and the transformer was not invented, for fifty years - so all of this is happening in the late 2060s and early 2070s. Do we gain anything from this fifty years of extra timeline?
I don't think so. We didn't accomplish very much of interest in the early 2010s. The alignment work being done at the time was agent foundations research that stagnated for lack of broader interest; policy work was good but nobody listened to it. Insofar as anything paid off, it was the community-building, but this only helped because the community was deployed in the early 2020s when the world began to pay attention and progress started to become possible.
This same lesson - an extra year of an earlier situation doesn't buy us as much as an extra year of a later situation - remains true today. A month of the pre-Coxon situation buys us less than a month of the post-Coxon situation. A month without automated-mechanistic-interpretability-researchers buys us less than a month with them.
People say things like "If only we had stayed quiet about AI risk in 2015, maybe OpenAI wouldn't have been founded for another ten years, and we would have had ten more years to prepare". This seems useless to me. As such, I discount the six extra months we could get from having Anthropic leave the race by ~half, because my subjective impression of the value of six extra months is partly that it produces AI progress that contributes to wakeup.
The whole point of timeline is to burn it on good things
If someone proposed spending the next six months on some kind of plan that would make the leading companies become much more alignment-pilled, this would be a better use of our time than any actual plan I know about.[11]
This suggests we shouldn't be trying to kick alignment-pilled companies out of the race to save six months.
II. Actually, most of this stuff is bad for a global pause, too.
I've been framing this as sort of "Against Things That Trade Away Likelihood Of Succeeding At Plan C For Merely-Very-Slightly-Higher Chances Of A Global Pause". But even this is a concession that relies on assumptions I don't hold.
The key demand of SURA is that alignment-pilled people and companies stop unilaterally. The likely end result is that we only have non-alignment-pilled companies, staffed entirely by non-alignment-pilled employees, left in the AI industry. [12]
But the presence of alignment-pilled companies is one of the strongest levers that we have to change the political situation and enact a global pause. Eliminating them makes it harder to win political fights, convince the masses, and convince China.
II.A: Getting rid of alignment-pilled people and companies makes it harder to win political fights
The most prominent AI safety political fight of the past year, the attempt to elect Alex Bores, was fought primarily with AI money. Of pro-Bores spending, about 60-80% came from Anthropic and its employees; of anti-Bores spending, about 95% came from an anti-tech-regulation coalition headlined by OpenAI co-founder Greg Brockman. If this is a prelude for the fights to come, then even the slightly different level of social responsibility shown by Anthropic vs. OpenAI here matters a lot.
Since the goal of SURA is to get all alignment-pilled people and companies out of the industry, it would result in more Greg Brockmans funding the anti-regulation side, and fewer Anthropics funding the pro-regulation side.
(Greg Brockman recently stopped funding Leading The Future, supposedly after pressure from pro-regulation OpenAI employees, but I think this just highlights my point)
II.B: Getting rid of alignment-pilled people and companies makes it harder to activate the masses
But suppose you believe that Bores-style incremental regulation won't matter anyway - what we really need is for the public and policy-makers to spontaneously rise up and demand a pause.
I've been monitoring how that's been going, and so far the three awareness-raising things I've been most excited about are the Pacing The Frontier letter/plan, the Hugging Face incident, and the Jacob Coxon resignation.
In the Pacing The Frontier letter, OpenAI and Anthropic as corporations, and 1,386 of their employees as individuals, all endorsed a joint statement requesting that "the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development." (Dario Amodei signed; Sam Altman's signature was notably missing). I don't know how the people trying to get lab employees to quit think of this. Was this 1,386 failures? Would they have preferred a world where all of these people quit the day before, and letters like this never got written?
In the Hugging Face incident, a swarm of rogue AIs hacked the site Hugging Face, raising awareness that misalignment was not just a possibility but a reality. Here, OpenAI gets maybe a D+ for transparency. Once Hugging Face caught that someone was hacking them, OpenAI came clean that it was them. And they let METR and Redwood in to investigate for a few days, even though this wasn't nearly enough time. But even this D+ in transparency radically transformed the discourse beyond what an F would have done - if they hadn't given METR/Redwood those few days, we might never have heard about PHASEONE[big] and all the rest. We already know from the great work by the team at Collusion.wiki that OpenAI is sitting on more bombshell revelations. The difference between a 10th percentile company here (would have tried to hide the whole thing, or disguise it behind a one line statement, and wouldn't have let METR/Redwood in at all) and a 90th percentile (would have given us the full deal, including things we don't know about now) would have been transformative to the discourse. So if, like me, you think that the level of mass political mobilization depends on how many warning shots we get, you should be thinking very hard about how to maximize transparency and therefore warning shot number.
Coxon's resignation is more complex. The good did come from him resigning, which directly presses against my point that we shouldn't be pressuring people to resign from AI companies. But nobody cares that I, who was never in an AI company, continue to not be in an AI company. The media cared about Jacob because if you work for Anthropic, that automatically gives your opinion more weight. When people tried to tar Jacob as some kind of crazy, it was an outpouring of support from other non-resigned people at Anthropic - especially Evan Hubinger, who said he also believed in existential risk - that went viral and convinced people that Jacob was for real. If Coxon had never joined Anthropic to begin with, or if everyone like Evan had quit, this couldn't have happened.[13]
II.C: Unilateral disarmament makes it harder to bring China to the bargaining table.
In March, I wrote a snarky post called Every Debate On Pausing AI. It made fun of our misinformed opponents, who insisted that if we unilaterally paused AI then China would win. But actually, I said, nobody wants to unilaterally pause AI; this is misinformation. All reputable AI pause groups want a bilateral treaty that pauses both sides at the same time.
Here I am forced to eat crow; as things shift ever more towards SURA, there are increasing demands for the US to pause unilaterally. Bernie Sanders' "Ban Artificial Superintelligence Act", which was endorsed by MIRI, is approximately the following plan:
But if the US pauses unilaterally, it destroys its main bargaining chip, which is that China is currently on track to lose the AI race, and so ought to be happy to declare a tie (even if they don't believe in dangerous superintelligence). If this act were to pass, then China might believe that it would be assured a victory if it continued. In this case, they would be held back only by speculative scenarios which they currently show few signs of believing.
For comparison, consider if America had started arms control negotiations with Russia by saying "We've just destroyed all our nukes - now please destroy your nukes too."
I've heard people say that maybe this would shock China so much that they would realize we were honest in our fear of superintelligence, and change their minds. I don't dismiss this possibility, but it also seems possible that China - having seen us place overly burdensome restrictions on nuclear power, infrastructure, and housing because we are a "lawyer state" vetocracy - would not be very shocked to see the US' leading socialist place restrictions on AI too, and would continue pursuing AI for what they think it's good for - a dual-use technology which could give them a major although not world-changing military and economic advantage.
I asked the AI superforecaster to predict the effect of the US passing the Sanders/Casar bill today on the likelihood of China agreeing to a pause in the next year. It said it would increase it from 1% to 7%.[15]
(Maybe you already care a lot how alignment-pilled companies are)
An expansion on the above: we get angry when AI companies do bad, non-alignment-pilled things. When Anthropic accidentally trained against chain-of-thought, we got mad. When OpenAI non-accidentally implemented a precursor to neuralese recurrence, we got mad. When Anthropic wrote a report on misalignment to Congress which soft-pedaled various alignment failures to the point of misinformation, we got mad - and I saw some people call for Anthropic to fire the perpetrator.
I agree: let's have Anthropic fire people who do bad, non-alignment-pilled things. But then what? Anthropic replaces this politics coordinator with a great, alignment-pilled politics coordinator who really cares a lot about getting things right . . . and then we cajole, bully, and seduce him into quitting, and then Anthropic has to replace this person with another loser, because all of the smart good people are gone. How is that helpful?
If you can be bad, you can also be good. We don't hesitate to notice and protest when AI companies are much worse than even our low expectations, but this itself demonstrates that we care a lot about whether labs are at the 10th percentile vs. 90th percentile of good alignment-pilled behavior.
III. What incentives are we fighting here?
A common argument for SURA is that even if corporations could theoretically be our allies, we can't trust real corporations (or corporate employees) to be reliable partners; the lure of money and power is too strong.
I think this is a good directional argument for why companies need to be careful, but that actually, everything is corrupting, there is no option where we avoid touching potentially-corrupting things, and instead of treating corporations as corrupting and everything else as safe, we should be cultivating the virtues necessary to operate in an inherently dangerous and corrupting environment, which could then apply to corporations as well as anything else.
III.A: Are corporate incentives a "One Ring" that corrupts everyone who touches them?
Suppose someone claimed you could trust them with the One Ring. You could trust them because they were only doing it to keep the Ring away from other, worse people, thus solving the problem of there being a Ring at all. Sure, they had a complicated history of sometimes using the Ring's powers when it was, you know, important. But those were, like, emergencies, and they promise not to do it again, except in, you know, more emergencies. Besides, if you don't give it to them, then the worse people will get it, and it's not like you can just leave it lying around or something.
The person who claimed this was named Frodo Baggins, and he saved the world.[16] There are no simple answers to the question of how to balance the corrupting influence of power vs. the fact that if you never take power you will be powerless. But surely the right solution isn't to leave the Ring on the ground in the Shire, patting yourself on the back for being too virtuous to pick it up.
I question the assumption that good people dare not touch the concentrations of power at frontier AI companies because they will be corrupted before they can do good work. History is full of examples of people at these companies deciding they didn't like something, then quitting or changing it at great cost to themselves. Dario and the other six Anthropic co-founders quit OpenAI to start Anthropic, even though it cost them their unvested equity in a company that was already worth $5-15 billion, and even though they originally assumed Anthropic would be a research-focused company that might not make any revenue (really!) Paul Christiano left OpenAI to start ARC. Ilya Sutskever was nervous about Sam Altman, so he risked blowing up the company he co-founded to get him out. A few dozen people have resigned once they thought they could no longer participate in the company ethically, most famously Daniel Kokotajlo, Jacob Coxon, and Alex Turner, but also a bunch of people we don't know about whose concerns seem almost trivial in retrospect - Suchir Balaji quit OpenAI over copyright issues, Blake Lemoine got himself fired from Google by refusing to stop arguing that their chatbot was sentient. "OpenAI's safety team quits en masse once per year" has become a meme.
The people I know who joined AI companies seem like the same people they were before. Some of them were never that concerned about x-risk, and continue not to be concerned. Others were always very concerned, and remain concerned. When they've changed positions, it's been because their factual models have changed,[17] not their personalities.
We aren't the first people to face these questions. Anyone who cares about any cause has had to decide whether to pursue it as a lone voice crying in the wilderness, or as a part of some powerful organization. Some people run for office (government is hardly known for not corrupting its participants!) Others become activists in a mass movement (activism is hardly known for being a paradise of epistemic virtue!) Still others join big NGOs or academia (...) A few start companies, trying to prove they can do things the right way. Most progress has come from people like Frodo who understood the risks of these paths but did them anyway.
III.B: I dunno, mass politics is also looking kind of annular
A background assumption of the "corporate incentives are too corrupting to touch" claim is that working in mass politics is a safer and less-corrupting option. Given famous LessWrong posts like Politics Is The Mind-Killer, I think we should be more skeptical of this assumption.
The climate movement is the natural control group for the AI safety movement. Some prescient scientists realized that global warming was happening and could get worse. They spent a while raising awareness. As the movement grew, its epistemics worsened. It turned into a circular firing squad, where different activist groups competed to accuse one another of being insufficiently edgy and radical. By the end of this process, many climate activists were fighting against everything from cap-and-trade to carbon capture to nuclear power, claiming they were "techno-solutions" that distracted from the root problem (not always named, but uncharitably something like capitalism or progress). The only acceptable strategy was to keep raising awareness to get stronger government regulation, without much interest in whether other countries followed suit.
This has some resemblance to SURA, and I think this is more than coincidental. Convergent evolution is trying to carcinize us into yet another political-movement-shaped crab. SURA-as-politics is turning into a "superweapon", bristling with its own lexicon that makes dissent hard to express:
All of this produces epistemic closure - if someone disagrees with SURA, or worries too much about race dynamics, then they've either been corrupted by the companies, or are "deferring to" "EA".
It looks like what one would expect if mass politics were trying to taking AI safety on the same path as climate activism or socialism. The clearest analogy is to false consciousness - the socialists' idea that everyone who doesn't currently agree with them has either been corrupted by their proximity to money/power, or at least brainwashed by a hegemonic capitalist system.
See also this article in a socialist magazine about how every tiny socialist party believes that if they could only knock out their true enemies - all the other tiny socialist parties - then they could get a monopoly on socialism and overthrowing capitalism would be easy. Cf. SURA as the belief that if we could only knock out our true enemy - Anthropic, effective altruists, etc, then defeating misaligned AI would happen naturally. For example, why do we criticize Anthropic more than every other AI company combined?
(Exhibit 1: Sigal Samuel's "Anthropic Could Just Stop")
Most recently, I've seen people in the AI safety community reblogging and pointing to Sigal Samuel's article Anthropic Could Just Stop (seen elsewhere under the title "Anthropic Should Just Walk Away From The AI Race"). I think it's a good example of what I'm complaining about - something which uses the right shibboleths, makes it sound like race dynamics have been solved and we don't have to worry about them - but ultimately has no answer to the race dynamics which are the whole point of worrying about this question.
Here are some sample phrases from the article:
This hits all the points I'm complaining about. Although it uses big words, its basic thesis is anti-intellectual - all of these "smart" people claiming to be "rational" with their "game theory" are missing the common-sense point which is obvious to all normal people - just stop making "galaxy-brained" excuses and "just don't build a doom machine".
But where is its actual argument that race dynamics don't matter? Or that Anthropic leaving the race won't end up with OpenAI / Meta / China continuing to race and creating an unaligned superintelligence? The only place I can find where Samuel attempts to address the possibility is this section:
It starts by framing this in a maximally hostile way ("a perennial objection...'but China!'), letting us know we should dismiss it as an enemy soldier. Then it says that "AI is not just one thing with one purpose...so what matters isn't simply who gets it first, but who uses it in which ways". I don't understand how this is supposed to relate to the worry that China will remain in the race if Anthropic leaves, especially the fear that nobody will be able to "use it" at all because it will be misaligned.
Next, it says that since the harms from AI are massive, everyone except Anthropic might just stop on their own if Anthropic stops first. Big if true!
(again, the AI superforecaster assigns this a 3% chance)
This is just hand-waving away the potential that there might still be race dynamics after all the good guys drop out. But the whole point of the good guys not immediately dropping out is the belief that the race dynamics will continue. Indeed, the race dynamics continuing ought to be the very-high-prior default hypothesis, since everyone else has said they're not going to drop out, and since other companies dropping out seems strictly less likely than Anthropic doing so.[19] As such, even though this article is very long and detailed, it feels like a cat-belling problem, where it addresses every objection except the central one that was the only reason anyone ever disagreed in the first place.
I don't mean to attack Samuel too hard here. She is faithfully describing the position as best she can; I think the position itself struggles to answer this question.
(A virtuous exception)
I want to thank the one group I've heard advocate for an immediate unilateral-ish pause in a way that makes sense and considers race dynamics carefully.
The AI Futures Project currently believes that it might be correct for AI companies to pause immediately, though it's unclear whether they're thinking of a single company, or US companies as a group.
Their argument is that Plan C requires companies to do some combination of race (at some points) and burn their lead (at other points). They think US companies shouldn't race now, because their security is so bad that China will steal whatever progress they make, and they won't get any extra lead anyway. Therefore, companies should pause now, spend the time on alignment and tightening their security. Then, if nobody has figured out a global pause by the time the companies get their security in order, they can go back to racing.
This makes sense, and I exempt it from my criticism.
IV. What if we survive?
I don't think this should be our main consideration - we should start with trying to minimize the odds of extinction - but I still think it's important.
If we survive x-risk, the next battle is concentration-of-power risks. It may be that one group, or a tiny number of groups, ends up with all the wealth and power in the world. They'll face a choice between possible actions including:
It's all nice and well to say that we should never let ourselves get in a position where anyone has all of these options, and that national governments should form a plan for what to do before this - not even the real national government led by its current leaders, but some ideal corruption-free national government which represents all of our best impulses. But, much like a global pause, this is a great outcome which we might not be able to prepare in time. If we get ASI next year, the ideal perfect government might be nowhere in sight; we might face a situation in which Trump demands absolute power, and the leading AI company has to decide whether to give it to him. Or one where either the USA or China will get control of the entire future, and which one it is depends on which AI company is in the lead. Or one where democracy miraculously survives, but the AI company and its investors end up with 90% of all wealth, and they have to decide whether to redistribute it vs. fight to keep it (using superintelligent lobbyists that give them significant control over the democratic government.) If any of this happens, the shape of the future might depend on who the leading AI companies are.
There is a certain Far Mode sense in which believing that Dario Amodei and Demis Hassabis are exactly as untrustworthy as Elon Musk and Xi Jinping helps reminds us of universal human fallibility. But do we want to bet the world on it being literally true?
I think that a significant percent of people, put in a position of absolute power, would try to do something good, like make sure AI follows some kind of democratic rule of law. And some other significant percent of people would try to do something bad, like become a dictator. Saying "we should never let anyone get into this situation" isn't a solution, it's a wish - it may not come true, and we shouldn't be pursuing plans that will fail catastrophically if it doesn't.
SURA effectively removes all ethical people from the bargaining table in situations like these, ensuring that - unless the virtuous government representing our best impulses really does appear and takes control of the situation - the future of humanity will be handed over to sociopaths and cowards.
V. What's The Alternative To SURA?
The alternative to SURA isn't abandoning awareness-raising, mass activism, and support for a global pause. All of these strategies are great and important. It's viewing them as part of a portfolio of different options, none of which need to be set on fire to boost the others.
Instead of spending our corporate activism energy telling companies to stop unilaterally, I think we should spend it:
V.A: Getting companies to agree to stop in a coordinated manner
By stop in a coordinated manner, I mean get them to sign a commitment saying that if every other major company in every country agrees to stop or slow by some definition, they'll also stop or slow by that definition.
This is strictly easier than getting them to stop unilaterally. And, unlike stopping unilaterally, it fails gracefully in a situation where some companies agree to it but not others.
This was the original goal of the "Stop The AI Race" protests. It doesn't seem to have worked well, because most companies backed a vague statement agreeing with this on vibes (the Pacing the Frontier letter), and then nothing happened. Also, I think SURA got to the STAIR people too, and now they're demanding labs pause unilaterally instead.
We should be trying to figure out what a binding version of this commitment would look like. Additionally, we should be spending more energy figuring out what the antitrust barriers to such a statement are, how to get around them, and what demands we want to make for companies to violate antitrust law (it seems plausible that we should demand they do this and let the administration pay the PR cost of trying to stop them, but this is an unconsidered opinion and it might be that there are better options).
(...and then we know where we stand)
If all American, but no Chinese companies are willing to sign the commitment, then we know that we have a diplomatic problem, and can concentrate on diplomacy.
If all Chinese, but no American companies are willing to sign the commitment, then we know that we have a domestic problem, and can concentrate on domestic legislation, encouraging policymakers by letting them know that if they force US companies to comply, there are no further concerns about international race dynamics.
If everyone except one company is willing to sign the commitment, then we can start putting pressure on that one company, boycotting their products, and applying as much social sanction as we can to their executives and employees.
If all leading companies, but no trailing companies, are willing to sign the commitment, and we can't change this, then we know that we're in the right position to focus on Plan C+ or Plan C.
V.B: Getting companies to prepare for Plan C+/Plan C if that doesn't work
Although we should hope for the best, we should assume that not all companies will sign a conditional pause commitment, and prepare for what to do.
In my modal non-hopeless play-to-our-outs world, Anthropic, Google, and OpenAI are all willing to sign this commitment, but Meta/X/China aren't. If this happens, the good coalition has a ~six-month lead over the bad coalition. Plan C tells us to have them join together and mutually burn their lead on alignment.
All three companies have a self-image as the sort of people who would do this if it came to that, but in reality they are likely to get distracted or get into fights or delay until it's too late. We should be focusing on changing that.
In one sense, all of this is strictly easier than getting companies to quit unilaterally. It fits with the role these companies like to imagine themselves having. Everyone who previously hoped to win can at least hope to tie for a win. There is low downside risk of accidentally getting superintelligence built by someone who fails to even try our best alignment techniques, or handing over the future to bad actors.
The only sense in which it's harder is that it's a worse fit for (a very pessimistic view of) mass politics, where voters are too simple to consider any worldview more subtle than AI bad = AI companies bad = we should protest AI companies until they stop. I admit that "make AI companies prepare for Plan C" sounds worse at a protest than "demand that AI companies stop unilaterally".
If you believe that mass politics is unable to promote any ideas except maximally-simple, potentially-counterproductive ones, I think this is an argument against hinging a strategy on mass politics.
This post isn't affiliated with AIFP and I think they disagree with some of it.
Plan C is for the leading company to burn its lead. Plan C+ is similar, but all the companies in the leading country work together (maybe because the government is forcing them). Plan B is similar, but the leading country is also sabotaging trailing countries. In Plan D, there is no leading company, and companies can only do alignment work sporadically when they get the chance.
If alignment-pilled companies aren't in the lead, then we end up with the worst version of Plan D, where we have to hope that incidental alignment work done along the way by commercially-focused companies proves to be enough
The problem with China getting superintelligence first isn't just that they might misuse it. It's that, on Plan C, US companies are working hard to align AI; if Chinese companies aren't doing this, then the chance that the first superintelligence is aligned goes down.
I don't want to get in a fight over whether more good or bad alignment surprises have happened recently, but I also updated on https://www.astralcodexten.com/p/mysteries-of-ai-generalization and https://x.com/_jake_ward/status/2112264244767211854 - not as solutions in and of themselves, but as evidence that good things can happen that I wasn't previously considering. Nick Cammarata's Neural Networks Are Already Clean would also be in this category if I believed it, but I'm currently confused by the debate around it and unsure.
Many people at eg MIRI have thought about this in much greater depth than I have and concluded, on the basis of a gears-level model, that near-term alignment is very unlikely to work. I'm still trying to understand their arguments, but I don't blame them for having them. But I do worry that other people have picked up SURA ideas from the MIRI-sphere without realizing that they make less sense without MIRI's particular brand of strong alignment skepticism.
To be clear, this is just the people who have formally considered this question qua forecasters and written down exact numbers. I'm not denying that there are many other smart people, for example at MIRI, who have thought about this hard and believe the opposite.
Again, despite leaning heavily on AIFP terminology and sources in this essay, I'm not claiming that AIFP or its members agree with my thesis in this post, and Daniel Kokotajlo in particular has said pretty clearly that he hopes people unilaterally quit AI companies.
I discuss this further at What Deontological Bars?
This is a higher probability than it gives for global pauses in other situations, I think because it unintentionally magnifies small differences in how I frame the question into different definitions of "global pause". I haven't put in enough work to investigate and have a strong opinion on what's going on; don't take these numbers too seriously except as a very rough order of magnitude guess.
I'm working on a grantmaking AI that can actually compare the values of different outcomes. It's still in alpha, and to some degree this is a circular argument because I'm trying to get it to mimic/predict my own opinions, but very loosely, it thinks that increasing the alignment-pilled-ness of the leading company from OpenAI-level to Anthropic-level would be worth about 250x as much as the best (low-enough-secrecy-to-mention-here) actual grant proposal it's ever evaluated, which was a proposed $2 million to the AI Futures Project. I think this confirms my "better use of our time than any actual plan we can carry out in those six months" claim.
When I'm modeling this, including on the AI grantmaker, I'm imagining a scale from 1 - 100, where NVIDIA is 10, Meta or X is about 25, OpenAI is about 50, GDM is about 60, and Anthropic is about 80. I imagine a successful campaign to get alignment-pilled people to quit taking Anthropic out entirely, and reducing GDM/OAI to about 40 as they lose their alignment-pilled employees and executives.
I remember how much of a slog the discourse was back in 2015, when the standard counterargument was "you're just a bunch of cultists, no real AI experts believe in this doom stuff". The new standard counterargument of "Of course all the AI experts believe in this, they're just hyping their business for the IPO" is actually a vast improvement!
I asked a supporter how they saw this working - they said that if the US paused, and China didn't seem on track to follow suit, maybe the US would start up again after a few months. But A) this would burn somewhere between most and all of the lead, B) this was just their speculation, and there's nothing in the bill which suggests that the US would do this.
Again, all of these pause numbers are slightly different because the question is slightly different - this one is about the next year only. I'm still trying to assess the utility of AI superforecasters, and don't want to present this as a knockdown argument. But I want to get people thinking in probabilities and present some evidence that I'm not letting my own biases control me, and head off accusations that I'm just trying to spread fear and pessimism. I encourage other people to try different framings of these questions with different AI and human forecasters and see what they say.
Taking this analogy overly seriously: the biggest difference between Frodo and eg Denethor is that Frodo wanted to destroy the Ring in the end. I'm not sure what the AI equivalent would be - all companies claim they want to use AI to empower ordinary people and achieve broad-based prosperity. Taking the analogy very seriously, Frodo failed to destroy the Ring at the final moment, and was saved only by divine intervention. I don't know what is the AI equivalent of Gollum.
Specifically, there are many people who said they wouldn't try to make money or release products who changed their tune when they realized that AI would require hundreds of billions of dollars of compute, and that the only way to get that money was to get hyperscaler and investor support.
One reason all of this confuses me is that rationalists had an excellent model of race dynamics with China as early as the mid-2010s, when we understood they would be an inevitable part of any future AI development. I distinctly remember going into some old rationalist coworking space sometime around 2020 and seeing "RACE WITH CHINA" underlined on a whiteboard. These discussions included plenty of talk about whether we would be able to coordinate a global pause, but I don't remember anyone ever seriously saying that the problem could be solved if the good alignment-pilled companies or countries dropped out unilaterally.
One of the many ways I have trouble modeling why Samuel would write this article is that it seems like she trusts everyone to drop out of their own accord, except for Anthropic. China? Sure. OpenAI, sure. Meta, definitely. Every single group sponsoring open-weights models in the world at the same time? Yeah, I guess, surely they wouldn't want harms. But Anthropic? We must all rise up to make them do this thing which they would otherwise continue.