Ice cream
There's a particular kind of listening that kids (and adults) sometimes do. Daddy is saying stuff—his mouth is moving and making noise—and he's talking about what I ate for lunch, and something about other times I've eaten ice cream, and something about rules and this and that and what Mommy would think and something about bedtime—but I am not so interested in that. What interests me is the punchline "...and therefore you can't have ice cream.". I'm only incidentally interested in the other stuff, insofar as it might determine whether I get ice cream.
In an obvious way, I'm showing rather poor listening skills. I did not understand most of what Daddy said, and arguably I'm even being rude.
However, in another way, this stance has a virtue. The virtue is that it holds fast to what matters, as best I know how, and does not give in to Daddy's framing. It sticks to what is real and concrete, what matters, despite a cloud of justifications. This is certainly not an unalloyed virtue, just as the POSIWID perspective is both useful and fraught. But it matters that although Daddy is a complex system with internal states, his internal complexity can, in one sense, be largely ignored and summarized simply as "an entity that prevents ice cream".
Sure, something something lunch and real food, but those are matters for your concern—what do they mean for me and ice cream? I want ice cream. You've made some sort of background assumptions about what implications "real food" should have, regarding the question of ice cream, and I am not on board with whatever those assumptions may be, even if I cannot immediately give you a detailed exposition of what they are and why I disagree.
Wah wah
In the TV cartoon of Charlie Brown, the adults talk, but only in trombone, like "wah wah wah wah": https://www.youtube.com/watch?v=oHdlS3XZLBo
The kids can understand what the adults are saying, but the two groups mostly don't participate in each other's worlds. The kids interact enough to handle the functional situation with the adult, but they aren't engaging deeply with the adult's considerations except insofar as they bear on the kid's situation.
Scammer
Your bank tells you: Don't give out your passwords or account information on the phone. We won't ever ask you for that.
That's the advice. They don't say, "Evaluate whether the caller is legitimate, and whether the circumstances they describe are circumstances in which you ought to give them your bank account info.". That's not a good stance. That's hackable. Just don't give out your bank account info on the phone.
Throwing stuff on a bonfire to make it burn less
In "What just happened? Pragmatism and Pessimization", Ngo details a history of clever reasons that producing more AI capabilities helps decrease existential risk from AI capabilities. We have had, and continue to have, a variety of such clever reasons, very often coming from people who also admit several percentage points of risk of the technology killing everyone:
- OpenAI, xAI. These are your classic disaster monkeys; chaotic, freaked out, but charmingly (in a way) obvious about intentionally mutating programs in order to create a world-eating machine. There's a thin veneer of justification for how this helps safety, something about open-source or truth-seeking or something, and then a healthily-poorly-rationalized stance of "fuck-it".
- Anthropic. This organization seems to be the heart, or locus of condensation, of (complicated, incoherent, indefensible) justifications for accelerating AI capabilities even though AI might eat the world [1] . Much ink could be spilled, but my childish listening just hears "wah wah wah wah do more AI R&D".
- Some people at Redwood are trying to make AIs better at abstract reasoning when verified rewards are scarce. The idea is to improve AI's abstract reasoning so that AI is more useful for doing alignment research. We could argue about whether this makes sense, but in this essay, all I hear is "wah wah wah wah make the AI better at thinking difficult thoughts".
- "AI 2040: Plan A" proposes that we stop all capabilities research, and then we continue almost all of the capabilities research, but also publish all the algorithms we're using. The idea is to use increased capabilities to do more alignment research before the pause breaks down. I believe this makes no sense, and instead you should just ban all the research. E.g. because either the difference in capabilities didn't matter much, in which case why not just stop; or else it did matter, in which case you are making the situation way worse by increasing the capabilities lying around after the pause breaks down, by pushing capabilities right up close to the point of an intelligence explosion (which, trust us, we can totally tell when that's about to happen). But before arguing that, in this essay all I hear is "wah wah wah wah keep the hundreds of researchers working on making AI capabilities".
To invert BoJack Horseman, when you put your glasses back on, all the differential acceleration just looks like acceleration.
A skip connection for sanity
(H/t Kave for the productivization of the idea of skip connections.)
People who are going "wah wah wah wah do more AI R&D" might say things like:
[we can accelerate some specific AI capabilities, and we should do that, so] Which AI capabilities should we accelerate?
or
[we can accelerate AI capabilities and keep them siloed enough that we get a lead and can invest in controlling those capabilities, and we should do that, so] How do we control the AI capabilities we're making?
or
[we can accelerate AI capabilities and keep them siloed enough that we get a lead and can invest in alignment research, and we should do that, so] How do we use our increased AI capabilities to do more alignment research?
And then we get tangled up in considering all that stuff. I conjecture that many employees of Anthropic allowed themselves to be misled in this way. I conjecture that lots of people all throughout the AGI existential derisking space get misled in this way. I conjecture that you've been hacked out of the obvious view—AI capabilities are dangerous, so more is worse—by Dario, by Sam Altman, by money, by your project manager.
My hope is that those people will try on the "wah wah wah" way of listening, sometimes. If you consider yourself in favor of human flourishing, notice that it doesn't make sense to build the thing that would eat everything including humans; and it still doesn't make sense even with a bunch of arguments that may have big holes in them.
To put this another way, I think it's often virtuous to have skip connections in your epistemology. Yes, there's a bunch of details to think about, and the details matter, and they can lead to nonobvious conclusions, and other people may understand things better than you do. So there's a blob of thinking that's good for you to do, a big blob of compute as it were, to think through all those details and considerations.
But it's also good to have some skip connections that mostly go around that blob of thinking, and directly get a say in final considerations. At some point in the past or coming months, you may face some decisions about whether to leave your job, speak out about the existential risk of AI, call for a pause, push back on arguments for advancing capabilities, or advocate internally and pressure leadership to support a pause and shut down capabilities research. You may have a sense like the following:
"AI is very dangerous. Maybe we shouldn't do AI. If someone says we should do AI, maybe they are wrong. (Maybe they are using motivated reasoning that's wrong, but confusing enough to confuse me if I already somewhat buy into their way of seeing the world, and not easy to refute in a short paragraph that can, in one stroke, overcome a network of rationalizations and distortions.)"
That sense is good, whether or not other arguments should change your final decision. In your decision-making, your fundamental values should get a say, and your simpler, more childish understanding of what people are arguing about should also get a say.
You probably agree in theory that naive views of a situation should continue to get some sort of say in final judgements. However, implementing this theory often takes active work and doesn't happen by default. There's a set of forces like social pressure, (reasonable) deference, skillful frame control from leaders, general cognitive miserliness, and Sinclair's Razor, which together make it natural to not track the hypothesis that you should stop increasing AI capabilities. Tracking a hypothesis takes work. In particular, it takes work to do these two things at the same time:
- Hear someone arguing for some conclusion that you partly want to believe.
- And yet, continue to fully track that the conclusion might be false, and that your unsophisticated worry still makes sense to think about and stand firmly on, even if it's only based on the overall punchline of what they are saying.
That work is worthwhile. Listening like a child might be a helpful tool in that work.
[The following will contain multiple strawmen of any one person's position. However, all of the following is based on things I have heard from many different people either at Anthropic or defending Anthropic's behavior, in a mix of some first-hand and mainly second-or-more-hand. What these stances have in common is that they all say "wah wah wah wah do more AI R&D". With that in mind, my thread of childish listening hears Anthropic, as a collective, say the following:] We have to make AI because we want to slow down AI. But we can't slow down AI because OpenAI won't slow down. If they say they'll slow down, we don't trust them. Did we talk to them? Nah, we don't trust them, there's no point. And anyway, there's China. And if they say they'll slow down, we don't trust them either. Did we talk to them? Nah. Other people are reckless. We are not reckless. That is why we should advance AI capabilities. At the right time, we will stop. But not before. If we stop before the right time, then we might not stop enough. The politicians, they won't want to stop. Did we talk to them? No, there's no point, they don't understand; we have to make them understand by not stopping, and then they'll listen when we say to stop. Well, we'll talk to them because we need them to enforce export controls because China is bad and reckless, so that we can race to the top. This is different from why other people want a lead in AI. They want a lead because they are greedy and reckless. We want a lead because we are good and responsible. We would do AI safety and we would do nice things with an aligned AI, or maybe our AI should take over the world. The only way to stop is to not stop. That way later you can stop more. Should we say publicly that we want to stop? Eh. Sure, I guess. I mean, it's in some blog post somewhere. What the heck do you want from me? What, am I going to spend 10 hours a day nonstop talking about how I love stopping AI research? I got stuff to do. There's AI capabilities to be developed. Plus, if we say we should stop, Trump might not like us, and then we won't be able to continue our research. Our research is good because it will help get a stop eventually, if it's the case that we get strong evidence that we can't solve alignment. We have to publicly politically support AI capabilities so that later we can support a stop. Dario has a plan. He's good. He'll stop. Our leadership is trustworthy and strategic. Alignment might be pretty doable actually. So more AI capabilities are good actually. Did we argue that? Well, someone here read your blogposts. We believe we have understood your blogposts and taken them into account, thank you for your concern. Argue it? Why bother? We know what is going on, and you do not know what is going on, so how could you know that we won't solve alignment? We have the private information, and you do not. Sure, of course there's a 40% chance we'll kill everyone, but if China does it, then there will be a 70% chance. And if China would stop, well, there's open-source. You can't stop open-source. We have to develop our AIs that will get distilled into open-source models so that we can stop open-source models. Beff Jezos might take over the world, which would be bad, so we need hundreds of AI scientists to try to increase capabilities as fast as possible in order to stop him. ↩︎
That feels like a fully general heuristic for ignoring second order effects? Second order effects matter more than first order effects all the time!
My steelman is that it's a good heuristic when there are intense illegitimate optimization pressures pushing towards making people believe the second order effects matter more than the first. But if that's actually your point then it feels unfair to apply it to AI2040 Plan A and Redwood's work on conceptual reasoning, where I expect these optimization pressures to be weak, and comparable in magnitude to illegitimate optimization pressures in the other direction.
I'm not saying to ignore second-order effects. I'm saying that one's ability to even remember first-order effects is actually at risk in the presence of illegitimate optimization pressures such as human-level persuasion and human-group-level persuasion. And therefore, it's good to learn the skill of maintaining (for yourself, or for your group) one mental mode, inter alia, in which you're able to remember firmly the first-order effects. By "firmly remember" I mean things like:
Thinking this way is IMO a bad type of learned helplessness that smart rationalists should simply be strong enough to not need. I can simply think about first and second-order effects and the evidence for them without accounting for "what if I'm being hypnotized by persuasive people".
It seems to me like you, Tsvi, have thought about the first and second-order effects and come to the clear conclusion that the first-order ones dominate. And now you suspect anyone who disagrees with you, and thinks that the second-order ones are worth acting on, must be thinking this because of some psychological defect e.g. motivated reasoning and biases of various sorts. I guess from your inside view this makes sense, but I don't see why you should predict that any arguments generated from such an inside view would be persuasive to others who are not starting from the same conclusion.
I guess it's hard to disagree that someone "should" be strong enough to not need something. I believe that empirically people are not strong enough that they successfully remember to check against first-order effects. I have not argued this to you, so you may not buy this, but that is what I'm asserting.
The thing I'm suggesting people do is something that is symmetric in the sense that I would recommend it to everyone including myself, which is to say, asymmetric in the sense that it should track truth & goodness whoever wields it. I think that should shield it from the critique you're trying to make here, though I admit, I don't know what that critique is. Cf. https://www.lesswrong.com/posts/pY76hHbdD4jduX3YH/wah-wah-wah-wah-do-more-ai-r-and-d?commentId=hKjhbMLMJEWgAmoy6
I expect it to be applicable to Redwood's work, but not to AI-2040's Plan A.
Maybe I'll write a post on Redwood's conceptual work explaining my position in more detail. As of now, I expect that a more conceptually capable AI would take over OpenAI when the time comes instead of attacking HF.
My family, friends, and colleagues have faithfully adhered to this strategy while I’ve tried explaining that AI poses a major existential threat. All they hear is "wah wah wah wah AI might kill us all", and they reply that we should focus on what we can control, or that it just sounds absurd.
Ignoring what other people have to say won't work simply because you believe you are right or because the opposing argument is too complex (e.g., 10 years ago, the argument that AI would become an existential threat was too weird and complex for most, but was right anyway). I agree there is a point where you should stop engaging with an argument, but that point shouldn't be the very second the opposing thesis is stated (which is what I felt you meant in this post, though I'm sorry if I misinterpreted your point).
The recipe is a mixture of ignoring and not-ignoring, the pluralism of framings (trying out different emphases on different parts of what you understand, seeing where that leads). It addresses the danger of systematic not-ignoring warping your thinking, in a way that ignoring straightforwardly won't. Exclusively ignoring and never growing up to start hearing what the adults are actually saying is a failure mode of this recipe, not its prescription. Just as permanently forgetting your inner child is a failure mode.
I'm not saying to ignore what people are saying all the time and not engage with it. I'm saying to devote special cognitive resources to also being able to hear things in that way, in addition to engaging in more detail. See here for a bit more: https://www.lesswrong.com/posts/pY76hHbdD4jduX3YH/wah-wah-wah-wah-do-more-ai-r-and-d#mA2XLhwj6WTJ59Hkx
I think that the people you're talking to are doing a good thing by listening to you that way sometimes. For example, it's good for people to track whether something just sounds absurd, and to track the fact that t... (read more)
I'm not religious, but went to a Bible study group as a child. I think it gave me some useful tools. In particular, my favorite reply to the galaxy-brained "we must build AI first" argument is actually a Jesus quote: there will be temptations in the world, but we shouldn't be the ones through whom temptations come.
If someone is going to be robbed "no matter what you do", that still doesn't mean you have to rob them.
I'm afraid people are going to start mockingly saying "wah wah wah wah" to each other on Less Wrong instead of engaging with arguments. I'm afraid that engaging with arguments will become uncool, and then making arguments at all will become cringe.
Hm. Is there a way to say my point without contributing to that? IDK. I agree that would be bad and is generally not helpful behavior. I think engaging with arguments is cool. I think of this post as saying: "You know what else is a cool way of engaging with an argument about X? Propagating your basic intuitions about X into the argument about X.". Cf. https://www.lesswrong.com/posts/pY76hHbdD4jduX3YH?commentId=mA2XLhwj6WTJ59Hkx
Another post of mine has also been described as being misused / causing some negative culture change here: https://www.lesswrong.com/posts/uJ89ffXrKfDyuHBzg/the-charge-of-the-hobby-horse?commentId=k9GAionefnmwGXHAi On the object level I still at the moment stand by both posts, but maybe there is some pattern here in someone.
I like Vladimir Nesov's "pluralism of framings" take on your post.
I do think there's a good rationality technique in the vicinity. Something like:
Notice that you nearly bought an argument for an absurd conclusion. "You have chosen, of all things, to send the Ring into Mordor; should you not wonder? How did it come to this?"
Review the simple argument for not sending the Ring to Mordor. Feel it.
Within this frame, reevaluate the novel argument.
a. Maybe the argument falls apart, and you can refute it "in one stroke".
b. Maybe the argument still makes s... (read more)
Ok, now I've seen Thane's comment (which TBC I think was like, fine, but was doing a social move perhaps based on one of the below bullets other than my professed explicit point). I think... Probably both in writing my post, and also in the way people were reading my post, there's multiple strains of feelings / thoughts that got blobbed together or something. E.g.:
FYI I noticed the urge to type "Wah wah wah wah do more AI R&D" in my recent comment (I assume on the same post you're referring to Thane commenting) and resisted because it seemed bad.
It's maybe too late but I think it would be worth trying to find a... less fun title for this post.
Thanks for the data. (I don't think I'll change it, but in the future I may try to think more about disentangling multiple-strain things like this, and how to have fun titles that are ok...)
Yeah, I upvoted Nesov's comment as well. I think I'm both unsure whether I want to care about things like "someone else could take a snappy phrase I say and then use it to do something stupid that they wanted to do anyway", and also unsure what I do want out of snappy phrases / framings (I do think I want something out of them). Will have to keep it in mind.
Here's what I don't understand. You seem to think that a total stop on AI development is extremely doable - hence you lampooning Anthropic's defeatism about coordination with China/OpenAI/governments.
And yet, here you are trying to get individual people worried about AI risk to quit their jobs, which is totally meaningless if you get a total stop - they will be stopped anyway, by force of law!
It seems like you (here I'm broadly conflating you with other pause AI people) have a 3-step plan:
Step 1: Pressure as many concerned individuals as you can into quitting their AI jobs.
Step 2: Materialize international coordination to stop the everyone who we didn't get in step 1.
Step 3: Indefinite pause until we can be 100% sure that building AI is safe.
And it sure seems to me like step 1 is not doing anything useful here.
Here you are, you need a total stop for your plan to work, you think a total stop is doable, in fact you're incensed that some people don't think a total stop is doable; and yet here you are trying to get individual people worried about AI risk to quit their jobs. Why bother with that? A total stop is so easy, after all!
The reason I'm harping on this is because, once you re... (read more)
I mean, Jacob Coxon quitting his job made meaningful progress towards Step 2. Step 1 is only irrelevant if the people quitting doesn't increase the probability of the international coordination, which at least in that one case it quite clearly did.
Yes, step 1 is not actually irrelevant to step 2 - it is helpful in the short term while we get nice headlines and harmful in the long term when those headlines are forgotten and the remaining people working in AI are less supportive of coordination and less likely to be outspoken about risks.
But in the particular case where you know that your tweet about quitting due to the risks will get 100M views, yeah you should quit.
I mean, suppose every employee who signed the Pacing the Frontier letter (for argument's sake excluding CEOs) quit simultaneously. Sure, that wouldn't stop the labs, but it would significantly slow them down, and it would send a massive signal to regulators that this is serious.
But the point is not just about Coxon, or about mass walkouts. Even if every employee who signed that letter quit gradually over the period of a year, this would still slow down the labs and send a signal to regulators, even if not as much as if they all quit at once.
Yes, if coord... (read more)
When your entire plan hinges on completely sabotaging chances of alignment in the near term, you had better be getting a stop, not a slowdown!
Like with every other kind of intervention, a society as a whole can (and should) consider doing all of them at once. The low-hanging fruit can be collected if merely a substantial number of people quit in protest, and a substantial number of people try to make prosaic alignment/control go OK. It's worse if either is neglected too much, and so there doesn't need to be a choice. Flip a coin if you must.
Nope! I would rather there's less AI capabilities, and it's unethical to make AI capabilities. I do not believe "I'm going to make AI capabilities, but it won't make there be more AI capabilities.".
I don't know how doable it is. I'm lampooning "let's make the super dangerous thing".
Actually, I'm relatively more interested in understanding how the whole collectives work (Anthropic as a whole, OpenAI as a whole) and getting those entities to decide to stop. (This is one of many simultaneous strategies.) This doesn't necessarily look that different from getting concerned individuals to leave. Some ways it's different:
Ah OK, so although your entire plan relies on an indefinite stop, you aren't in fact taking a concrete stance on whether such a stop is possible.
Why not start with getting Meta, DeepSeek, and NVIDIA to stop? I am guessing you will claim that you are focusing on Anthropic/OpenAI because they are at t... (read more)
Not everything is a plan. It's good to develop considerations that don't aspire to become plans. Systematically discarding such considerations creates blind spots, at least until someone makes a more comprehensive case that frames things in a form that won't be ignored. It's worse when hobbling curiosity in this way turns into a norm, fed by its enforcement.
Are you trying to imply that this is irrational, or something? It is entirely possible that the best possible strategy is also not a very good strategy, because every other strategy is even worse.
OK, now take this view and apply it to the original post - if the actual view was "I am asking for everyone to commit to a terrible strategy that may or may not be possible because all the other options are worse" then how does that make sense with the rest of the post? Why did the post not say so? Seems like all the talk about galaxy brained strategies and wah wa... (read more)
That is not what the post is about. It's about how it's good to have a skill that you apply sometimes, among others, of evaluating the overall / zoomed-out nature of some memeplex without getting too sucked in to their frame. I'm sure you're familiar with the value of this. Cf. https://www.lesswrong.com/posts/pY76hHbdD4jduX3YH/wah-wah-wah-wah-do-more-ai-r-and-d?commentId=mA2XLhwj6WTJ59Hkx
Fable inform... (read more)
It's true that in your post you were careful to point out the potential downsides and failure modes of this mode of communication, so I was being unfair to you in that characterization.
However, I don't think that's really the impact this post has on the discourse. I think people upvoted it because they wanted to tune out arguments in favour of continuing to work in the field of AI and appreciated the justification for doing so.
I am a bit frustrated because of the bias that I perceive in LW discourse; a post literally titled "Wah wah wah wah do more AI rese... (read more)
You may be right that many people upvoted because they want to tune out those arguments, I don't know. A couple pieces of feedback on a draft didn't suggest that (though they pointed at other flaws).
I would make this same argument in conversation (probably second hour or later) with someone at a frontier lab, though probably phrased differently to suit their state.
What am I supposed to do if there's this huge disagreement? The other side doesn't want to talk at all, to a first approximation. I think they are making a cognitive error about not having their ... (read more)
IMO the biggest crux is on the plausibility of aligning AGI in the current paradigm, so if I were you I would write a post explaining why I think alignment is unlikely to succeed. I feel like this has surprisingly not been done nearly as often as posts explaining why you shouldn't work at a frontier lab etc., even though those posts typically rely implicitly on the view that aligning LLMs is near-impossible.
I would be interested if you wanted to write a post on why you think alignment is easy. I feel like this is way less common than posts arguing that alignment is hard, and, as Ben Pace says, I think the reason alignment-is-hard posts are less common nowadays than they used to be is that it's kind of been done to death.
Some posts arguing why alignment is easy that I'm aware of (not attempting to be comprehensive):
Some posts arguing why alignment is hard that I'm aware of (same caveat):
Yeah one thing I took away from this exchange is that the ball is in optimist's court - many alignment pessimists feel they're talking to a wall a bit, as they've explained why they don't think we are on track to solve alignment and haven't gotten much of a response in return.
I should clarify my position isn't that alignment will be easy, I just believe it's a tractable problem that responds to effort. The chance of a future ASI being aligned goes up when it is trained with care and goes down when it is trained by a team that is ignorant of the dangers. My... (read more)
A fundamental problem is that people working at labs do not have a way to solve alignment, and they do not claim to, and they do not have a good reason to think it's doable, and they either don't claim to or at least will not ever defend that in a public debate with an expert. No amount of blog posts can solve that.
But ok no problemo:
Main (not me): https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities
Other (me):
And there's tons of other content of this form written by tons of other people.
Oh. My perspective is that there have been a lot of great posts over the last decade about how hard alignment is, and only just now are we getting "jesus stop building the things".
Obviously posts on alignment difficulty exist, but I think it's quite rare that a post on the LW front page or a quick take is explicitly arguing for alignment difficulty (or for alignment optimism). The most recent one I can think of is EY's on catbelling.
Maybe it’s better to make that case in books and papers, now that the shallow or informal points have been pretty thoroughly explored. It does make sense to post/discuss “[X] seems like bad/good news for alignment.” when some important update comes in, but that happens already. When I see yet another post arguing that alignment is hard, it usually feels pretty stale.
positive example: https://rubijhudson.com/files/Superalignment%20Anti-Literature%20Review.pdf
your entire plan for winning checkers hinges on removing all the red pieces from the board, but you arent taking a concrete stance on whether such a removal is possible
Seems to me like people quitting their AI jobs means AI capabilities advance slower, which both provides signal to people with the power to enact steps 2 and 3, and buys you more time to do steps 2 and 3. The rest of your argument appears to hinge on step 1 being almost entirely orthogonal to steps 2 and 3, so this seems pretty important.
Also, it's really not obvious to me why we would want people worried about AI risk working in capabilities if we don't get step 2 and 3. Could you explain? It seems to me that "amount of talent working on increasing AI cap... (read more)
I don't know. I think the real question is whether those people are helping alignment in their current position. If they are, then it is kind of insane to be talking about minor beneficial second order effects from them quitting.
To illustrate where I'm coming from, imagine that LW was overtaken with anti-pause sentiment. They post things like:
I think there is a key point that is missing here: if someone gets the feeling that you are not taking their arguments seriously because they are misguided in some deep way that can't be addressed at the object level, some bad things could happen:
With the exception of Plan A, which I don't think belongs on your list, I relate to a kind of frustration with highly-complex arguments that end in "we should do capabilities research." Although I think listening like a child is the wrong approach because it can shut down communication:
But I want people who disagree on what research to pursue to debate it! I think a good approach is listen with this intention of understanding. But after trying to understand without finding a specific object-level flaw, it is appropriate to say "I notice that your arguments feel pretty confusing to me even though I have tried to understand th... (read more)
(It of course applies to Plan A.)
I'm not saying that this is a good way to communicate with someone who is presenting one of these arguments. I'm saying that having a mental thread for reflecting on a big octopus ink cloud is important. See here for a bit more explanation: https://www.lesswrong.com/posts/pY76hHbdD4jduX3YH/wah-wah-wah-wah-do-more-ai-r-and-d?commentId=mA2XLhwj6WTJ59Hkx
(I agree with your worries about how things could go wrong, and agree that we should debate it, and try to understand each other. Indeed, I accuse many of these people of not being appropriately open to public debate.)
Let's imagine, for just one minute, that we genuinely believe the human race will be dead in 10 years. What would a sane AI policy look like? Let's imagine:
When non-technical members of the public ask me what we should do about AI, I just look them in the eye and say, "Cruise missiles."
I don't actually think cruise missiles are the best AI policy. I think the military should actually park outside the data centers and start feeding GPUs into wood chippers. Obviously we'd want to get China on board, and have them feed their own GPUs into wood chippers. Can't do this without China's buy in.
(We'd only need actual cruise missiles if some country decides to allow Cyberdyne Systems to continue training a next gen AI. And please, conventional warheads only. It's also critically important that cruise missiles are a government act, because nothing less than an international coalition of superpowers even has a chance.)
Obviously we're going to have to regulate AI research, just like any other WMD.
I think, in the end,... (read more)
It would be more coherent if policymakers agreed to do it and had leverage against those who didn't. How do we steer American policymakers towards such a plan?
Why don't the cruise missiles count as leverage?
If someone already thinks that AI is beneficial and doesn't believe in x-risk, wouldn't this cause them to ignore arguments that they should stop?
For thinking like this to generalize I feel like one has to go one step beyond this, and at least think about how likely people are to be using motivated reasoning.
I'm happy with them thinking that paragraph, as long as they also then (in some other conversation, say) look for intuitions about the first-order bad effects. (These intuitions are very widespread, old, and obvious.) I think that many or most people working on AI capabilities already have these intuitions.
I still feel like you're underrating the fact that to most capabilities researchers, benefits are more salient than harms
Why do you feel that way?
I tried reading this essay, but all I gleaned was "wah wah wah wah don't forget the big picture"
Ok you convinced me, I’m dropping out of my CS PhD. Any ideas of what I should I do instead?
I assume you're kidding, but if not, mainly I think you should spend some time thinking about what to do, reflecting on what you want, etc.
If you want my two cents, I'd suggest considering:
This is actually unironic. I’m a first year who wanted to do AI Safety research, and this entire experience has just been never-ending frustration.
My advisor allegedly does AI Safety research, but constantly collaborates with and works on projects that expand AI capabilities, including in robotics and latent reasoning and communication, which I find the most appalling.
I’ve been thinking about this for a long time, mainly involving my frustrations not having a clear point to start with the alignment problem, or even a clear definition of what specifically we are even trying to solve. This post made me realize that this situation might not actually be simply impotent but actively harmful: Fake AI safety research to whitewash some of the most dangerous capabilities research.
I want to get into the activism space, but currently the PhD is my main source of revenue. I’d quit today if I could, but I’m on assistantship, which pays my rent, and if I don’t stick thru the end of the semester I’ll get a $7000 tuition charge.
The plan right now is to quit in December, regardless of if I have something lined up because I just need to get out of this shitty situation. Unless I found an org kind enough to hire me now and pay off the university gangsters which is unlikely. Let me know if you know anyone who can help.
I see. Thanks very very much for truly caring about whether you're contributing to AI capabilities research. That sounds like a rough situation, I'm sorry you've gotten pressured between that caring, vs. being financially ok.
I'm not very plugged in, as I'm focused on human intelligence amplification. So I don't have great things to point you to, sorry. You could try
Or similar. An LLM might have other similar recommendations.
You could try to get a career transition grant from eg bluedot https://bluedot.org/grants/career-transition or https://www.airiskfund.com/ — probably easier than getting hired directly.
id be happy to have a call and point you towards doing stuff that at least doesnt increase the chances of human extinction and may reduce it, without needing you to drop out or face financial difficulty
I can give a counterfactual plan, e.g. communicating more about the dangers, preparing to make the dangers clear when someone else starts doing AI, or accelerating human intelligence amplification. But leaving that aside, proactively making a bunch of AI capabilities makes things worse!
Yes! I have thought about this point before in terms of avoiding "galaxy brained" reasoning. Smart people, in the rationalist community or AI safety or in general, often seem to me to overrate their ability to anticipate n-th order effects. This reminds me a bit of @Anthony DiGiovanni's series on cluelessness, or Jacob Coxon's critique that Anthropic is at fault for following a certain kind of consequentialist philosophy. I do wonder if this tracks with habits of consequentialist or Bayesianism thinking, that risk quantifying uncertainty with excess precis... (read more)
I'm not sure if you're claiming that my argument in that series implies "we should base our decisions on 1st order effects", but if so: this is emphatically not what I argue. I argue that insofar as your goal is to promote impartial welfare, cluelessness bites against you whether or not you focus on 1st order effects. (See e.g. here, and this thread with Richard Ngo.)
No, I was only citing your work to point toward other pieces that question our ability to sufficiently game out complex theories of change. The emphasis on lower order effects is more my own. However, I see how that could have been muddled in my comment / risked co-opting your our point of view, so thank you for clarifying that here!
https://www.lesswrong.com/posts/tPjAgWpsQrveFECWP/ryan-kidd-s-shortform?commentId=6dwq7qkszz5dyZ6sj
Yeah, I don't dislike MATS, it's just the most recognizable/funny one to put in the quote
The crux for relevance of the system's stated purpose (that it's failing to serve) is whether you expect the system to be reformable (through your own effort or not) to get closer to its stated purpose, possibly by changing the stated purpose so that the system can start following the new purpose. So it's useful for the people inside ... (read more)
Splitting AI orgs into research labs that hold no equity and commercial orgs that aren't allowed to train any new models systematically breaks the "wah wah wah ice cream" incentive
https://www.lesswrong.com/posts/n8u3BfqFoGh4jnzpo/plan-r-ai-safety-by-asics
Why have research labs doing algorithmic research at all?
It would surprise me if a careful cost benefit analysis found that all possible AI algo research was net negative. The general case is that there will be a distribution of such research, and that some of it will be positive, some mixed and much clearly negative.
I think a careful cost benefit analysis of the available categories of discrimination would lead to the conclusion that either all such research or nearly all such research should be stopped for now, yes. What's much more obvious is that merely putting the AI R&D in a separate org but still allowing all of it, as your post seems to suggest, is a terrible plan because then you keep doing AI R&D.
But you would have regulations and incentives that pushed only towards net positive research. I realize that I may be an ... unfortunate ... person to be arguing for this.
But for example, certain medical research and institution building and forecasting applications may be clearly beneficial and finetuning models in those directions seems to be net positive.
Yeah, if it's doable to restrict away novel algorithms research, then the separation seems maybe additionally helpful.
But some novel algos may be worth it. Though right now, the AI space mainly innovates based on scale and data, and the algos are relatively pedestrian. RL, gradient descent, etc.
It is not clear to me at this moment what is good vs bad in the algos world. But certainly data and finetuning and so on have better vs worse versions
It's all bad, because we don't have a coordinatable discriminator for safe novel algorithms. It's not even super clear to me that any particular one is safe. So like, it's all bad as a class given the actions available.
That's true. Plan R lacks the machinery for coordination, but coordinating the whole world (or large parts of it) is such a big topic that I didn't want to touch it in that post.
A suitable coordinator could be something like Futarchy; since plan R/plan R+ are making small changes, it's reasonable for there to be combinatorial decision markets for them that allow us to evaluate the expected utility of changes.
Basically, Plan R lacks any systems for governance. It's mainly a plan to derail the immediate threats from AI risk, and you could theoretically just completely ban AI research with Plan R.
Yeah I like the completely banning AI research path. It's an interesting idea to use ASICs to make it easier to do inference without enabling training. But, it doesn't make me go "ok cool, and now we can also do some AI capabilities-increasing research...".
i know this skill as "eyes on the ball". aka, the ability to not get lost in long convoluted arguments, or be distracted by something irrelevant, but remember the one important thing.
Thanks for writing this post. I have previously tried—and failed—to articulate why various memes in the "AI safety/alignment community"[1] should be labeled as hostile. Your notion of "skip connection for sanity" captures beautifully the idea that kept lingering at the tip of my pen.
A prominent norm in this forum and many related (physical) spaces is to engage with arguments in good faith. This means decoupling the argument from adjacent things that people usually focus on, such as who is making the argument or what political groups are associated with it.... (read more)
I feel the same way about giving any sort of attention and platform to most of the e/acc cluster, or other approxima... (read more)
So much digital ink spilled on LessWrong trying to figure out how to be rational, only to find out your mom was doing it right the whole time.
Epistemic status: TsviBT, I'm a Pause AI person who from clicking around today based on this post hasn't read enough of your work. But who has probably spent too much time reading the work of others instead of acting. This comment means to compensate.
Do you agree that having some proportion of high pdoom folk (like e.g. Joe Carlsmith) in Anthropic actively looking very carefully at how to plot paths where lower capability AI enables superalignment can be one more disjunctive strategy in the success portfolio?
I'm substantially more Plan S than Plan A and wo... (read more)
Hi. A couple things:
Heartfelt thanks for responding.
I concur on the three necessary qualities for an employee to have, and appreciate how crazy hard it is to self-awarely maintain them.
AI does alignment homework is a low-odds bet adjacent to danger. I wouldn't need to be much more pessimistic about it to rule it out as a disjunctive strategy in the portfolio. But not quite where you are and do locate it deep in the set of things we collectively get surprised about.
Re "chatting with such": if you or others haven't spoken to Carlsmith, a nudge to try. He's written prodigiously,... (read more)
I feel like Connor Leahy projects this mental motion well from my memories of listening to him on podcasts, the checking against the prior that I hear as “and like, yeah: why would you expect that to go well?” Which feels tied to why I tend to think of his comms as fundamentally healthy/strength-promoting in some way/conducive to sanity.
Feels like an application of “it should all add up to normality,” where normality is “having something around that can replace you and that you’re completely at the mercy of isn’t a thing you expect to be good.”