This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
1.

In hindsight I should have talked to them about the curriculum before accepting the offer. My reasoning at the time was that accepting the offer first would send a signal that running the curriculum wasn't a hard requirement for me. However, I don't think that message got through.

2.

For context, some of the topics that are covered in the curriculum include the feminization of academia, racial IQ differences, and anti-white racism. Though note that it's a weird curriculum in a few different directions, rather than being uniformly right-wing—e.g. it has some Lacan-inspired stuff, some hippie-inspired stuff on beliefs as emotional strategies, a bunch of rationalist writing, etc.

3.

The specific message I sent them after learning that were concerned about risks from hiring me:

FWIW I’m pretty familiar with feeling a sense of liability related to what allies or affiliates might say, it was a recurring theme at OpenAI. So I expect I have a reasonable sense of where you're coming from, and I'm happy to brainstorm ways to mitigate that sense of liability (e.g. not putting Resolution in my twitter bio). More generally, I want to be collaborative on this topic, and won't hold you to any commitments you guys have made if you want to re-evalaute them now.

I don't know exactly what risks you're most concerned about, but it might be useful for me to explain why I personally don't feel like I'm making a costly tradeoff against making AGI go well by opining on controversial topics online (and in fact believe that it's meaningfully helpful). I'll talk about my inside view and an outside view. Also happy to discuss any qualms you might have on a personal level/re office culture tomorrow.

Inside view: for better or worse, a lot of the strategic considerations around the future of AI are actually deeply entangled with the dysfunctions of the wider modern world. For example, academia's inability to grapple with either AGI or AGI risk seems like a very important dynamic to understand, and that seems quite related to how academia has changed over the last century. As another example, if we want to reason about how more capable entities might dimore sempower less capable ones, probably the best example we have is colonialism—and I think discussions of AI risk would be significantly less vague and abstract if people were less scared of honestly analyzing historical power struggles. (Daniel Kokotajlo's writings on the conquistadors as precedents for takeover are one big exception, but still miss a lot of the potential insights.)

Outside view: one of the biggest lessons of the last few decades politically, I think, has been that the costs of intellectual taboos are much larger than most people expected—e.g. they meant that the massive populist upheavals of the last decade took almost all intellectuals by surprise. I also think that AI safety is in a much worse position today than it would have been if it'd taken a more principled truth-seeking stance a decade ago (for example, AI governance would have been less likely to throw in with the Democrats in a way that alienated MAGA). Indeed, with respect to engaging with the current administration, it seems like standard EA views are more of a liability than any of the things you mentioned by a significant margin, and I expect that having a right-winger at Resolution would be a net gain in political capital. (To be clear, I don't think that this should be a significant consideration for you, I mention it mainly to highlight that the risks are actually more symmetric than they might seem.)

In general, though, I wouldn't want to join Resolution if I planned to spend a significant proportion of my time thinking about this stuff. Indeed, a key reason I'm excited about Resolution is because my research has lately been steering towards much more foundational-level questions that I previously expected, and so I intend to be pretty focused on pinning those down.

1.

In hindsight I should have talked to them about the curriculum before accepting the offer. My reasoning at the time was that accepting the offer first would send a signal that running the curriculum wasn't a hard requirement for me. However, I don't think that message got through.

2.

For context, some of the topics that are covered in the curriculum include the feminization of academia, racial IQ differences, and anti-white racism. Though note that it's a weird curriculum in a few different directions, rather than being uniformly right-wing—e.g. it has some Lacan-inspired stuff, some hippie-inspired stuff on beliefs as emotional strategies, a bunch of rationalist writing, etc.

3.

The specific message I sent them after learning that were concerned about risks from hiring me:

FWIW I’m pretty familiar with feeling a sense of liability related to what allies or affiliates might say, it was a recurring theme at OpenAI. So I expect I have a reasonable sense of where you're coming from, and I'm happy to brainstorm ways to mitigate that sense of liability (e.g. not putting Resolution in my twitter bio). More generally, I want to be collaborative on this topic, and won't hold you to any commitments you guys have made if you want to re-evalaute them now.

I don't know exactly what risks you're most concerned about, but it might be useful for me to explain why I personally don't feel like I'm making a costly tradeoff against making AGI go well by opining on controversial topics online (and in fact believe that it's meaningfully helpful). I'll talk about my inside view and an outside view. Also happy to discuss any qualms you might have on a personal level/re office culture tomorrow.

Inside view: for better or worse, a lot of the strategic considerations around the future of AI are actually deeply entangled with the dysfunctions of the wider modern world. For example, academia's inability to grapple with either AGI or AGI risk seems like a very important dynamic to understand, and that seems quite related to how academia has changed over the last century. As another example, if we want to reason about how more capable entities might dimore sempower less capable ones, probably the best example we have is colonialism—and I think discussions of AI risk would be significantly less vague and abstract if people were less scared of honestly analyzing historical power struggles. (Daniel Kokotajlo's writings on the conquistadors as precedents for takeover are one big exception, but still miss a lot of the potential insights.)

Outside view: one of the biggest lessons of the last few decades politically, I think, has been that the costs of intellectual taboos are much larger than most people expected—e.g. they meant that the massive populist upheavals of the last decade took almost all intellectuals by surprise. I also think that AI safety is in a much worse position today than it would have been if it'd taken a more principled truth-seeking stance a decade ago (for example, AI governance would have been less likely to throw in with the Democrats in a way that alienated MAGA). Indeed, with respect to engaging with the current administration, it seems like standard EA views are more of a liability than any of the things you mentioned by a significant margin, and I expect that having a right-winger at Resolution would be a net gain in political capital. (To be clear, I don't think that this should be a significant consideration for you, I mention it mainly to highlight that the risks are actually more symmetric than they might seem.)

In general, though, I wouldn't want to join Resolution if I planned to spend a significant proportion of my time thinking about this stuff. Indeed, a key reason I'm excited about Resolution is because my research has lately been steering towards much more foundational-level questions that I previously expected, and so I intend to be pretty focused on pinning those down.

1.

Dialogues still seem possible to create through here.

1.

I will stop short of claiming this should apply to AI development as well since I haven't thought this through, inviting the readers to think about that.

2.

This is why jobs may feel like “bullshit” to the people doing them, even as they command high wages in the market. (Footnote Smith's, not mine—P.)

1.

I di

1.

Although, interestingly, yesterday I heard about some bacterium that is so adapted to endocellular parasitism that it lost the capacity to produce ATP and relies on stealing it from its host cell.

1.

Typically if funders don't like optics considerations regarding a grantee they're much more likely to just not fund something. Tbc I also think this is usually bad.

1.

partly through introspection + the Co

2.

A bit anecdotal but: there are ~ a dozen people who went to our college in 2017-2020 now working full-time in AI safety, which is much higher than other colleges at the same university. I'm not saying any of us are particularly "great" -- but this suggests social contagion / information cascade, rather than "we figured this stuff out from the empty string". Maybe if you go back further (e.g. 2012-2016) there was less social co

1.

If you doubt this, there's an advisor to my local school district who is a self-avowed Marxist and professor of education at the state university, and writes book reviews like this one:
«For decades the educational Left and critical pedagogues have run away from Marxism, socialism, and communism, all too often based on faulty understandings and falling prey to the deep-seated anti-communism in the academy. In History and Education Curry Stephenson Malott pushes back against this trend by offering us deeply Marxist thinking about the circulation of capital, socialis

1.

Quick proof sketch (this is a very standard result in RL and is the motivation for advantage estimation, b

1.

There are also more complicated possibilities, such as "if P runs then I want Q to run but if P doesn't run then I rather that Q also doesn't run" or even preferences that are only expressible in terms of entanglement between computations.

Richard Ngo's Shortform
Only the owner of a Quick Take's container can leave top-level comments on it. (Questions? Send an email to team@lesswrong.com)
945 comments, sorted by Click to highlight new comments since:

A few weeks ago, I was offered a role as a Staff Research Scientist at Resolution (formerly Sequent) to do agent foundations research. However, they later rescinded the offer due to concerns that having someone with my (publicly expressed) political opinions as an early hire might make it harder for them to hire other researchers. I want to give some brief thoughts on this, starting with the concrete details of what happened.

I talked to Jesse Hoogland, Dan Murfet, and Geoffrey Irving before they gave me the offer. These were fairly informal chats, since they all already knew me. I signed the offer a few days later, then soon afterwards I messaged Jesse about my politics curriculum.[1] Specifically, I suggested that I should run the next iteration of the curriculum before joining Resolution, so that people wouldn't interpret it as work that I was doing at Resolution.

Jesse noted some controversial aspects of the curriculum,[2] then set up a meeting with me, him, and Dan. I was expecting them to express some concerns, such as asking me not to run it (though I did indirectly mention the possibility that this might put them off from hiring me).[3] Instead, they directly told me that the... (read more)

I find some of Richard's attitudes very distasteful, and if I was running an org and thinking of hiring Richard I would sure be pretty scared about him being seen as a representative of the org or just being off-putting to employees and stakeholders; I think Resolution was absolutely correct to be scared about that. But idk how that should cash out. This situation seems unfortunate. :(

I find some of Richard's attitudes very distasteful

Examples? What attitudes, specifically? Can you link to something Ngo wrote that expresses the attitude you find distasteful? (Same question to any of the agree-voters.)

I feel like this should be easy. (I think that when I find someone's attitudes distasteful, I usually don't have a problem retrieving a link; I need the links to write a convincing critical blog post explaining why the views are bad, as I so often do.)

Taking a wild guess (I don't know if this is what you had in mind), if the distasteful attitude is that he thinks it's plausible that there are ancestry-cluster ("racial") differences in socially-relevant traits like cognitive ability (mentioned in footnote 2 of the OP), I think that's more of a "hypothesis" than an "attitude". (It's a claim about the world that could be true or false, not a claim about how we should behave or feel.)

If the distasteful attitude is that he thinks scholars should be permitted to reason in public about decision-relevant consequences of that hypothesis while also getting paid to do decision theory research, I can see reasons why someone would find that attitude distasteful (see my discus... (read more)

I said "attitudes" because I'm thinking about things like the propensity to retweet https://x.com/DoDeportations, rather than beliefs he holds. Many people around here find various taboo hypotheses plausible.

Is there something specific about the Department of Deportations account that you find distasteful, aside from their general enthusiasm about deporting people?

EDIT: while this is admittedly a somewhat defensive question, it was nevertheless a genuine question—I didn't remember what their tweets were like when I asked it.

Do you find these recent posts from the DoD account to be in good taste?

"Any nation that allows women to be judges inevitably ends up releasing violent felons and rapists back into the street.": https://x.com/DoDeportations/status/2085676339323945039

SICK TO DEATH OF BLACK PEOPLE repost, "The fatigue is real": https://x.com/DoDeportations/status/2085576421070475738

"Every single black would be in jail": https://x.com/DoDeportations/status/2085561849731326364

"We’re going to do things to you that have never been done before" threat: https://x.com/DoDeportations/status/2085530148753809697

"A black man not showing up to work? I don’t believe it.": https://x.com/DoDeportations/status/2085450882036985952

These are all from the past 24 hours.

Hmm, yeah, in hindsight when I asked the question I was implicitly conflating "distasteful" with something stronger like "reprehensible".

I'm happy to concede that they're not very tasteful in how they approach this topic (and FWIW wouldn't have/didn't retweet any of the above myself).

EDIT: I think people are misinterpreting this comment because they assumed I was already familiar with the DoD account's tweets. But I didn't have a cached recollection of the kind of stuff they tweeted when I asked the question (otherwise I wouldn't have asked at all, because... (read more)

I think these are quite reprehensible?

This seems like outright bigotry. Maybe I'm reading into it too much or projecting (I don't think I am), but this account seems to be gleeful in it's opposition to black people. (The "Every single black would be in jail" tweet, especially.)

Aside from the attitude being expressed, these tweets seem to be eroding extremely valuable (though already much decayed) liberal norms.

I could totally imagine having a stance against immigration (and for deportation), but even so, I think I would consider these tweets to reflect very badly on me and my policies? "Deserving of strong blame, criticism, and censure" seems right to me.

Screenshot 2026-08-07 at 11.45.12 AM.png


I think the tweets above are probably reprehensible by the norms of liberal elites from maybe 50-80 years ago. Unfortunately, these norms have not decayed in a random fashion, but rather specifically in ways that make them a useful vehicle for the illiberal left. I'll quote from my exchange with Kelsey:

Unfortunately, I view modern self-identified liberalism as being pretty hollowed out, in the sense that it keeps the trappings of actual liberalism but concedes most object-level battles to the woke left, pushing back on its excesses at each point in time but not challenging the foundations (and therefore continually losing ground for most of the last century). Some examples of lines it could have drawn:

- meritocracy (as opposed to affirmative action, disparate impact, etc)
- freedom of association (even if it’s “sexist” or “racist”)
- property rights (as opposed to environmentalist regulations)
- cultural compatibility as a key immigration criterion (as opposed to many things since at least Hart-Cellar)

So from my perspective, modern “liberalism” is in part defined by its lack of ability to pin down a position it won’t retreat from. Which then makes it hard for me to engage with it with

... (read more)

This feels like a fragile strategy, in the sense that it's good conditional on a specific theory of change / model about cultural evolution (that the malicious energy in those tweets will help people snap out of the deep preference falsification the anti-anti-normativity people talk about) and bad if that's false (in the straightforward virtue-ethical sense of being cruel and sloppy with the truth). I feel like there's a tension here with your writings about robust and gracefully failing strategies, pessimization/polarization, etc. Like, it feels like the maliciousness in the tweets is purposely pessimizing polite / leftist people's feelings. Curious how you resolve that tension.

(Also, less important, if you say one reason that you're not calling the tweets reprehensible is that the post-liberal norms will be different, does that mean you think that there's a decent chance that the correct post-liberal norms will say they're fine?)

To be clear, my claims aren't meant to be conditional on the idea that these tweets will be helpful. I do not consider the DoD account to be doing a robustly good thing, and that's related to why I'm not making tweets like that myself.

Instead, the thing I'm doing is more like inaction: people want me to say "this is reprehensible", and I am declining to make such a statement. In general, inaction is often a more robust strategy than action (especially when the "null policy" is as obvious as "decline to make a statement").

I think the main way that this strategy might be non-robust is if I'm declining to criticize the right while also getting triggered by analogous behavior from leftists and calling them names. To some extent I've made this mistake over the last year by e.g. not speaking up more about ways that Trump is behaving badly (e.g. various kinds of corruption), and I intend to fix that.

On a psychological level, I've spent the last few years trying to carve out space to be publicly honest about very controversial topics, and forceful on issues where it does matter, without wildly lashing out at the cowardice and malfeasance of the vast majority of Western elites. There's some... (read more)

Appreciate this reply! Your clarification makes sense. I'm reassured hearing you talk about your efforts to avoid polarization (e.g. with respect to Trump). I think you've done an impressive/inspiring job with it overall, at least in your online presence (like in the exchange with Kelsey).

Although I do think there's something deeper here that you're missing about the exchange about the tweets upthread, especially with this "inaction" concept. My advice definitely wouldn't be "be more careful", if anything it would be more in the direction of doing more - I... (read more)

You're welcome to talk about this publicly fwiw, up to you.

Great, thanks.

Basically, I do feel there's an energy in your initial replies of "defensiveness" or something? (not sure if you even disagree, but e.g. not checking the tweets, not mentioning that you only retweeted one tweet by that account and that you weren't super familiar with the account, not fully showing how you actually feel about the tweets, etc. Like, I imagine it would've felt too vulnerable to justify yourself to what felt like a disproportionate attack, or there was some feeling of solidarity with that twitter user, or something). And this ext... (read more)

Thanks for the comment!

Defensiveness obviously isn't immoral, but it's not "inaction". A "hackles raised"/"defensive" energy is something one is actively "doing to" the social field, it's not an absence of action. It's a purposeful increase in internal/relational tension, and an increase in the strength of one's boundaries to reduce vulnerability (i.e. let less information through, like conspicuously not mentioning relevant stuff). So this to me looks like adversariality, i.e. polarization.

Yepp, this basically seems right (and insightfully phrased). I do c... (read more)

Cool, makes sense!

I think the main problem with my other replies is that they weren't direct enough in owning that defensiveness

Yeah, I had something like this intuition as well but wanted to keep it simple in my original comment. One might more precisely say that there's some intrinsic adversariality in any ideological conflict, but the lever is more about not being reactive about it, processing it before putting it back into the world, making it conscious/overt so our system-2 can deal with it, etc. And that this generally leads to less intense polarizat... (read more)

pretty defensive, and a mistake for that reason (as was Zack Davis' original reply upthread).

What do you think would have been the superior play for me in that situation? When someone is deploying a "distasteful" epithet without specifics, I think asking for specifics (forcing them to take a stand on which side of the fact/value boundary the distaste is supposedly on) is the natural follow-up. (And then after he answered that fact-beliefs themselves aren't culpable, there was nothing to do except thank him for answering the question.)

Both your questions and mine felt very pointed. I think questions can be asked from a stance of genuine curiosity, versus a stance of "justify yourself!", and the difference is very noticeable and also changes the dynamics of the resulting conversation significantly.

If I'd had that curious stance, for example, I would have scrolled the DoD feed and then agreed that it was distasteful, saving myself from what has turned out to be a fairly unproductive and time-consuming thread.

[+][comment deleted]30

I'm only responding to one (minor) part of your whole comment here.

LessWrong has largely failed to track or oppose because a lot of it was justified in terms of suppressing the "far right"

I don't have that strong of a handle on what was happening with the broader rationality community, but this definitely doesn't describe my self-narrative.

I wasn't paying that much attention to this...because it wasn't promoted to me as an important trend until recent years. I still feel pretty uninformed about what happened (I'll read that post). I definitely would not have had any reaction like "oh this doesn't matter much / this is worth it, because it's punishing the far right." I have an internal grimace whenever anyone uses the word "misinformation", because it's so clearly a framing device that skips over the hard part of epistemology and actually figuring out what's true.

Rationalists did a better job with covid, but still largely focused on experts being object-level wrong, rather than trying to understand how and why networks of experts could have been so unethical and incompetent.

I'm sympathetic to claims that we didn't do enough, but Zvi at least wrote a whole sequence on moral mazes, a... (read more)

I wasn't paying that much attention to this...because it wasn't promoted to me as an important trend until recent years.

Yes, this is what it looks like when a community's attention is diverted. Similarly, academics might have said "AGI wasn't promoted to me as an important trend until recent years", but in fact plenty of rationalists were trying to promote it, it's just that academia as a whole was very good at diverting its attention away from weird people saying weird things. Same for LW and right-wing stuff.

I have an internal grimace whenever anyone uses the word "misinformation", because it's so clearly a framing device that skips over the hard part of epistemology

This feels directionally correct but far from sufficient. Like, if you were in the USSR and people started talking about misinformation or hate speech, thinking "oh man the authorities are using framing devices that are epistemologically unsound" would radically underestimate the scope of the problem. The thing you actually need to be tracking is the extent to which this is part of an implicitly or explicitly coordinated power play—see Arctotherium's article for much more.

Re Zvi, yepp, he's less bad at this than almos... (read more)

how much people in the past cared about "honor" (e.g. to the point of duelling to the death over it)

Honor cultures are not generally seen as enlightened liberal Western stuff, so I'm a bit confused by this connection. Would you call gang members shooting each other in the streets over small slights today an example of this good thing you're pointing at? (EDIT: Oh, I guess you mean specifically that they're more honest?)

I'd be interested in you saying how you feel (on a gut level) about the tweets if "reprehensible" doesn't feel like the right word.

E.g. I probably wouldn't call them "reprehensible" either since I generally don't use words like that for political stuff (it feels too "pretend-objective" or something, trying to imply/hyperstition a larger social consensus and leverage it, and therefore slightly manipulative - maybe that's also what you are trying to point at). I would probably say something like "oh, yeah I don't like those tweets :/ I feel a slight feeling of sickness and fear reading them".

Does anything similar apply to you?

Mused on it for a bit, and there's no particularly notable feeling.

Granted that they are not very tasteful in their approach I'm happy to call that agreed. Do you disagree with the content (rather than the form) of what they posted?

I think it's germane to the conversation to clarify your position in relation to the posts you haven't retweeted on an account you've otherwise endorsed.

I think it's germane to the conversation to clarify your position in relation to the posts you haven't retweeted on an account you've otherwise endorsed.

I dispute that retweeting something means that you endorse the account, as opposed to only the specific thing that you retweeted.

I think that's a fair point; I'm willing to stand corrected as over-reading his position if Richard would like to object. I remain interested in his response to my question regarding agreement/disagreement with the content of those posts either way.

I agree with Eli, me retweeting one post of theirs isn’t an endorsement of the account as a whole (and indeed one reason we got into this long thread in the first place was that I was sufficiently unaware of what their tweets were like that I didn’t realize what a can of worms I was opening by asking about it upthread).

Re the object-level stuff, a lot of it reads to me as exaggerated and distasteful but containing a kernel of important truth. E.g. I do believe that female judges tend to be soft on crime in a way that is corrosive to the legal system (see a... (read more)

Surely they are just basically wrong?

  • I don't imagine the causality between women judges and liberal policies is that strong (i imagine that liberal male judges do about the same?)
  • I can imagine a fatigue around homeless people, who are disproportionately black, but I don't think most people are "sick to death of black people"
  • I have pretty wide uncertainty around which races get treated more leniently for which crimes. I guess that for each race there are some crimes for which they would be treated to leniently. But "every single black would be in jail" seems ludicrously too strong and hugely inflammatory
  • The fourth seems like an odious threat.
  • The fifth, I guess that black men work at lower rates than white men, but skin colour is pretty weak evidence for predicting given what one could know. Like if someone was saying stuff like this and I were black I would ask why skin colour was the relevant factor here. It's kind of a pretty standard racial epithet if used seriously.

Like this stuff seems pretty straightforwardly inaccurate and inflammatory in a way that damages discourse. Isn't "reprehensible" pretty reasonable for that?

Now I don't think you retweeting the odd tweet from an acco... (read more)

Distasteful is what those of us who aren't attacked by these statements, and tend toward an impersonal and dispassionate reading, think of them. These are reprehensible and outright hostile to Blacks, Latinos, women, or any other group this account has attacked. You're the sort of guy to tolerate such language when accompanied by insight, but I don't feel like a meme account has much to offer in that respect.

[+][comment deleted]20

I probably don't want to engage further—and I think those tweets speak for themselves—but to be clear I'm much more bothered by the (1) cruelty/glee, (2) racial animus, and (3) lack of interest in truth than the general pro-deportation policy position.

And sorry to psychologize you, but I think there's a massive blind spot in your world-modeling if you are unable to predict that most people will find these tweets much more disgusting than pro-deportation policy views.

Oh yeah, I definitely would have predicted that most people will be outraged by these tweets. I just wasn't that familiar with the Department of Deportations account (I believe I've only retweeted that one tweet of theirs) and didn't scroll their feed before asking the question above (which in hindsight I should have).

Ah, the fact that you only retweeted one tweet of theirs seems pretty important. If that's accurate, 1) maybe edit to add that further up the thread and 2) Zach talking about your "propensity to retweet" then seems misleading - it made me assume that he had seen you retweet a lot of their tweets. (i.e. that you're probably familiar with the account).

I see. Thanks!

I want an emoji which says "this post is reasonable but it has been given too much weight". Like ZSP can absolutely give his opinion and bow out, but that does leave this thread as 'Richard holds some distasteful views as shown by this account which he has occasionally retweeted’ and I don't think that's much evidence. Like it is a true things from ZSP and a fair thing, but I think 115 karma is way too much given the actual direction of the discussion which follows it.

You seem to think that people are deferring to my opinion? My strong guess is that people have their own impressions, mostly based on stuff outside this thread, and I just articulated something that many people appreciate or agree with.

That is possible. I am uncertain.

The kinds of virtue that would make me comfortable with an organization doing a bunch of dual-use research seem pretty strongly anticorrelated with decisions like rescinding my offer. AI alignment will only become more central to the coming power struggles over AI, and being willing to give in to vague concerns about potential hires doesn't bode well for Resolution's ability to navigate those struggles in high-integrity ways.

I just wanted to say, as a guy sometimes sketched out or confused by Richard's politics, I agree with this.

I feel particularly worried about it about dual use, but, that's maybe just a subsection of: If Resolution is trying to be a place doing world class intellectual work, people there will need to think thoughts that go in directions that don't fit consensus, which are correlated with people finding uncomfortable.

I realize most of what it's trying to do is in a STEM-y frame that you might theoretically hope is orthogonal to politics. But I expect a lot of people to do their thinking about solving superalignment as part of their thinking about AI macrostrategy, and in general the lines here are kind of blurry.

I also just think... from a sorta normal politics ... (read more)

Talking to Richard and hiring Richard are two different things. It's plausible to me that all these things are true simultaneously:

  • Richard would do a great job at contributing useful research directions / thoughts / etc.
  • Richard would be great for the org culture and treat everyone well / respectfully / etc.
  • Richard's politics have substantial admixtures of attitudes that are dumb (both incorrect and harmful), and indicative of general attitudes / thought patterns that are reasonably taken as hostile / toxic / etc.
  • Other good people have low thresholds for that sort of indication, for good reasons, and in fact would be less likely to join and/or fund the org if Richard is there
  • The expected net effect of hiring Richard is negative, including integrity / upholding principles (it's not necessarily unprincipled to be like "I think Richard's ideas are good enough that it's worth some tail risk, but I get that lots of other good people wouldn't have that information, and furthermore have good reason to distrust him for political reasons, so I won't hire him")

I would not guess that all of these are true, but I wouldn't be so confident without, for example, having talked to several potential hires for the org.

Yeah the better comment I'd like to have written engages more with the tradeoffs here. I totally agree there are (or at least might be) real tradeoffs here.

I would not guess that all of these are true, but I wouldn't be so confident without, for example, having talked to several potential hires for the org.

Absolutely I expect there to be potential hires who are turned off. But, I think the current amount that people expect to be able to work at companies without dealing with people who have politics that feel crazy and bad, are waaaaaaaaaaaay out of whack... (read more)

Mainly I meant to say "you probably should have substantive uncertainty about the extent to which it's a sign about their capability, because (I assume) you don't know the sorts of people they're trying to hire and how those people orient to politics and whether that's a good orientation or not".

I would be quite surprised if I were surprised (upon learning more) about what sort of people they were trying to hire, or how those people relate to politics.

I could still totally be overconfident about the overall frame here but, like, I'm already assuming "they are basically correct about the immediate implications of hiring Richard." Is there a particular thing you're imagining I might not be imagining?

If your org culture can't talk to people like Richard, you are implicitly throwing out like half of America?

Resolution is the kind of org that has approximately all its conversations with career AI people, who are demographically likely to be left-wing, and likely to be closeted if right-wing (to avoid the career consequences that Richard demonstrates). I doubt they're throwing out more than 10% of their pool.

My guess is Geoffrey do agree that it is important for AI safety orgs like Resolution to be more diverse intellectually, and it is in fact the reason why he made that decision. Richard himself notes that "[Geofrey]'s excited about hiring academics who are new to alignment, and worried that those people might not join if they hear about such tweets".

Neoreactionaries (not claiming Ngo would necessarily identify as NRx specifically, but he is broadly in that direction, and it is dishonest to claim him as a reasonable stand-in for your average US Republican vo... (read more)

Richard's viewpoints seem explicitly incompatible with a majority of the world and a plurality of Americans; if your goal is getting the best researchers overall, a coworker who is actively hostile and antagonistic to approximately 84% of the global population seems like a bad way to cast a wide net.

This is to say nothing about his views on alignment (while I think one could argue that ASI aligned to Richard's point of view would be unaligned with most of humanity, it's irrelevant here); he could likely contribute best to the alignment debate from a more scoped organization.

actively hostile and antagonistic to approximately 84% of the global population

How did you get this number?? From what I know about Richard Ngo's positions, it seems a very surprising estimate (but maybe I'm missing something).

The org is trying to hire Agent Foundations researchers. Suppose that becoming such a researcher requires being educated, but the education system usually infects the candidates with leftism in a manner which doesn't affect their ability to make advancements. Then Ngo would end up hostile to 84% of potential candidates and not to 84% of the population.

As far as I understand Ngo, he worries that leftism either does affect the ability to make advancements or is a demonstration that the researchers are stumped by lack of diversity. For example, if the researc... (read more)

I think "actively hostile and antagonistic" is unfair, but even if it were true, majority and might do not make right, in politics, rationality, AI alignment, or anything else. There are plenty of important issues and beliefs of all kinds for which most people are deeply mistaken about. Coming to terms with this fact can sometimes make people cynical or despairing, or make them come across as hostile / alienating, but I don't think that's actually true of Richard or most other folks around here.

(OTOH, not acknowledging that most people are wrong about some... (read more)

Looking over the [edit: specific] cited tweets and tweet conversation, I don't get it.

This seems really quite tame? Richard is laying out an abstract taxonomy of views nationhood, and critiquing them. Then he has a cordial exchange with Kelsey, in which she requests and he attempts to pass her ITT (and does a good job according to her). This all seems polite, reasonable, and well within the norms of online intellectual political discourse?

I notice I'm confused.

Are there more tweets that I couldn't find?

Do I radically misunderstand what reasonable people find offensive?

Is even this level of political analysis verboten in conventional professional circles?

Should I update that resolution has really different standards than I do, or is exceptionally cowardly with regard to possible social attacks?

Do they have a spidey-sense that Richard endorsing the "national right" view is too close to ethnonationalism, which is too close to racism, and that potential employees wouldn't want to work for Resolution, because Richard is also employed there? (To which I'm like, "Are you kidding me? Even if that's true, why do you want to work with such people?")

Are their real concerns about Richard ... (read more)

I agree the linked tweet is quite tame. But I looked at the 20 most recent tweets/RTs on Richard's timeline and they include a RT about the Great Replacement, a RT of the "Department of Deportation" about non-white sex criminals and a RT endorsing race realism.

Indeed, I was expecting stuff more like that when I clicked the link.

Why was this interaction cited as damning, instead of much more overtly edgy stuff? Were they just embarrassed to even bring up that other content, or something, and so pointed to a nothing-burger instead?

My guess:

1) The cited tweet got more attention (see the interaction with Kelsey Piper)

2) That tweet is by Richard himself whereas the ones I cited are all RTs and I think it's reasonable to form less strong opinions of people based on what they RT than what they post under their own name.

Do I radically misunderstand what reasonable people find offensive?

Yes. Most people make decisions about exclusion on the basis of coalitions and alliances and affiliations, not on the basis of the literal content of the person's beliefs.

Do they have a spidey-sense that Richard endorsing the "national right" view

Yes. They accurately discern that he is a member of the other coalition and so they don't want to hire him. See e.g. Zach Stein-Perlman elsethread saying he agrees with Resolution's concerns not because of Ngo's beliefs, but because of which other people Ngo affiliates with. (As I write this the comment is at 29 agreement points with 8 votes, so some others strongly agree with Stein-Perlman's view.)

and that potential employees wouldn't want to work for Resolution, because Richard is also employed there?

No. That whole line of argument is a dodge. The bottom line was written because Ngo is an outspoken member of the other coalition. If this dodge didn't work, they'd move to a different one instead.

It's not some utilitarian effort to achieve the company's hiring goals most efficiently. Lots of people just want to exclude their political enemies. This is very common. In my life... (read more)

It's not some utilitarian effort to achieve the company's hiring goals most efficiently. Lots of people just want to exclude their political enemies. 

This seems like a surprising and implausible hypothesis, because if it were true, I would have expected them to not make a job offer in the first place, rather than make and rescind an offer.

You're a lot less likely to overlook who your first order political enemies are, when hiring people, than you are to overlook second order effects like "will other people find this guy offensive and not want to work for u... (read more)

What's implausible about the story in the OP?

I signed the offer a few days later, then soon afterwards I messaged Jesse about my politics curriculum. … Jesse noted some controversial aspects of the curriculum, then set up a meeting with me, him, and Dan. … I had a long conversation with Jesse and Dan, and later a long conversation with Geoffrey. In the latter I learned that, although the decision was prompted by me sending them my curriculum, Geoffrey in particular was more concerned about my tweets.

So, they hadn't read the curriculum, and then they did. At that point they learned where Ngo stands and changed their minds. If they had seen the curriculum beforehand then yeah they wouldn't have made the offer in the first place. I'm sure they would have preferred that. No mystery here. Perhaps in the future they'll do deeper political vetting before making offers.

why did they cite this tame exchange, instead of the much more conventionally offensive stuff that Richard supports

That's not what happened. In the quote from Ngo's OP above, Hoogland explicitly cites the "much more conventionally offensive stuff" as the reason. My read of "although the decision was prompted by me sending them my curriculum, Geoffrey in particular was more concerned about my tweets" is that probably Irving didn't read Ngo's Twitter until after they saw the curriculum and flagged the political concern.

Do they have a spidey-sense that Richard endorsing the "national right" view is too close to ethnonationalism, which is too close to racism, and that potential employees wouldn't want to work for Resolution, because Richard is also employed there? (To which I'm like, "Are you kidding me? Even if that's true, why do you want to work with such people?")

Yes.

I think there are non-cowardly reasons to view someone's political opinions as disqualifying for a senior role in AI alignment research. It would be cowardly to say "your views are controversial and will turn people off", but not to say "your views are reprehensible and cause us to genuinely trust you less to do a good job at work that is not just narrowly technical but creative and philosophical and sometimes even moral in nature."

Now, you did say they explicitly talked about optics (in terms of recruitment). In general I am pretty down on doing things because of optics, though I do concede that it can be instrumentally necessary for getting things done in this messed-up world. But it's also very possible that the object level matters to them too. I don't think all views are equally worthy of taking a costly stand for.

I would be pretty surprised if the people thought the views were actually reprehensible? Or if so, I would strongly suspect the perceived reprehensibility of it would be downstream of a rationalization, not actually thinking about it.

But IDK, maybe someone should make a case for that sometime? I have strong disagreements with Richard, but I don't really see a strong case for his views on these things being morally reprehensible in any substantial magnitude.

Descriptively, many people are more prone than you to finding political disagreement reprehensible, and Richard has exactly the kind of politics that many people find reprehensible.

I could make this case more extensively with citations to opinion polls and such, but unless you live under a literal rock I imagine that would only be telling you things you are already aware of, so I'm not sure why you'd be surprised here.

Like, when I talked to Richard he seems to have opinions milder than what like 35% of Americans on the conservative right believe? IDK, like, I disagree with those as well, and common people have many views I think are very dumb, but I am kind of confused what people are pointing to in the moral reprehensibility.

Like, his beliefs seem much less outlierish or reprehensible by common-sense morality from most people's perspectives than all the post-humanists and successor-species and negative utilitarians and AI capability maximalists and e/accs and all kinds of other things people around us tolerate, so I am very skeptical this is downstream of some genuine model of "moral reprehensibility" as opposed to a model of "what would get me attacked by my potential political allies".

Like, when I talked to Richard he seems to have opinions milder than what like 35% of Americans on the conservative right believe?

Have you not met the people who say that 35% of Americans on the conservative right are reprehensible fascists? What do you think is going on in the heads of the potential political allies who would attack them, if not that?

Sure, I have seen people say that, and usually it’s strong evidence that what is going is tribal politics, not someone having any real model of morality that indicates someone is likely to be bad ally, or of low integrity, or beliefs things that are important and you have actually strong justified belief in that they are wrong.

Maybe you disagree? Again, I would be fine just going into the object level.

Without going into the object level, per your own number, "the 35% most extreme members of the ideological camp you disagree with are being outright reprehensible" doesn't seem like an insane thing to believe at all?

I don't think that's insane! But I think at a policy level refusing to coordinate/work with 70%-ish of the population (if you are being symmetric about it) is probably a pretty bad mistake? At least for this kind of stuff.

Like, seems like this would be an orthogonal dimension from competence, intelligence, integrity and other things you also want to select on, so seems like you would be very heavily constraining the set of people you would want to collaborate with. Which I mean you can do, there are a lot of people, but does seem like it would be super costly, and in some sense non-viable (like, it's hard to maintain any kind of cooperative stance towards the world if you consider the majority, or close to the majority of people to have opinions so reprehensible you need to actively distance yourself from them).

I appreciate your sincere and principled engagement, but I also want to note that there's something oddly mistake-theory-ish about a bunch of your comments. I first noticed it with the "reprehensible" comment, where it seemed like you were assuming a bunch of common knowledge that didn't actually exist—like you were using terminology that doesn't make sense to use when talking to an audience that would downvote that comment to -61.

And now here, where you say "seems like this would be an orthogonal dimension from competence, intelligence, integrity and other things you also want to select on" as if that's something other people will nod along to, whereas it seems more likely that most people who read this will think "no it's not, my political enemies are actually just low-integrity and incompetent and dumb".

It seems like there's something important about the dynamics of leftists and liberals that you're not modeling right now. It's a bit analogous to a thread I had with Kelsey where I called her out for continually using the phrase "I don't understand why" because it seemed obvious that she was doing motivated cognition to block out her actual hypotheses. My best guess is that you'r... (read more)

? Where are you getting the 70% number from? If we're 'being symmetric around it' then the 35% most extreme of both ideological camps cannot combine to be more than 35% of the general population. Obviously?

Ah, sorry, that was me being unclear/being bad at writing. I intended to communicate "he seems to have opinions milder than what like 35% of Americans (e.g. most on the conservative right) believe".

Your read of my sentence is definitely more natural though, so this was definitely me failing to communicate, sorry for the confusion!

Oliver meant the (35% of Americans) [who are] on the conservative right, presumably referring to President Trump's approval ratings, not 35% of (Americans on the conservative right).

Without going into the object level, per your own number, "the 35% most extreme members of the ideological camp you disagree with are being outright reprehensible" doesn't seem like an insane thing to believe at all?

If we assume the camps (left/right) are split 50/50, then it would follow that 35% (35% of 50%, the 17.5% most left-wing plus most right-wing fraction) of the population "are being outright reprehensible", i.e. more than a third of the entire population. That seems to be a rather extreme view.


For example, most academics and journalists are far ... (read more)

Most academics and journalists are Democrats, but "most of them would fall into the 17.5%" doesn't follow. I'm not particularly interested enough in this tangent to dive in the FEC data myself, but per the articles Claude found about it, academics and journalists disproportionately did not donate to Sanders among the presidential primary candidates.

Most people don't donate to anyone, so this is not informative. We know that there is a heavy imbalance between left/right faculty in academia. For example, [29% of Harvard faculty say they are "very liberal", while only 1% say they are "very conservative"](https://www.foxnews.com/media/fewer-harvard-faculty-members-identifying-liberal-although-staff-still-strongly-left-leaning-survey). [Many departments employ zero registered Republicans or show heavy imbalance in registered Democrat/Republican ratio.](https://www.ff.org/nearly-four-in-five-college-depart... (read more)

No I basically agree that such people are doing silly tribal partisanship and in some sense "not actually thinking about it". I'm not sure if our disagreement is semantics or theory of mind, but I think that whatever pejoratives we throw at that kind of cognition, it still outputs real conclusions that people really believe.

"This person's politics mark them as morally unqualified to work in this career" is an argument people sometimes make, and whatever is going on in their heads seems closer to sincere belief than "I am playing a Machiavellian power game in which I say whatever it takes to secure political advantage by getting this guy fired".

Cool, yeah, then I think we are just arguing semantics. I was intending to include that in this second sentence:

Or if so, I would strongly suspect the perceived reprehensibility of it would be downstream of a rationalization, not actually thinking about it.

I think people's rationalization can result in things pretty reasonably described as "sincere belief" so I don't disagree. I just also think it reflects badly on you if you end up doing that. Hence my belief "I don't really see a strong case for his views on these things being morally reprehensible in any substantial magnitude".

Fwiw, I'm planning to write a sequence making the case for those views being morally reprehensible (among other theses on the relationship between politics and rationality), but it may be a while before the sequence is out due to busyness/akraisia.

It would be cowardly to say "your views are controversial and will turn people off"

Would it? If God came down from the heavens and told Resolution that X many qualified candidates would refuse to join their org if they employed Richard, surely there is some value of X that makes it non-cowardly to say "We are not hiring you because your controversial views will turn people off".

Lacking divine insight, Resolution must have made their own estimate of X and decided it was too high. This is of course the sort of thing they would say if they were a bunch of di... (read more)

Disappointed to hear about this. Sounds like a really stupid decision from Resolution.

While I'm not sure I entirely agree with the decision, I don't think it's obviously really stupid. Richard's views, whether you agree with them or not, are very controversial and I think it's plausible having him in a senior role would indeed discourage others from working for Resolution - especially people who are moving into alignment work and don't know Richard's background.

Richard's views, whether you agree with them or not, are very controversial

They are very controversial only for certain subgroups like academics, who are far more left-wing than the general population, but not for the general population itself.

Richard pretty clearly believes in some form of race realism/HBD which is just about the most unpopular view you can hold in modern society. Even MAGA won't go near it

"Race realism" is a completely mainstream idea. People in the US routinely classify people by race, statistics do the same, medicine routinely also does so (while often using the synonym "ethnicity") when it is medically relevant, approximately nobody doubts that "East Asian" people exist and are different from "Europeans" etc.

I don't know what you mean exactly with HBD, but opinions on race differences in intelligence are in fact quite mainstream, in fact more so than many other fairly common beliefs, e.g. that eating meat is wrong.

What? This seems like a very left-leaning view. Believing in substantive relevant genetic differences between human subpopulations is really just super contested on the left. It's also obviously correct, and e.g. Scott Alexander obviously subscribes to it.

We don't have to go into IQ discourse, but of course Kenyan's are faster runners than other human subpopulations, for largely genetic reasons. What on earth would it even mean to not believe in that?

To be clear by 'race realism' here I am just referring to the IQ discourse. I agree that other differences like the Kenyan distance running example are more widely accepted. I mean, in that Scott Alexander post you cited, the second paragraph is about how controversial the research is.

To be clear I'm basing Richard's views on the essays in the curriculum he wrote with Samo Burja (https://www.21civ.com) which includes A Guide to the Heriditarian Revolution by Nathan Cofnas, a piece which led to Cofnas being essentially forced out of Cambridge for writing.

Cool, I agree that the IQ stuff is more controversial, but my guess is mostly downstream of the same left-leaning political forces. Also, yes, it's very hard to accept substantial genetic differences in a huge number of human traits, but not to accept it for IQ, so in as much as the broader hypothesis here is widely accepted, trying to specifically contradict that on the IQ question seems like an uphill battle (not impossible, of course, weirder taboos have taken ahold in the past).

Hmm, this makes me worry that "fairly senior role" might imply that I was going to be running a team or managing people, which was not the case. Have edited accordingly.

Very disappointing, sad to hear this!

I also think that AI safety is in a much worse position today than it would have been if it'd taken a more principled truth-seeking stance a decade ago (for example, AI governance would have been less likely to throw in with the Democrats in a way that alienated MAGA). Indeed, with respect to engaging with the current administration, it seems like standard EA views are more of a liability than any of the things you mentioned by a significant margin, and I expect that having a right-winger at Resolution would be a net gain in political capital. 

I'd want to argue that this is in fact not the case, and AI safety is in fact in a reasonably good (though not perfect) position now, and a lot of this is because we do have bipartisan AI safety bills that basically every faction from Democrats to MAGA can agree on like the FRONTIER act by Trahan and Olbernolte, and Trump is sufficiently scared by Mythos that he has started to restrict AI capabilities (unfortunately they've generally targeted external deployment, which is a poor choice because in the worlds where it matters, the alignment problem is much more likely to be solved by default)

I do believe that AI safety will become partisan by 2... (read more)

I think this feels similar to the thread with LongView Philanthropy (linked below). I don't think this reflects positively on any involved party.

https://www.lesswrong.com/posts/FuGfR3jL3sw6r8kB4/richard-ngo-s-shortform?commentId=RHDvund5npvwWgyAz

This seems very disappointing

  1. In hindsight, given the poor track record of large alignment-branded research organizations, it was a significant mistake for me to accept Resolution's offer without having thought much more carefully about ways that joining might be bad (such as the points above). There's a common failure mode whereby rationalists and EAs are averse to thinking about how their cognition is skewed by incentives and power dynamics; though I often talk about this issue abstractly, I still failed to account for it in this case.


I'd be very interested in a 5-whys on this.

There's a common failure mode whereby rationalists and EAs are averse to thinking about how their cognition is skewed by incentives and power dynamics

Man, I do often feel that rationalists, and especially EAs are too willing to be credulous based on social signals that aren't all that epistemically reliable (and I plan to write about this sometime). But it also seems like rationalists and EAs do better at this than almost any similarly sized subculture?

Most people just uncritically adopt the views of their social peers, without almost any reflection!

The o... (read more)

Good point, I've added a parenthetical: "(especially relative to how clearly these groups think about other things)"

But it also seems like rationalists and EAs do better at this than almost any similarly sized subculture?

What is the relevance of this? It is written of the eighth virtue of humility that "it is useless to be superior [...] There is no guarantee that adequacy is possible given your hardest effort; therefore spare no thought for whether others are doing worse."

I think most onlookers would interpret Richard as saying "Rationalists and EAs are unusually bad at this". Insofar as Richard thinks that, I want to publicly dispute it. Insofar as he doesn't, and this is a miscommunication, I want to correct the record.

That rationalists are better than average does not exempt us from the problems caused by our not being remotely good enough, which is why I plan to write about it.

Well, in theory it could for example suggest, directionally, whether it's more a "my rationalist tools are breaking in my hands" situation vs. a "I have failed to live up to my rationalist tools" sort of situation.

On a different note, I've become increasingly worried that Resolution's plan to automate alignment research will lead them to produce a lot of capabilities progress (since "automated alignment researcher" and "automated capabilities researcher" are such similar things to aim for).

You’ve been in favor of interpretability research in the past, wouldn’t that similarly increase capabilities?

It seems to me that the decision not to hire you didn't cause that many problems.

  1. The right-wing analytical framework seems to have major actual issues, and I expect some of them to be avoidable, e.g. via studying non-Western data and sociologists' works. For example, the feminization of academia seems to have happened in the USSR, but didn't cause wokeness in Russia and was partially reversed, and postmodernism's hazardous forms which caused many problems in the West don't seem to be as widespread in Russia.
  2. A post-AGI world would cause us to rethink ethi
... (read more)

FWIW, as a generally leftish post-agi-thinker:

  • I'm not totally sure what you mean by "the right wing analytical framework" or what it having issues implies
    • If you're saying that there is one right-wing analytical framework, which richard has, and it uniquely has issues, such that it is undesirable to use it to reason about the future, and therefore it would not be useful for sequent to have him (or people like him) on board, I think I disagree with ~all the premises and implications
  • I do not think post-agi means rethinking ethics specifically towards the left; it seems like many of the major questions are more 'right-coded', especially around republicanism and liberty in the face of overwhelming power, or around the appropriate degree of social/cultural conservatism (in the technical sense)
  • I would be wary of comparing mainstream leftism to right wing conspiracy theories
  • (Again, as someone who is mostly a leftist) I wish people working on constitutions and character were more attentive to the pitfalls of leftism, and I expect this generalises to alignment solutions

[ed: we are discussing this more over private messages]

Are Richard’s views common in the AI Safety community? And is it in general common to be right-wing? If so it’s pretty surprising to me. I‘m happy to elaborate, but don’t want to waste time in case no one engages with this comment.

This should be a top level post.

Was there meant to be more at the end of the last footnote?

Ooops, no. There was meant to be two words less, so I've deleted them.

Thanks for writing this up.

Looking over these, it seems like the hits that have landed on Richard are that he has occasionally retweeted an accounts that have tweeted some bad stuff. This doesn't seem like the sort of thing that I would call for cancellation (and I think that's what job offers being rescinded is) for.

I tend to dislike comment threads where it's implied that there are big hits taken but when one chases them down, they turn out to be nothingburgers. It's fair for Zach to bow out at this point, but that does leave the argument pretty weak to me.

Personally I'll say tha... (read more)

What? I'm obviously not trying to do a Richard-takedown or "call[ing] for cancellation" or "impl[ying] that there are big hits taken." And I don't recall anyone else in this thread doing so. There are points to make about Richard and Resolution and the world outside of taking down various parties. I think you might be insufficiently decoupling here.

"call for cancellation (and I think that's what job offers being rescinded is)"

I say here that what I am implying is cancellation is Resolution's actions.

"tend to dislike comment threads where it's implied"

I am talking about the whole thread, not your comments specifically.

I don't have any particular criticism of you here. Seems useful for you to say your opinion, with evidence, then choose to bow out. But I do think there is something wrong with the process in general.

Why would they go through with hiring in the first place, your politics curriculum is separate from your work and also quite similar to your general public posting? This feels so sloppy by them. Even the reason for renaming / rebranding from Sequent was flavoured in the same way.
Doesn't feel good and don't agree with their decision.

Around 8 years ago Dario Amodei, Jan Leike and a few others predicted that we’d probably have AGI by now. Since then, AI capabilities have progressed so fast that many people updated that the short timelines advocates were right. And yes, their expectations about the speed of AI progress were far more directionally correct than almost anyone expected. But we don’t in fact have AGI (even under relatively weak definitions, like a reliable drop-in replacement for central examples of white-collar jobs), and the world would look very different if we did.

Now a large proportion of the AI safety community is implicitly or explicitly orienting to futures where an intelligence explosion occurs within a few years. My default expectation (absent an extensive pause) is that a similar thing will happen: they'll turn out to be directionally correct (relative to the expectations of almost anyone not linked to the community) but factually wrong. Specifically, we won’t have superintelligence within the next 8 years, but things will still be moving so fast that it’ll *feel* like the people who argued for short timelines were right.

I’ll write about this more extensively soon, but I wanted to say somet... (read more)

155211

i still think the short timelines people are too short timelines, but they were more correct than me a few years ago. my timeline used to be around 2035, and i've updated to 2031 over the past year. for me the huge update was when coding basically mostly died earlier this year. i used to be super pessimistic about model coding, and actively skeptical of people who claimed to have the model run all of their experiments. but i think this was a mistake, and in large part because writing code is a large source of meaning for me as a person and it was unpleasant to admit that it was on the way out; and as a relatively good engineer, it took longer for avoiding reliance on AI to become untenable for me. but at this point i basically do not write a single line of code by hand anymore, and any cope i had about being better at architecting code than the models feels like it's rapidly aging very poorly. i occasionally manage to get better stuff out of the model than other people because of my understanding of the stack, but those moments are getting rarer. i wasn't expecting to get this level of coding automation for several more years.

i agree some people should work on work that doesn't assume RSI soon - at the very least because we will hopefully have coordination that buys us more time, and it would suck if nobody was planning to take advantage of that time. but it also feels like cope to ignore this.

I doubt Richard is proposing to ignore coding automation trends; I took him to be arguing that recent progress isn't necessarily on track to deliver on AGI in a few years. I'm also skeptical of confident short timelines, especially on the basis of these kinds of arguments. The relationship between "coding automation" and "AGI" seems tenuous to me in a way that people often brush past. E.g. people often talk about "coding automation" as one coherent thing, much like people talk about "horizon length" as one coherent thing. But these are massive clusters of tasks spanning all kinds of knowledge work, and the kind of knowledge work AI automates might matter a lot for how quickly RSI goes (or if it goes at all).

Also, I agree with Richard that the disconnect between measured ability and real-world impact is weird; I think this should count some against the usefulness/powerfulness of this paradigm, and the counter-arguments I've heard aren't very good. E.g. a popular one is that every automation step just moves us to a new bottleneck, such that despite models becoming more powerful we still might not end up seeing much increased output from the organization/economy overall. But this l... (read more)

I pretty strongly disagree with you here fwiw. I look forward to reading your argument when it's ready. For now, I'm dismayed at how many people seem to be agreeing with you.

fwiw, I've previously had much more weight on short timelines before[1] (I was already working on automated alignment in 2022 partly for this reason and have kept at it since then!), but the distribution has certainly expanded.[2]

Largely, I've found that many people arguing for short timelines have collapsed into lazy thinking, relying far too heavily on "the models are getting better" and "those who predicted longer timelines have continued to be proven wrong" (I'm not saying this doesn't count as bits to update on). I don't say this (and haven't said this) to disparage the short-timeline view (I still place considerable weight on it), but I want to encourage clearer writing and more convincing arguments (not just for myself). Ultimately, I find many arguments lacking in the mechanistic description of where current capabilities come from and how that relates to RSI and "True AGI."

Personally, my core reason for fixating on this is how much the details matter for understanding progress on superalignment, and how much hand-waving can make much of the AI safety field focus on entirely the wrong things! It may be true that timelines are short (again, I also can imagine quick paths to ... (read more)

(For the most part I have stopped arguing about this because approximately no one seemed willing to defend the confident short timelines view in a public debate, with the notable exception of Abram, despite me putting this challenge to people a number of times (not sure how many). But for the record, AFAICT everyone with confident short timelines is overconfident, i.e. their evidence doesn't match their degree of confidence. This continues to seem to me to have practical implications for resource allocation, leading to mistakenly underinvesting in long-term interventions.)

approximately no one seemed willing to defend the confident short timelines view in a public debate

Now that you mention it, I'm feeling pretty interested in trying to devil's-advocate short timelines / red-team my most bullish-on-LLMs views. I think I grok the case for them pretty well; or, at least, there are some short-timelines cases that I think hang together well enough for me to be worried about them. Would you be up to it? Same format as with Abram.[1]

(Though I can see it ending up with me speculating about possible ways to advance capabilities in ways I wouldn't want to make public. Hopefully at worst we'll need to edit out a few paragraphs, though.)

  1. ^

    Dialogues still seem possible to create through here.

I think I'm interested, though significantly less interested than in a defense from someone who actually holds the view. It sounds like you could maybe occupy some AGI-soon hypotheses well enough, so could be good. I'm super busy at the moment--maybe we can try in a month or two? Or if you like you could start the dialogue post thing, and then we can slow-cook it (like, a "correspondence game")?

Or if you like you could start the dialogue post thing, and then we can slow-cook it (like, a "correspondence game")?

Yeah, that sounds good. I'll figure out my opening statement and do that within probably a couple days.

What are your timelines?

See: https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#Timelines And: https://www.lesswrong.com/posts/5tqFT3bcTekvico4d/do-confident-short-timelines-make-sense

Since then I've probably updated longerward based on a bit more hope in a slowdown, and shorterward from (logical and empirical) updates regarding neuro research (see https://www.lesswrong.com/posts/MnroTdcCCZoFSXEHy/a-case-that-whole-brain-emulation-research-is-net-harmful-by). Maybe they roughly cancel out, or rather, concentrate mass in 20-40 years a little bit more, relative to before? Overall it just feels like "mostly no one has any idea basically".

(To be clear, it makes sense to be freaked out by gippity behavior, and we definitely should have a global ban on AI R&D yesterday.)

I worry about the next decade feeling like a Shepard tone: an auditory illusion of a sound that's continually going up.

This is a great analogy, and is pretty good articulation of a model of the world that I have ~25% probability on.

But I do want to keep in mind, even under that model, the world is being dramatically transformed, industry by industry, and at a cultural level, as the AI capabilities that already exist become central to how every part of the world works.

What happened to software engineering this year will happen to many more domains, and to a bunch of slices of the world that we don't currently consider as their own "domain".

What happened to software engineering this year will happen to many more domains

This is interesting because I would say that I have been surprised at the seeming lack of transformation of the software engineering industry this year? I think that in January/February 2026, there was a widespread expectation that we would see major layoffs this year, which don't seem to have materialised (of course there have been layoffs, but they don't seem to have been much greater than in previous years, and below the highs of 2022-23). I still don't really have a good explanation to reconcile the reported transformation of the day-to-day work of SWEs with the fact that the industry doesn't seem to be changing much, except for companies being surprisingly slow to cut headcount. They seem to have been slower to cut headcount in response to this than they were in responding to the Fed raising interest rates in 2022-23, for example.

The consumers are buying software services not lines of codes. Coding agents made lines of code cheap but the process of shipping a service have a lot of human-related bottlenecks so Amdahl's law applies.[1]

The example of radiologists is often invoked and even received pushback on this website but the fact that workers moved to different tasks when some were recently automated can also be illustrated by travel agents and translators, as Noah Smith points out: https://www.noahpinion.blog/p/ai-keeps-stubbornly-refusing-to-take

It seems like we’re uncovering a consistent human blind spot here: People don’t actually know how they produce value at their jobs. Modern jobs are much more than a simple collection of tasks — they are pieces of a complex machine that produces value in ways that an individual worker often doesn’t even see.[2]

  1. ^

    I will stop short of claiming this should apply to AI development as well since I haven't thought this through, inviting the readers to think about that.

  2. ^

    This is why jobs may feel like “bullshit” to the people doing them, even as they command high wages in the market. (Footnote Smith's, not mine—P.)

Around 8 years ago Dario Amodei, Jan Leike and a few others predicted that we’d probably have AGI by now.

Do you have a link for this? It really matters whether they were like "it's really quite plausible we will have AGI by now" (which aged IMO very well) or "we will definitely have AGI by now" (which would have aged relatively badly).

Unfortunately I do not. Jan had an internal presentation at DeepMind around 2019, I might ask him for the slides. My recollection is that in it he aggregated a few different methods, and ended up with 6-ish year timelines, but I could wrong (e.g. maybe the 6-ish year timelines were just one method rather than the all-things-considered prediction, or maybe they were actually 8-year timelines, or...).

I guess there's another question which is: how did the short timelines vibes actually propagate into the community? There was Daniel's What 2026 Looks Like; there was the Big Blob of Compute doc (though I don't think that had dates?) Maybe it was in-person conversations? (From Daniel's discussion with Ege and Ajeya, he makes this prediction: "Median Estimate for when 99% of currently fully remote jobs will be automatable: 4 years". That would be November 2027, maybe with a bit of leeway for rounding.)

Partly I'm indexing on the mental recollection that people had shorter timelines than Shane, and Shane's were always around 2027. But this is all messy and it would be really nice if we had more evidence about Dario's views in particular, since I think they were pretty socially influential in illegible ways.

Two more datapoints about short-timelines predictions:

  • I don't recall exactly how the Superalignment team's 4-year deadline was chosen in 2023, but that was something to do with their timelines estimates (maybe it wasn't the median/mean though).
  • The OpenAI Policy Research team under Miles had little timers on our desks counting down until (some estimate Miles generated of) when AGI would arrive. I don't remember what the estimate was though, I believe less than 6 years (and we got them maybe in 2022?)

I went with Dario and Jan's rather than other people's because I feel like they were advocating for short timelines earlier than others were.

My thought on it: culture and adoption lags capabilities, A LOT. By far most people don’t know there’s models more capable than free-tier Chat Jippity and have never engaged with something Astra/Fable class. To understand the frontier is one massive hop, and to figure out what to do with such capabilities is yet another massive hop. You can’t forget the people active in the AI communities are fractions of a percent, the genpop largely doesn’t comprehend the state of the art. It’s not a strike on anyone’s character, it just is. It’ll take time.

culture and adoption lags capabilities, A LOT

I always feel that there's something off about this argument. Like, if a given tool provides massive benefits, it would stand to reason that it would be widely adopted quickly, because early adopters would start massively out-competing everyone else, thus pushing everyone else to figure out how to adopt those tools as well. No?

Slow diffusion makes sense if it's bottlenecked by physical constraints, or by coordination across many actors, or if we're talking about e. g. government institutions which are insulated from competitive pressures. But if we're talking about market actors adopting tools that boost their individual productivity, I don't think this mechanism makes sense.

LLMs are strangely bad at tasks in this category: scientific research, writing a novel, running a business, designing a video game, coming up with app ideas, writing funny jokes, high-level software architecture

All of these things involve high-level integrative reasoning that incorporates information from multiple domains at once.

For LLMs to get good at AI research and trigger a singularity, this integrative reasoning barrier would have to fall. Will it? I see two possibilities:

  1. Integrative reasoning is fundamentally of a different nature than more domain-specific reasoning, and we need fundamentally new techniques to unlock it
  2. Labs have been focusing on domain-specific reasoning like coding and math at the expense of integrative thinking, leaving LLM minds fragmented. Fixing this fragmentation mostly requires a change of focus; a change in training methodology which does not require any major breakthroughs.

If 1 is true, the singularity is far away. If 2 is true, then we're in an integrative reasoning overhang, and the singularity is just around the corner.

To elaborate on the scientific research part:

AI are at the frontier of math research, solving Millennium Prize problems and other famous problems on a regular bases. AI is bordering on superhuman at math. Where are all the science breakthroughs?

I think current AI models are only superhuman at one half of math. They're really good at chugging along, finding proofs using mostly existing techniques (especially if you consider that existing techniques include private data from chats with mathematicians). But that domain-specific proof-finding ability does not come with an accompanying integrative advancement of the field of mathematics.

Terrance Tao complains that AI math proofs are huge, complicated artefacts that are impossible to verify by humans. They do nothing to actually advance the field of mathematics conceptually.

Why? Because AI is not doing the integrative reasoning that would find new connections and develop new insights, the kind of work where new techniques and new discoveries are made along the way that push the field forward, like Isaac Newton inventing calculus in order to solve physics problems. The AI takes care of the busywork of finding technically-correct proofs, b... (read more)

I agree that LLMs are not currently advancing research conceptually.

Why do you think this is a barrier and not just an area of slower progress?

Most of those and definitely science require general purpose reasoning integrated over a bunch of areas, requiring a lot of time. Memory is a know deficit of models relative to humans. Scaffolds and schemes to organize that reasoning are a separate project from expanding LLMs. Google AI co-scientist project focused on scaffolding and produced remarkable results from Gemini 2 pro (maybe even 1.5?).

So I expect LLMs to get better at complex open-ended projects if and when people develop scaffolding - OR as a byproduct of scaling. Fable is in my opinion markedly better at the generalized reasoning you need to do good science. I agree that this is a weak area, but I expect slower progress relative to coding or math with their extensive training data, not a brick wall.

I agree it's mostly a scaffolding problem. Even if the LLMs do make conceptual progress, no one wants to read LLM outputs, so progress in science/research/epistemics is still diffusing at the rate of the humans communicating it. That rate is scarcely faster now than it was before, and in some ways it is slower as the institutions of science are inundated with 2x as much real research progress, combined with 10x as much slop to wade through, and no easy way to distinguish the two.

I suspect many people are having their LLM agents make the same discoveries over and over.

I don't think the agents are incapable of discerning good from bad either, it's just that they aren't currently set up to do so.

I'm building a scaffold (minerval.ai) to organize scientific knowledge and have frontier LLMs assess the quality of the evidence for every claim. I'd be interested to hear what you think about my approach.

Agree Vladimir's posts on this are illuminating. See also The Goodhart Singularity (though compare Data bottlenecks won't prevent an intelligence explosion).

My view is still uncertain, but incorporates elements of the above. I expect R&D automation is 'taste complete', and that maintaining frontier taste requires ceteris paribus human-level sample efficiency. So not much acceleration prior to that. But speed/scale will provide some degree of qualitative adjustment (perhaps not much?). Compute likely remains a constraining factor.

And then actually 'moving atoms' (as you put it) requires a bunch of task-/domain-specific data. That means (perhaps slow, faffy?) integration of sensing, log ingestion, etc.

If sample efficiency remains elusive, we might even see an industrial 'explosion' before an intelligence explosion, since manufacturing at existing tech levels is relatively repeatable compared with frontier-pushing R&D.

I plan to instead argue that “there’s plenty of room at the top”—i.e. that LLMs could become superhuman at a wide range of skills (like math research, or long-term strategic planning, or running companies) without triggering a singularity.

How would they fail to create a singularity or something like @Daniel Kokotajlo's five centuries in five years? Additionally, what is the case against the singularity emerging from idiots scaling the goddamn neuralese?

UPD: How exactly is "a pretty sharp divergence between the measured capabilities of models and their real-world impacts" affected by the loss of Navier-Stokes and EpochAI's report of a major advance(!!!) in math?

UPD2: How do we rule out mirror life synthesis by the first ASI if it decides to commit genocide?

I previously noted that we’re seeing a pretty sharp divergence between the measured capabilities of models and their real-world impacts, which should suggest that something weird is going on.

There's something there but I think its scary to reason from the the lack of normal economic impacts because they're such a lagging indicator.

I think the current situation is:

  • Frontier AI labs are really getting to the point where the nature of their work is fundamentally changed and productivity really is up by a large fraction (cf recent Ant, OAI acceleration posts)
  • In most other places the impact is pretty modest
  • AI labs are willing to spend hundreds of thousands of dollars per member of staff on compute to boost productivity
  • AI labs have a combination of beliefs and competitive pressures (including deep throughout the company) that allow them to aggressively redesign workflows around new tools
  • A large amount of the training effort of labs goes into the kinds of work done by these companies, not least just because the feedback loop from the massive usage volume is so tight
  • Pretty much no other workplaces have this combination (maybe like Jane Street/Citadel? would be very curious to hear
... (read more)

I partly agree on lagging factor, but I disagree on the acceleration condition being unique to labs.

First, AI labs do not have a monopoly on using expensive coding agents to develop. OpenAI reported their research team uses ~$600/day/person, which is still in the range of an AI-forward SF company (I agree the majority of older software companies do not do this). Note that OpenAI's spend might be slightly exaggerated by virtue of likely not caring about internal costs and not doing the most mild things to reduce it (e.g. smaller auto-compact windows, cheaper models for tasks that require less intelligence, etc.). Additionally, given the labs are locked into themselves, their capabilities might not be that far ahead of what the public can use. (Was OpenAI only having Astra in August that much SOTA over the public who was already using Fable 5 and Opus 5?).

In my domain, I'd ballpark engineering productivity up ~80%+ compared to early 2025, but it's nuanced how that manifests itself. Onboarding (both as a new employee and into new functional areas) are very fast. But ultimately software engineering is under Jevon's Paradox - our additional productivity translates to making better s... (read more)

As an even more visceral (IMO) 'Shepard tone' analogue, this time in tempo, see the rather incredible Troy track from Goransson's Odyssey soundtrack.

I previously noted that we’re seeing a pretty sharp divergence between the measured capabilities of models and their real-world impacts, which should suggest that something weird is going on.

A lot of people in this community are programmers and have the experience of the coding models having real-world impacts on their daily lives that aren't just about the models getting better at benchmarks.

You could be correct that takeover-capable AI will take longer than short timelines. You are definitely correct that there's a bit of groupthink going on around short timelines in the prosaic alignment and developer crowd.

But you could also easily be wrong. And you are entirely missing the fact that there's some groupthink going on around longer timelines in the agent foundations crowd.

Everyone is very tempted to sip a little hopium of some sort. The PA folks are hoping maybe alignment is so easy we can do it at a dead run. And the AF folks are hoping LLMs stall out just short of transformative so they have longer to work on alignment if it is, as they tend to believe, quite tricky.

I've written about the empirical evidence for hopium and groupthink in Motivated reasoning, confirmation bias, and AI risk theory. I hope you'll read it, at least the intro/summary and section on group biases in alignment thinking.

So by all means lay plans for the eventuality of longer timelines and difficult alignment. But please don't go projecting confidence when we need to also prepare for short timelines.

You accuse short timeliners of making decisions based on an emotional stance. I think if you can... (read more)

I agree with claims that we don't have AGI now, and I'd put reasonable probability that we don't get a superintelligence in 8 years (maybe around 60%?)

And yes, some short timelines people have to update now (thinking of you @Nathan Helm-Burger) But not all short timelines people have to update, like Shane Legg, for example.

I previously noted that we’re seeing a pretty sharp divergence between the measured capabilities of models and their real-world impacts

I feel like this is probably the largest disagreement by far. I'd sympathize for 2025-era models for coding, but I now think we have much more hard evidence that AI models are useful beyond benchmarks, and I think the best example of this is the way OpenAI and Anthropic's ARR accelerated in 2026 because coding models finally became good enough that you could use it as a subsitute rather than a complement, unless you have extremely good coding skills now.

There are still areas where humans can productively contribute, but more and more of this is non-coding tasks.

I think the main way in which short timeline predictions were wrong is that people underestimated how jagged the frontier was for AIs, because they incorrectly used human b... (read more)

I agree this is worth taking seriously, and I'm glad you have also converged on the interestingness of things like Vlad Nesov's ideas! I've suspected for some time that Dario's "country of geniuses in a datacenter" concept is specifically intended to allow for hedging in the case of worlds where LLMs are superhuman at a wide range of skills without triggering a singularity; the country of geniuses by some definition will arrive, and yet the world could look much the same.

But we don’t in fact have AGI

This may be a hot take, but Astra meets my bar for AGI. I was surprised to find this to be the case, but so far I haven't found anything in which it doesn't perform better than my median friend. Of course it's quite possible I've simply failed to try some large category of tasks in which it sucks, but take that for what you will.

and the world would look very different if we did.

As much as I disagree with his long-term conclusions, Arvind Narayanan's work (AI as normal technology, etc.) should at least make you suspicious about whether you can use "the world hasn't changed much from 1 year ago" as strong evidence for "we have not developed AGI in the past year".

Here's a list of my donations so far this year (put together as part of thinking through whether I and others should participate in an OpenAI equity donation round).

They are roughly in chronological order (though it's possible I missed one or two). I include some thoughts on what I've learned and what I'm now doing differently at the bottom.

  1. $100k to Lightcone
    1. This grant was largely motivated by my respect for Oliver Habryka's quality of thinking and personal judgment.
    2. This ended up being partially matched by the Survival and Flourishing Fund (though I didn't know it would be when I made it). Note that they'll continue (partially) matching donations to Lightcone until the end of March 2026.
  2. $50k to the Alignment of Complex Systems (ACS) research group
    1. This grant was largely motivated by my respect for Jan Kulveit's philosophical and technical thinking.
  3. $20k to Alexander Gietelink Oldenziel for support with running agent foundations conferences.
  4. ~$25k to Inference Magazine to host a public debate on the plausibility of the intelligence explosion in London.
  5. $100k to Apart Research, who run hackathons where people can engage with AI safety research in a hands-on way (technically made with my
... (read more)

I'm glad you're betting on your own taste/expertise instead of donating on behalf of the community—that seems sensible to me.

What are the best places to start reading about why you are uninterested in almost all commonly proposed AI governance interventions, and about the AI governance interventions you are interested in? I imagine the curriculum sheds some light on this, but it's quite long.

Hmm, I don't have anything substantive out on this specifically; the closest is probably this talk (though note that some of my arguments in it were a bit sloppy, e.g. as per the top comment).

>> " Even more recently, I've decided that I can bet on my research taste most effectively by simply hiring research assistants to work for me. " 

This seems right to me - plausibly a massive multiplier.

For example, John Wentworth told me he found a large productivity boost when he started working David Lorell.

This seems like both a good process, using your existing knowledge to find good opportunities rather than doing normal applications seems in line with my guess at how high EV grants her name, and a set of grantees I am generally glad to see funded.

*Lighthaven->Lightcone (at least in the case of SFF matching)

Thank you for donating!

I feel kinda frustrated whenever "shard theory" comes up in a conversation, because it's not a theory, or even a hypothesis. In terms of its literal content, it basically seems to be a reframing of the "default" stance towards neural networks often taken by ML researchers (especially deep learning skeptics), which is "assume they're just a set of heuristics".

This is a particular pity because I think there's a version of the "shard" framing which would actually be useful, but which shard advocates go out of their way to avoid. Specifically: we should be interested in "subagents" which are formed via hierarchical composition of heuristics and/or lower-level subagents, and which are increasingly "goal-directed" as you go up the hierarchy. This is an old idea, FWIW; e.g. it's how Minsky frames intelligence in Society of Mind. And it's also somewhat consistent with the claim made in the original shard theory post, that "shards are just collections of subshards".

The problem is the "just". The post also says "shards are not full subagents", and that "we currently estimate that most shards are 'optimizers' to the extent that a bacterium or a thermostat is an optimizer." But the whole point... (read more)

65553321

I am not as negative on it as you are -- it seems an improvement over the 'Bag O' Heuristics' model and the 'expected utility maximizer' model. But I agree with the critique and said something similar here:

you go on to talk about shards eventually values-handshaking with each other. While I agree that shard theory is a big improvement over the models that came before it (which I call rational agent model and bag o' heuristics model) I think shard theory currently has a big hole in the middle that mirrors the hole between bag o' heuristics and rational agents. Namely, shard theory currently basically seems to be saying "At first, you get very simple shards, like the following examples: IF diamond-nearby THEN goto diamond. Then, eventually, you have a bunch of competing shards that are best modelled as rational agents; they have beliefs and desires of their own, and even negotiate with each other!" My response is "but what happens in the middle? Seems super important! Also haven't you just reproduced the problem but inside the head?" (The problem being, when modelling AGI we always understood that it would start out being just a crappy bag of heuristics and end up a scary rational ag

... (read more)

In other words, shard advocates seem so determined to rebut the "rational EU maximizer" picture that they're ignoring the most interesting question about shards—namely, how do rational agents emerge from collections of shards?

Personally, I'm not ignoring that question, and I've written about it (once) in some detail. Less relatedly, I've talked about possible utility function convergence via e.g. A shot at the diamond-alignment problem and my recent comment thread with Wei_Dai. 

It's not that there isn't more shard theory content which I could write, i... (read more)

FWIW I'm potentially intrested in interviewing you (and anyone else you'd recommend) and then taking a shot at writing the 101-level content myself.

  • a) gaslit by "I think everyone already knew this" or even "I already invented this a long time ago" (by people who didn't seem to understand it); and that 

Curious to hear whether I was one of the people who contributed to this.

Nope! I have basically always enjoyed talking with you, even when we disagree.

Ok, whew, glad to hear.

But shard theorists mainly aim to address agency obtained via DPO-like setups, and @TurnTrout has mathematically proved that such setups don't favor the power-seeking drives AI safety researchers are usually concerned about in the context of agency.

I read the section you linked, but I can't follow it. Anyway, here it is its conclusive paragraph:

Conclusion: Optimal policies for u-AOH will tend to look like random twitching. For example, if you generate a u-AOH by uniformly randomly assigning each AOH utility from the unit interval , there's no predictable regularity to the optimal actions for this utility function. In this setting and under our assumptions, there is no instrumental convergence without further structural assumptions. 

From this alone, I get the impression that he hasn't p... (read more)

A key part of instrumental convergence is the convergence aspect, which as I understand it refers to the notion that even very wild utility functions will share certain preferences. E.g. the empirical tendency for random chess board evaluations to prefer mobility. If you don't have convergence, you don't have instrumental convergence.

Ok. Then I'll say that randomly assigned utility over full trajectories are beyond wild!

The basin of attraction just needs to be large enough. AIs will intentionally be created with more structure than that.

The issue isn't the "full trajectories" part; that actually makes instrumental convergence stronger. The issue is the "actions" part. In terms of RLHF, what this means is that people might not simply blindly follow the instructions given by AIs and rate them based on the ultimate outcome (even if the outcome differs wildly from what they'd intuitively think it'd do), but rather they might think about the instructions the AIs provide, and rate them based on whether they a priori make sense. If the AI then has some galaxybrained method of achieving something... (read more)

An analogy that points at one way I think the instrumental/terminal goal distinction is confused:

Imagine trying to classify genes as either instrumentally or terminally valuable from the perspective of evolution. Instrumental genes encode traits that help an organism reproduce. Terminal genes, by contrast, are the "payload" which is being passed down the generations for their own sake.

This model might seem silly, but it actually makes a bunch of useful predictions. Pick some set of genes which are so crucial for survival that they're seldom if ever modified (e.g. the genes for chlorophyll in plants, or genes for ATP production in animals). Treating those genes as "terminal" lets you "predict" that other genes will gradually evolve in whichever ways help most to pass those terminal genes on, which is what we in fact see.

But of course there's no such thing as "terminal genes". What's actually going on is that some genes evolved first, meaning that a bunch of downstream genes ended up selected for compatibility with them. In principle evolution would be fine with the terminal genes being replaced, it's just that it's computationally difficult to find a way to do so without breaking do... (read more)

65411

But of course there's no such thing as "terminal genes". What's actually going on is that some genes evolved first, meaning that a bunch of downstream genes ended up selected for compatibility with them. In principle evolution would be fine with the terminal genes being replaced, it's just that it's computationally difficult to find a way to do so without breaking downstream dependencies.

I think your analysis is incorrect. The book is called "The Selfish Gene". No basic unit of evolution is perfect, but probably the best available is the gene--which is to say, genomic locus (defined relative to surrounding context). An organism is a temporary coalition of its genes. Generally there's quite strong instrumental alignment between all the genes in an organism, but it's not always perfect, and you do get gene drives in nature. If a gene could favor itself at the expense of the other genes in that organism (in terms of overall population frequency), it totally would.

I think this is a good analogy for how human values work. We start off with some early values, and then develop instrumental strategies for achieving them. Those instrumental strategies become crystallized and then give

... (read more)

I think your analysis is incorrect


Well, it's kinda true, right?  Ontogeny recapitulates phylogeny (developing embryos look like worms, then fish, amphibians -- they mirror the path of evolution).  That's because it's easier for evolution to add steps at the end than to changes steps in the beginning.  It happens with computers too -- modern Intel and AMD chips still startup in 16 bit real mode.

In the coalition of genes that make it into a gamete, newer genes support the old genes, but not vice versa.  The genes that control apoptosis (p... (read more)

Right, the part about layers of more and less conserved genes is true AFAIK. (I think actually ontogeny doesn't recapitulate phylogeny linearly, but rather there's a kinda of hourglass structure where some mid-development checkpoints are most conserved--but I'm not remembering where I saw this--possibly in a book or paper by Rupert Riedl or Günter Wagner.)

What I'm objecting to, is viewing that as a growth of a values structure for the values of [the evolution of a species, as an agent]. That's because that entity doesn't really value genes at all; it doesn... (read more)

Thinking step by step:

I like the point that fundamentally the structure is tree-like, and insofar as terminal goals are a thing it's basically just that they are the leaf nodes on the tree instead of branches or roots. Note that this doesn't mean terminal goals aren't a thing; the distinction is real and potentially important.

I think an improvement on the analogy would be compare to a human organization rather than to a tree. In a human organization (such as a company or a bureaucracy) at first there is one or a small group of people, and then they hire more people to help them with stuff (and retain the option to fire them if they stop helping) and then those people hire people etc. and eventually you have six layers of middle management. Perhaps goals are like this. Evolution and/or reinforcement learning gives us some goals, and then those goals create subgoals to help them, and then those subgoals create subgoals, etc. In general when it starts to seem that a subgoal isn't going to help with the goal it's supposed to help with, it gets 'fired.' However, sometimes subgoals are 'sticky' and become terminal-ish goals, analogous to how it's sometimes hard to fire people & how t... (read more)

This is great, and on an important topic that's right in the center of our collective ontology and where I've been feeling for a while that our concepts are inadequate.

Top level post! Top level post! 

I think this sort of assumes that terminal-ish goals are developed earlier and thus more stable and instrumental-ish goals are developed later and more subject to change.

I think this may or may not be true on the individual level but it's probably false on the ecological level. 

Competitive pressures shape many instrumental-ish goals to be convergent whereas terminal-ish goals have more free parameters.

In logical time, instrumental convergence precedes the terminal-ish goals that cause it?

I agree that human values are more accretive like this, but I would also call those genes “terminal” in the same sense that I call some of my own goals “terminal.” E.g., I can usually ask myself why I’m taking a given action and my brain will give a reasonable answer: “because I want to finish this post,” “because I’m hungry,” whatever. And then I can keep double clicking on those: “I want to finish the post because I don’t think this crux has been spelled out very well yet” and I can keep going and going until at some point the answer is like “I don’t kno... (read more)

Yeah, one possible successor concept to the instrumental/terminal distinction might be something like "does this thing clearly draw its raison d'être from another thing, or is it its own source of raison d'être or some third thing which is like a nebulous, non-explicitized symbiosis", where the raison d'être is itself something potentially revisable by reflection (or whatever mind-shaping process).

We start off with some early values, and then develop instrumental strategies for achieving them. Those instrumental strategies become crystallized and then give rise to other instrumental strategies for achieving them, and so on.

This seems true of me for some cases, mostly during childhood. Maybe it was a hack that evolution used to get from near-sensory value specifications to more abstract values. But if I (maybe unfairly) take this as a full model of human values and entirely remove the terminal-instrumental distinction, then it seems to make a bunch o... (read more)

creating new goals is always purely negative for every other currently existing goal, right

No more than hiring new employees is purely negative for existing employees at a company.

The premise I'm working with here is that you can't create goals without making them "terminal" in some sense (just as you can't hire employees without giving them some influence over company culture).

You did a good job of communicating your positive feelings about this kind of value system, I understand slightly better why you like it.

I can see how it can be worth the trade-off to make a new goal if that's the only way to get the work done. But it's negative if the work can be done directly.

And we know many small-ish cases where we can directly compute a policy from a goal. So what makes it impossible to make larger plans without adding new goals? And why does adding new goals shift it from impossible to possible?

I think you're making the distinction more confusing than it has to be.

There are things that has motivational pull, and there are things that don't but I do them anyway because they are a necessary step to get what I actually want.

Say I want to get an apple, and the easiest way to get one is going to the store and by some. Going to the sore in this story is clearly an instrumental goal, and enjoying eating my apple is a terminal goal[1]

Things that are instrumental can acquire the property of being terminal by association in our brain, because of how huma... (read more)

I think humans do value-inference in both directions. Our "terminal" values are in part grown out of which high-level things correlate with low-level things. An example is John Wentworth---who seems to lack the circuits for feeling (companionate) love---saying he thinks relationships and kids are lame compared to his goals of saving the world, to the point where he would prefer not to be modified to be able to feel love, and says he would view a drug which enabled him to feel love similarly to a syringe of heroin. Clearly, his brain has built up terminal v... (read more)

We start off with some early values, and then develop instrumental strategies for achieving them. Those instrumental strategies become crystallized and then give rise to other instrumental strategies for achieving them, and so on. Understood this way, we can describe an organism's goals/strategies purely in terms of which goals "have power over" which other goals, which goals are most easily replaced, etc, without needing to appeal to some kind of essential "terminalism" that some goals have and others don't.

I think it would help me if you explained wha... (read more)

In my "goals having power over other goals" ontology, the instrumental/terminal distinction separates goals into two binary classes, such that goals in the "instrumental" class only have power insofar as they're endorsed by a goal in the "terminal" class.

By contrast, when I talk about "instrumental strategies become crystallized", what I mean is that goals which start off instrumental will gradually accumulate power in their own right: they're "sticky".

We start off with some early values, and then develop instrumental strategies for achieving them. Those instrumental strategies become crystallized and then give rise to other instrumental strategies for achieving them, and so on.

I want to call this something like "developmental axiology" but that sounds more like Kegan whereas I mean it more like evo-devo.

How does the Shareholder Value Revolution fit into your picture? From an AI overview:

1. The Intellectual Origins (The 1970s)

The revolution was born out of economic stagnation in the 1970s. As U.S. corporate profits dipped and competition from Japan and Germany rose, economists and theorists argued that American managers had become "fat and happy," running companies for their own comfort rather than efficiency.

Two key intellectual pillars drove the change:

  • Milton Friedman (The Moral Argument): In a famous 1970 New York Times essay, Friedman argued, "The
... (read more)

I'm in the middle of writing an essay which discusses the shareholder value revolution (amongst many other examples) as the process of making a conceptual mistake.

without needing to appeal to some kind of essential "terminalism" that some goals have and others don't.

That appeal doesn't seem overly problematic though, as some goals are clearly terminal. For example, eating chocolate (or rather: something that tastes like it). Or not dying. Those goals are given to us by evolution. Chocolate is a case where we actually have an instrumental reason not to eat it (too much sugar for modern environments), which counteracts the terminal goal in the opposite direction. Which means they are clearly different. Are there pe... (read more)

From the alternative perspective I've outlined above, though, the process of instrumentally growing and gaining resources is also simultaneously the process of constructing values.

Reminds me of Daniel Polani's talk on sensor evolution. The gist is: basically, all viable ways to grow/evolve new sensors for getting the information that would help you achieve your goals get you sensors that deliver OOMs more information than you need, and then this information "has to go somewhere",[1] so you grow new goals. (Of course, rinse and repeat, in a loop.)

  1. ^

    I di

... (read more)

genes for ATP production in animals

Nitpick, but: ATP is not specific to animals. It's "found in [and synthesized by] all forms of life"[1].

  1. ^

    Although, interestingly, yesterday I heard about some bacterium that is so adapted to endocellular parasitism that it lost the capacity to produce ATP and relies on stealing it from its host cell.

Re-reading this post, I again feel that it is insightful reframing and the problem it addresses is critical to thinking about our world, and how the future will go.

However I dislike the last word.

but by building larger creatures much more strange and wonderful

This is claiming that this development of ever newer, richer, and more nuanced goals, is well, "wonderful". But "wonderful" is relative to some specific value system. It might be wonderful, it might be horrifying, most likely it will be extremely weird.

I admit that if this development of a hierarchy o... (read more)

Responded I think resolving this with: Managed vs Unmanaged Agency

This treats sexual selection as determined by instrumental genes and selecting for instrumental genes, but I feel like it makes more sense to say that sexual selection selects for terminal genes (or at least terminal phenotypes), since those are the ones organisms will spontaneously collaborate to promote.

The bit about layering creating functional fixedness reminds me of organisms (especially humans, but more broadly evolution as a search process) as 'homeostatic envelope extenders' a la Nozick's take on Quine.

Evolution doesn't determine what's right, it only shows you what's left! After selection, that is. Indeed, there are conceivable environments where the chlorophyll gene would be rapidly outcompeted by something that is better at exploiting free energy gradients, e.g. environments where chlorophyll is for some reason less useful than it is today. Its ubiquity today is basically contextual and so doesn't represent any kind of terminus. If we model evolution as an agent with values, then it seems to me that it has one coherent terminal goal, which, if we go b... (read more)

Early last year Longview Philanthropy invited me to attend an AI safety fundraising dinner they were hosting for Jane Street traders in Hong Kong as an expert guest. In the lead-up, I got on a call with them, and gave my honest opinions about their funding recommendations.

I summarized these opinions in a follow-up email to them at the time as "Overall I'm very excited about three of them ([redacted], [redacted], [redacted]), and moderately excited about most of the rest. But I'm pretty skeptical of some of the policy advocacy (particularly [redacted] and [redacted])." (Though it's possible I came across as more pessimistic in the call than I sound in this summary.)

As a result, they rescinded the invitation. It's not clear to me that they were wrong to do so, because my sense is that it's normal to prioritize bringing more sympathetic experts to such events. EDIT: Upon reflection, even if this would be normal in most philanthropic contexts, I want to hold organizations associated with AI safety to a much higher standard. I expect that selection effects like this one add up over time to give funders a distorted view of the ecosystem, and IMO a more virtuous fundraising process would ... (read more)

1261

Thank you for writing this. I thought I'd chime in with another example of this kind of thing I'd recently seen in (perhaps, contributed to) in our own work at Manifund.

A couple of months back, we were just kicking off our Falcon Fund (rapid grants for animal welfare, esp AI x animals) - see https://manifund.org/projects/falcon-fund. My original draft stated fairly strongly that the existing EA Animal Welfare Fund was dispensing money way too slowly eg couple months behind schedule for reviews; which was part of what we were excited to create Falcon. However, I was asked to remove this kind of internicine criticism from the launch post, on the grounds that (stylized) "everyone already knows that AWF is slow" and "we don't to want to scare donors and reduce even further the small amount of money AW gets"

Also, one of the animal charity founders who Marcus thought was one of the best examples of donation he's made also asked to be removed, because they did not want to be associated with EA, even though many of their donations come from EA sources.

I complied on both of these cases on a general heuristic of trying to be polite and play nice and aim to be positive sum with our counterparties, but imo these hurt Falcon's ability to raise and Manifund's general mission of increasing transparency in the ecosystem. I'm still not sure what the right call was.

Imagine if early EA orgs had refused to publicly criticize conventional philanthropy because they didn’t want to reduce the amount of money going to those causes. Showing why one donation opportunity is better than another is The Whole Point, and conditioning people’s access to information or opportunities on the basis of their willingness to keep quiet regarding your inefficiencies is one of the patterns upstream of failures in the traditional nonprofit ecosystem. Failures EA’s entire brand is based on criticizing.

I think the choices you and Richard have made are reasonable/locally prosocial, but the people who asked you to make them should change their behavior.

(It‘s also plausible to me that the reasons they gave you are not reflective of their true reasons, in which case they should surface their true reasons.)

Thank you for sharing that this happened, and thanks also to Richard!

because they did not want to be associated with EA, even though many of their donations come from EA sources.

Is it a common thing in the EA philanthropy sphere for grantees to take funding without disclosing its source? This strikes me as very bad.

Not sure the bar for ‘common’ but I’m aware of maybe half a dozen roving donation funds that don’t disclose more about themselves than is legally required (or maybe have heard a handful of different stories from different people about the same one? Unclear).

AFAIK you have to disclose to the government, so someone at your org always knows the proximal source, and an auditable paper trail usually exists back to the ultimate source (although there are probably ways to thwart this kind of tracing, even by the government, although my guess is most of these are ... (read more)

@Elizabeth Disclosing funding sources enables the public to identify conflicts of interest. On the balance, I think it's better to prevent conflicts of interest than it is to provide grantmakers with optical cover for bet-hedging (re: yams post, perhaps I don't understand the pressures on grantmakers here).

IME It's much more common for grantees to not want to be associated with funders than the other way around[1]; back when I was doing more grantmaking (I haven't done it much the last year or so), I tried pushing back against this, with mixed success.

  1. ^

    Typically if funders don't like optics considerations regarding a grantee they're much more likely to just not fund something. Tbc I also think this is usually bad.

Thanks, this was very helpful insight from this side of the grant-making barrier!

One mechanism that could solve this without causing interpersonal grief is some sort of third party exit poll platform for funding applications, basically a “Glassdoor for Philanthropy” which would make it common knowledge which funds do fast turnarounds and other desiderata.

For the anonymous grantee thing, maybe you say something like “no more than 10% of falcon donations will be to anonymous parties”. I think it’s principled to be boundedly transparent, it’s like principled lessetarianism

That might solve the issue of applicants choosing where to apply, but I don't think glassdoor-style data would make AWF any happier about Austin bringing up the unfavorable comparison in his marketing copy.

Totally agree - but Austin could sidestep that by just pointing to “our Gd4Phil score is really good” and not feel any need to point out anything about anyone else’s score

Thank you for sharing. Giving good criticism is really difficult in general, I've found that the expected norms when communicating with a grantmaking/fundraising audience can feel much more reserved than the register I'd use with technical or executive staff. I've settled into a pattern where I write very candid feedback, post it, then filter/translate the original through an LLM and link to the sanitised/"professional" version, which doesn't seem ideal at all.

I try to be sharp and sincere but the directness of the language which comes natively to me can i... (read more)

I'm sharing here a rough draft of a new curriculum I've made, called Understanding Intelligent Agency. I've written a lot lately about how AI alignment should be trying to develop a new scientific paradigm; this curriculum is my attempt to point directly at what that new paradigm looks like.

I'm sharing this in draft form because I don't know how long it will take me to get it to a state where I'm happy to promote it widely (at the very least, I'll be offline for most of the next week). I'm hoping it can be useful to people before that; and please feel free to leave comments or suggestions for additional readings.

I wonder how one could make useful additions or point at relevant conjectures. For example, Yudkowsky's Value Theory, the idea that FDT alone produces the ethics-like result prohibiting us from looting the universe even of a paperclip optimizer or my take trying to suggest that beyond-FDT morals aren't THAT far from sociopolitics.

Ah, a fellow student of Hasok Chang? I'm looking forward to digging into this.

In response to an email about what a pro-human ideology for the future looks like, I wrote up the following:

The pro-human egregore I'm currently designing (which I call fractal empowerment) incorporates three key ideas:

Firstly, we can see virtue ethics as a way for less powerful agents to aggregate to form more powerful superagents that preserve the interests of those original less powerful agents. E.g. virtues like integrity, loyalty, etc help prevent divide-and-conquer strategies. This would have been in the interests of the rest of the world when Europe was trying to colonize them, and will be in the best interests of humans when AIs try to conquer us.

Secondly, the most robust way for a more powerful agent to be altruistic towards a less powerful agent is not for it to optimize for that agent's welfare, but rather to optimize for its empowerment. This prevents predatory strategies from masquerading as altruism (e.g. agents claiming "I'll conquer you and then I'll empower you", which then somehow never get around to the second step).

Thirdly: the generational contract. From any given starting point, there are a huge number of possible coalitions which could form, an... (read more)

6432

How would this ideology address value drift? I've been thinking a lot about the kind quoted in Morality is Scary. The way I would describe it now is that human morality is by default driven by a competitive status/signaling game, where often some random or historically contingent aspect of human value or motivation becomes the focal point of the game, and gets magnified/upweighted as a result of competitive dynamics, sometimes to an extreme, even absurd degree.

(Of course from the inside it doesn't look absurd, but instead feels like moral progress. One example of this that I happened across recently is filial piety in China, which became more and more extreme over time, until someone cutting off a piece of their flesh to prepare a medicinal broth for an ailing parent was held up as a moral exemplar.)

Related to this is my realization is that the kind of philosophy you and I are familiar with (analytical philosophy, or more broadly careful/skeptical philosophy) doesn't exist in most of the world and may only exist in Anglophone countries as a historical accident. There, about 10,000 practitioners exist who are funded but ignored by the rest of the population. To most of humanity, "ph... (read more)

One of the main ways I think about empowerment is in terms of allowing better coordination between subagents.

In the case of an individual human, extreme morality can be seen as one subagent seizing control and overriding other subagents (like the ones who don't want to chop off body parts).

In the case of a group, extreme morality can be seen in terms of preference cascades that go beyond what most (or even any) of the individuals involved with them would individually prefer.

In both cases, replacing fear-based motivation with less coercive/more cooperative ... (read more)

I'm not sure that fear or coercion has much to do with it, because there's often no internal conflict when someone is caught up in some extreme form of the morality game, they're just going along with it wholeheartedly, thinking they're just being a good person or helping to advance the arc of history. In the subagents frame, I would say that the subagents have an implicit contract/agreement that any one of them can seize control, if doing so seems good for the overall agent in terms of power or social status.

But quite possibly I'm not getting your point, in which case please explain more, or point to some specific parts of your articles that are especially relevant?

there's often no internal conflict when someone is caught up in some extreme form of the morality game

Belated reply, sorry, but I basically just think that this is false—analogous to a dictator who cites parades where people are forced to attend and cheer as evidence that his country lacks internal conflict. Instead, the internal conflict has just been rendered less legible.

In the subagents frame, I would say that the subagents have an implicit contract/agreement that any one of them can seize control, if doing so seems good for the overall agent in terms

... (read more)

In some sense this is a core idea of UDT: when coordinating with forks of yourself, you defer to your unique last common ancestor. When it's not literally a fork of yourself, there's more arbitrariness but you can still often find a way to use history to narrow down on coordination Schelling points (e.g. "what would Jesus do")

I think this is wholly incorrect line of thinking. UDT operates on your logical ancestor, not literal.

Say, if you know enough science, you know that normal distribution is a maxentropy distribution for fixed mean and variance, and therefore, optimal prior distribution under certain set of assumptions. You can ask yourself question "let's suppose that I haven't seen this evidence, what would be my prior probability?" and get an answer and cooperate with your counterfactual versions which have seen other versions of evidence. But you can't cooperate with your hypothetical version which doesn't know what normal distribution is, because, if it doesn't know about normal distribution, it can't predict how you would behave and account for this in cooperation.

Sufficiently different versions of yourself are just logically uncorrelated with you and there is no game-theoretic reason to account for them.

Sufficiently different versions of yourself are just logically uncorrelated with you and there is no game-theoretic reason to account for them.

Seems odd to make an absolute statement here. More different versions of yourself are less and less correlated, but there's still some correlation. And UDT should also be applicable to interactions with other people, who are typically different from you in a whole bunch of ways.

Absolute sense comes from absolute nature of taking actions, not absolute nature of logical correlation. I.e., in Prisoner's Dilemma with payoffs (5,5)(10,1)(2,2) you should defect if your counterparty is capable to act conditional on your action in less than 75% of cases, which is quite high logical correlation, but expected value is higher if you defect.

2nd point is a scary one.

Empowering others in the relative sense is a terrible idea, unless they are trustworthy/virtuous. Same issue as AI risk

In the absolute terms sure

Yeah, this definitely needs to be a limited sorry of empowerment, in my mind. Like, imagine you wanted to give a 5 year old child the best day ever. You wanted to give them really fun options, but also not cause them to suffer from decision fatigue, or regret about the paths not taken. More importantly, if they asked for an alien ray gun with which to shoot bad guys, giving them an actual extremely dangerous weapon would be a terrible idea. Similarly, offering them a ride on a cool looking roller coaster that was actually a 'death coaster' would be a terrible trap.

(I'm inspired to write this comment on the notion of empowerment because of Richard Ngo's recent comment in another post on Towards a scale-free theory of intelligent agency, so I'll both respond to the empowerment motion and part of the comment linked below:

https://www.lesswrong.com/posts/5tYTKX4pNpiG4vzYg/towards-a-scale-free-theory-of-intelligent-agency#nigkBt47pLMi5tnGd):

To address this part of the linked comment:

Lastly: if at each point in time, the set of agents who are alive are in conflict with potentially-simpler future agents in a very destructiv

... (read more)

Would be interested in a quick write-up of what you think are the most important virtues you'd want for AI systems, seems good in terms of having things to aim towards instead of just aiming away from.

This prevents predatory strategies from masquerading as altruism

I did not understand that. Is the worry that it's hard to distinguish a genuine welfare maximizer from a predator because you can't tell if they will ever give you back power? I don't understand why this does not apply to agents pretending to pursue empowerment. It is common in conflicts to temporarily disempower someone to protect their long-term empowerment (e.g. a country mandatorily mobilizing for war against a fascist attacker, preventing a child from ignoring their homework), and it is a... (read more)

Ah, yeah, I was a bit unclear here.

Basic idea is that by conquering someone you may not reduce their welfare very much short-term, but you do reduce their power a lot short-term. (E.g. the British conquered India with relatively little welfare impacts on most Indians.)

And so it is much harder to defend a conquest as altruistic in the sense of empowering, than it is to defend a conquest as altruistic in the sense of welfare-increasing.

As you say, this is not a perfect defense mechanism, because sometimes long-term empowerment and short-term empowerment conf... (read more)

There's a strong pattern in ratfic of the protagonist "winning" by gaining the power to design a new world order from scratch—i.e. taking over the world. It's a very High Modernist mindset (as I pointed out in a recent tweet). And once you see how crucial this is to the rationalist perspective on what a good future looks like, it's hard to unsee.

You might respond: the worlds these protagonists find themselves in are usually so bad that seizing absolute power is in fact the most ethical thing to do. But the worlds didn't have to be that bad! The writers chose to design them that way—I expect in significant part because that provides a narratively compelling backdrop for the thing they wanted to write about, which was their heroes taking over the world.

From my current perspective, this feels ominous in a kinda similar way as if I read a bunch of Marxist fiction from the 1800s and noticed that it always ended in a communist revolution.

Spoilers for a range of ratfic (unfortunately telling you which ratfic itself constitutes a spoiler):

EDIT: the spoiler tags seem not to be working on the images. Mods, can you help? For now I'll add a bunch of space below. Expect many spoilers if you scr... (read more)

I also liked this one for foreshadowing the same problems Ngo was complaining about: https://substack.com/home/post/p-179661551

lol, incredible.

Perhaps the direction of causation goes the other way? In the real world, it seems like we are headed towards a situation where 1-3 giant armies of superintelligent AIs transform the planet. Whoever controls those AIs, if anyone does, will be in a position analogous to all those fictional examples you give.

All the authors of all those works of fiction know this, or at least have been seriously entertaining this AI-power-concentration-hypothesis for years.

So, they are writing fictional analogies for the situation they expect to actually happen in real life. Except of course, since they are writing fiction, it has to have a happy ending.

Their expectation that it's going to happen in real life is in fact correct unfortunately, as best as I can tell. Will the ending in real life be as happy as the endings in fiction? Of course not, but they shouldn't be taken as implicitly claiming that, anymore than Tolkien should be taken as implicitly claiming that all this mechanization, industrialization, technology, etc. that he was reacting to was going to end happily ever after in real life thanks to the plucky heroism of some country bumpkins who end up destroying it all.

So, they are writing fictional analogies for the situation they expect to actually happen in real life. Except of course, since they are writing fiction, it has to have a happy ending.

Well, exactly what I'm disputing here is how happy the ending is. For example, imagine that all of these stories played out exactly the same, with the exact same amount of concentration of power. But instead of the heroes getting to use that power to reshape the world, the power instead goes to.... a random person off the street. I expect that these authors, if they were to write that kind of ending, would portray it as a maybe-happy-ish ending, but one that's still pretty scary and uncertain.

And indeed, this is roughly how I'd describe the stories mentioned above where a mostly-aligned AI gets total power—Friendship is Optimal, Branches on the Tree of Time, and Metamorphosis of Prime Intellect. These stories really grapple with the sense of unease and tension that comes with almost everyone losing almost all their power.

Whereas when I look at the examples of ratfic above, the stance they're taking seems to be "our heroes became dictators of the universe. This is a straightforwardly happy ending." And indeed, on several occasions (maybe as many as half a dozen?) I've heard people describe the ending of Worth the Candle as one of the best utopias they can imagine. All of this really seems like a big ideological blind spot.

I don't think I understand how you are objecting to my point. Straightforwardly happy endings are the norm in fiction. I think you are reading too much into their choice to make the ending straightforwardly happy.

Let's think about, idk, the Star Wars series for comparison. In it, the plucky heroes go to war against the bad guys, kill loads of them in a series of military engagements, and ultimately emerge victorious with their faction the new rulers of the galaxy. This is presented as straightforwardly happy ending.

In fact lots of mainstream popular stories are of this pattern: The heroes defeat the bad guys in armed combat, kill loads of them, and then their faction has more political and military power afterwards and they all live happily and it's great and uncomplicated.

Applying your reasoning to these popular stories, I guess you'd say: "Ominous. It seems like most Hollywood writers expect there to be a giant war in the future, possibly a civil war or series of assassinations. They are encouraging people to fight in this war / battle / revolution / assassination-series, and win, and they are portraying this as uncomplicatedly good."

But I feel like there's a better explanation f... (read more)

I do actually think that the general trope of "the rebels winning is sufficient for a happy ending" is pretty indicative of poor ethical thinking.

But even Hollywood balks at their heroes ending up with literal godlike control of the world. For example (though I haven't watched the series) my impression of the Avengers franchise is that they introduce a plot device (the infinity gauntlet) that gives its wielder godlike powers, the heroes use it specifically to defeat the bad guy and undo the damage he caused, and then they destroy the device.

In other words, they got to the exact point that ratfic heroes got to, and then their happy ending specifically involves them giving up the same kind of godlike power that ratfic heroes typically use to make themselves dictators of the universe.

Similarly for Superman: his happy endings involve him successfully using his godlike powers to beat the bad guys without changing the established world power structures basically at all. And I feel pretty confident that a big reason Superman doesn't end up taking over the world is because the writers and viewers would have moral qualms about that kind of ending.

tl;dr: there are many ways to make a story ... (read more)

And I feel pretty confident that a big reason Superman doesn't end up taking over the world is because the writers and viewers would have moral qualms about that kind of ending.

Yes. And I claim they're wrong about that.

There's lots of banal evil (some of which that is not regarded as evil by typical social morality, some of which is, but is generally treated as normal and ignored). I would fight a war to end factory farming, if that would help.

If I ended up with "ultimate power" somehow, by some mechanism that didn't involve me taking on ultimate power for... (read more)

Oh, I think of "ending factory farming" as very far from "taking over the world".

If Superman were a skilled political operator it could be as simple as arranging to take photoshoots with whichever politicians legislated the end of factory farms.

Or if he were less skilled it could involve doing various kinds of property damage to factory farms (potentially even things which there aren't laws against, like flying around them in a way which blows the buildings over).

This might escalate to the government trying to arrest him, and outright conflict, but honestl... (read more)

letting factory farming end on whatever the natural counterfactual timeline is (without AI, maybe 20 or 30 years?)

This is very optimistic. Why do you think so? Alt. proteins outcompeting factory farmed meat?

Relatedly I'm increasingly coming to believe that this reasoning applies to Lincoln, and that we'd be in a much better position if he'd let the Confederacy secede and then imposed strong economic and moral pressure on them to end slavery.

Why? Because that was the beginning of the centralization of power with the federal government?

I think Alexander Hamilton was the beginning, but this seems like a big step. Vassar talks about how, from the civil war onwards, the American legal system needed to be optimized to rule a vassal state while also pretending that they weren't ruling a vassal state. Can't remember the specific examples he cited to me but I found it fairly compelling.

This certainly seems to be the case with Trump's (in my understanding) limited ability to govern blue states and cities.

This might escalate to the government trying to arrest him, and outright conflict, but honestly if Superman isn't skillful enough to defuse that kind of thing, given his influence, then he doesn't have much business imposing political changes on the world anyway.

This seems like a weird set of claims to me.

First of all, yes, it seems very likely to escalate to conflict? Many nations would consider Superman's property damage to their factories to be an act of war by a foreign power, and that he's doing it in ways that technically circumvent existing laws is ... (read more)

Are you saying that if the diplomatic negotiations deteriorate to the point of military action, that means that our hypothetical superman has failed, and he would be better off retiring? Don't existing legitimate countries go to war for far less noble reasons all the time?

I indended this to refer to scenarios where the US itself (or other leading western powers) were taking military action against Superman. I care much less about whether he destabilizes North Korea or Eritrea or even countries similar to those but better-governed. But I care a lot about wh... (read more)

I'm still not sure what standard you're holding our hypothetical superman too. You just don't want him to destabilize the countries that you consider most important?

If he overthrows the Communist Party of China, is he violating your standards?

If he forcibly institutes electoral reforms to make the US government more functional, is that violating your standards? Is the "forcibly" there a problem, and if it comes to using military force to push for electoral reform, he should hold off?

What's the line that he shouldn't cross?



I claim that there are some people such that, if they were dictators of China, that would be much worse than the current situation. And there are some people such that, if they were dictators of China, that would be much better than the current situation. Which category a given person falls into depends a lot on their honesty, integrity, wisdom, ability to understand political dynamics, ability to resist manipulation, etc.

There are no particular limits I'd want to place on a sufficiently virtuous Superman. E.g. I want Superman to follow a policy that leads... (read more)

Ok, I agree with all this!

(Modulo, I have more uncertainty about how much force the wisest path entails when, one has hegemonic power. Certainly using military force to get your way has major costs, and so, taking those costs into account, I would expect the wiser courses of actions to be more peaceful, generally.)

Should I then summarize your criticism of ratfic protagonists as something like:

Trying to radically reform the world isn't bad. The world does need reforming in many respects. But whether you are doing will do a good job at such a high stakes task depends on your personal virtue.

We can get evidence about how virtuous a person is by seeing how wisely and skillfully they comport themselves in lower-stakes situations where they don't have all the power. Wise and virtuous people can generally make meaningful progress on their goals without needing ultimate power over everything, and without remaking the whole world in one shot.

Therefore, if a given person's plan is to attain ultimate power, and only then use it to remake the world (instead of a more incremental process that doesn't depend on centralizing power in their own hands) that's a big red flag that even if they did

... (read more)

Yes, great summary, I fully endorse it.

I'm also reminded of something that Val used to say: "power felt is power wasted". Deploying a lot of force to make things go your way is very inefficient.

If you're skilled, you should be able to get what you're aiming for while deploying very little actual force ("speak softly and carry a big stick" for instance, but also using soft power and good leadership more generally). Someone with a little power and a lot of skill can often do as much (and with less collateral damage) as someone with a lot of power trying to muscle through.

Power is definitely usef... (read more)

I broadly agree with this comment too, though not as much as I agree with the other one.

Power felt can also be a kind of honesty—e.g. if a law is backed by force, then it's often better for this to be unambiguous, so that people can track the actual landscape of power.

(Of course, being unambiguous about how much force backs up your laws can also be a kind of power move. I expect that there are ways to get the benefits of honesty without making it a power move, but I don't have enough experience with this to be confident.)

In other words, I expect that the kind of inefficiency Val is talking about here is actually sometimes load-bearing for accountability.

Can you elaborate on that? I've never heard anyone say that before.

This version of the criticism I do kinda agree with. However I think it's relatively mild.

Brandon Sanderson's books have some interesting variants of this as well.


Spoilers for the Mistborn series (to the best of my recollection and with some consulting of the fandom wiki) (putting in collapsible for now, because I can't get spoiler blocks to work with this new editor):


At the end of The Well of Ascension, Vin gets a hold of the power of Ruin (one of the Shards, roughly god-like entities of the universe). She knows that she can use it to eliminate lots of the world's atrocities etc. Sazed, informed by a misty ghost, tells her that "it's a trap" a

... (read more)

Very interesting.

Relatedly, A Practical Guide to Evil is one of my favorite books/series, and grapples with the tension between trust and power very well. It's one of the very few narratives I've seen written skillfully enough that the protagonist giving up power didn't seem straightforwardly stupid to me (even when I was in a classic rationalist mindset).

Interesting. Looks like plausibly in a similar vibe/direction as Venkatesh Rao's Be Slightly Evil.

It seems plausible that what you suggest is one significant contributor. Here's one more thing that imo plausibly contributes significantly:

  • Most of these people are consequentialists, i.e. they think of ethics in terms of sth like designing a good spacetime block. [1] Like, when making a decision, you are making a decision as if standing outside the universe and choosing which of two spacetime blocks [2] is better. Given this view of ethics, it is very natural to imagine a future in which there actually is some guy that designs/choo
... (read more)

"Unilateralist consequentialists think they're just temporarily embarrassed omnipotentates."

Most of these people are consequentialists

Given this view of ethics, it is very natural to imagine a future in which there actually is some guy that designs/chooses a good spacetime block, and it becomes somewhat less natural to imagine futures in which the spacetime block keeps getting "designed/chosen" in a messy way by all the messy stuff inside the spacetime block, with the designing/choosing and the being-valuable done by the same entities

This seems to fit extremely poorly with the number of rules-based religions that have creator gods. It seems like your argument is proving too much.

There's a story that the reason why you have a lot of anime that goes in very weird sexual directions is that Japan is an incredibly sexually repressive society, so the Japanese channel their compressed libido towards art.

I don't really believe this story, but I think the pattern it exemplifies is at play here.

Rats have very ambitious goals of fixing the world. They see that the world is in a pretty bad shape and demands fixing. But they can't fix it. No one else can fix it either (nihil supernum, deep atheism), and even if they could, it would likely be bad, because their values are not yours (even deeper atheism). So you yearn for a world in which you smash those limitations. "Power" (or a specific kind of it) is the thing you need, and its value is not bounded, so the threshold of a superstimulus is largely your imagination, perhaps constrained by some ontological assumptions. That's why this theme recurs through ratfic so much.

This seems somewhat obvious to me[1] and so I am somewhat surprised that people in the comments mostly seem to explain it in terms of game-theoretic (or other) realism or people having consequentialist-ish views, etc.

  1. ^

    partly through introspection + the Co

... (read more)

It recently occurred to me that powerful gods caring about humans is the baby version of the phenomenon being described in the OP.

"Sure, I cannot hope to ascend to power, but there is a deity that knows about me and cares about me and can do a lot of stuff and will move the heavens to rig the matters in my favor if I pray enough, sacrifice enough, and exercise virtue enough."

E.g., the whole idea of the Last Judgment is quite reminiscent of many quotes in the OP.

 because their values are not yours

First of all, there might exist cases when a god with different values is preferable to the current world state (e.g. if the world is clearly heading towards self-destruction which would terrify even the god). Additionally, I doubt that the theme of gaining power and fixing the world recurs just through ratfic and not through fiction whose authors display some bias which I cannot describe more precisely than "finding it hard to restrain themselves." Finally, Max Harms has been trying to construct an agent whose sole goal... (read more)

First of all, there might exist cases when a god with different values is preferable to the current world state (e.g. if the world is clearly heading towards self-destruction which would terrify even the god).

Sure. Maybe you misunderstood me. I didn't write this as an endorsement of the "even deeper atheism" view.

First of all, there might exist cases when a god with different values is preferable to the current world state (e.g. if the world is clearly heading towards self-destruction which would terrify even the god).

Sure, it's not an exclusively ratfic i... (read more)

I have unvoted this comment because I can't decide whether I feel happy that I posted it. However, I did feel it was important to leave it here anyway.

I agree with the complaint about rationalist fiction. Your choice of concern example is understandable and I would also find it disturbing if I experienced it. I have a similar sense of disturbing feeling when considering the memetics of other modern ideologies, and I hope to someday become confident that your choice of who to criticize does not have a systematic exception. I can't tell if it's real, but I have a sense of isolated demand for rigor when you pick on the left and center but not the right.

it seems to me that left vs right isn't a particularly important dimension compared to the dimensions of auth-vs-liberty, prosocial-vs-antisocial, and egalitarian-vs-takeovertheworldism, that we should be focusing on broad-spectrum anti-authoritarianism and prevention of power concentration; in which case, I would hope you can also criticize authoritarianism on the right. But what I see is someone who endorses anti-egalitarianism and hasn't visibly engaged with the value prop for egalitarianism or how you would achieve value satisfactio... (read more)

Alternate explanation: ratfic tends to come out this way because it prioritizes a certain kind of economic / game-theoretic realism, and in the long term multi-polar equilibria just aren't that realistic in many settings (including, IMO, our present non-fictional setting...).

Like, regardless of how good / bad / ethical taking absolute power is, it's often inevitable that some entity or faction will end up winning decisively, or at least negotiating some kind of lasting truce or grand bargain, where the stakes, outcome, and enforcement are determined by hard power.

Another subversion example:

Three Worlds Collide comes to mind as another partial subversion of the trope, and also illustration of my point - the superhappies are the ones trying to impose absolute power on the other civilizations in the story, and the humans in the true ending blow up their own planet just to be left alone. But once you're in a setting with intergalactic civilizations, conflicting values, and access to WMDs, there's no way to avoid reckoning with hard power and decisive outcomes.

Alternate explanation: ratfic tends to come out this way because it prioritizes a certain kind of economic / game-theoretic realism, and in the long term multi-polar equilibria just aren't that realistic in many settings (including, IMO, our present non-fictional setting...).

I generally think rationalists gesture at this as inevitable more than they actually demonstrate that it is inevitable; i.e., long term multi-polar equilibria has been quite sticky for Westphalian states or for plankton.

Eh, conversely I think that historical examples that are not at the technological and competitive frontier are not very useful for reasoning about the limiting behavior and outcomes of AGI. History, nature, and business are full of examples of both unipolar and multipolar dynamics that were at equilibria for a while... until they were disrupted in some form or another, often forcefully and suddenly.

Another strain of thought from the early days of OB/LW is that the only or main alternative to a Singleton in the long run are Malthusian scenarios. I remember writing Non-Malthusian Scenarios (2009) to push back against this, but looking at it now, most of the non-Malthusian/non-Singleton scenarios aren't actually that plausible or attractive.

I wrote a bit about Malthus maybe having less teeth than he seems here: https://tomasbjartur.bearblog.dev/contra-my-own-doomerism/

Not particularly well thought out and I may be rehashing things, but I haven’t seen it put this way before.

> That is, in the worst case they could just behave exactly like a pure replicator. And they could do this without actually surrendering their values. So any argument of the form "there is no way anything that cares about us can survive in Malthusian equilibrium" seems false.

I think it's quite plausible that this is actually not possible, i.e., either at technological maturity or in the runup to it, transmitting values like caring about humans into the next generation of agents is actually difficult or costly enough that such agents are outcompeted and ... (read more)

I'd say ratfics are more about becoming God, and as God you can naturally Fix the world. So you can view rats as atheists who believe that since God doesn't exist, we must build Him.

Edit: Really, ratfics are about becoming more you are, with becoming God as the natural limit.

as God you can naturally Fix the world

This phrase reminds me of a Russian sci-fi piece literally named Hard to Be a God. I expect this piece to be relevant, but I find it hard to explain the relevance without spoilers.

Your overall point seems true and important. The virtues that are enshrined by the rat-fic tradition are primarily "sanity in the midst of insanity (eg. social decoupling and thinking for yourself)" and "heroic responsibility". These attitudes are not strictly counter to cooperation or working with the existing systems of the world, but they sure do tend to push in the opposite direction. The more you distrust the world, the less you think that you should cooperate with it.

And further, actual practice, "thinking for yourself" directly cuts against human political coordination, since political coalitions almost always maintain unity by coordinating their beliefs, and challenging those coordination beliefs undermines the coalition.

(I find it amusing when people sometimes say that the "one job" of rationalists is to be able to coordinate. I think that's ass-backwards. The rationalist tradition is about prioritizing independence of thought and the epistemology to discern the truth at the expense of human coordination. There's some hope there's another even stronger mode of coordination on the other side of the uncanny valley, but that, so far, remains an unvalidated hope.[1])

However, ... (read more)

Eliezer has said that one of the reasons he writes fanfiction, is that he doesn't have to invent the world. All of the horror and badness was already present in the source material.

A large majority of fantasy settings don't have literal hells as a key component, so I think my point is still applicable to Project Lawful if you replace "design them that way" with the more general "select for that trait".

I do agree that this is a good point with regards to HPMoR, which is one reason why I didn't include HPMoR in my original list of examples.

I think one of the other works has a wrong explanation as well, namely

(title)

Worth The Candle

(heavy spoilers)

WTC actually has an extreme amount of power-seeking behavior, exactly as the post says, but the actual end happens that way, it appears, because the author believes that retiring his characters in a manner of their choosing is desirable. To this end, one character abdicates responsibility, living out their life on earth. Another is given control of the universe, abolishes Hell, and then allows all the universe's participants to weigh in on the

... (read more)

I… am not sure how much gentler the author could have made this, to be honest. A singleton forms and allows diverse values— basically ideal.

This is precisely the attitude I am critiquing, and therefore I don't find your comment very persuasive.

What I mean is that your theory pays out as a ban on stories about any singleton, including in stories where the singleton does un-singletonly things. If that's the case, any author who expects there are RSI-dynamics in their story is verboten, which is rough if it makes sense!

My understanding of your dynamic has a high false-positive rate: if I wrote fiction about Napoleon, it would look power-seeking (because he was), but this doesn't seem like a useful red flag to me.

I don't think Unsong fits the pattern.

Aaron doesn't take over the world alone. He merges with seven other wildly different minds, including the villainous Dylan Alvarez. "In William Blake’s prophecies, Albion was the entity formed at the end of time, when all of the different aspects of the human soul finally came together to remake the world", as one of them says.

And I don't think the ending is about recreating the world as some kind of rationalist utopia (how would you do that with Dylan an Erica on the team?) - I interpret it more as a "cycle continues" ending where they carry forward God's already perfect plan into a new world.

See for example this point in the Tosefta, where Scott explains all the Easter eggs:

"As for THARMAS, seven of the ten towers were smoking ruins; the other three were heavily scarred. In the epilogue, THARMAS is going to be used to make the new universe. Seven of ten towers destroyed plus the rest damaged = seven of ten sephirot cracked plus the rest damaged, indicating the new universe will work the same as our own."

I could debate these details but honestly if we rank all books by "to what extent does the protagonist end up with total power over the world at the end" these seem like they shift Unsong from the top 0.01% to the top 0.02%, or something like that?

For example, yes they're carrying forward God's perfect plan because everything in Unsong is. But that plan still involves them conceptualizing the kind of world they want and wielding God's name to make it happen.


I noticed this a long time ago and tried to write a ratfic that didn't have this dynamic; I didn't get particularly far, mostly because I don't think I'm that great of a fiction writer.

I think a large part of this is which settings ratfic writers choose to write fanfic in. It is very easy to take JK Rowling's Harry Potter setting and put ratfic in it, because it's about as screwed up as the real world, and that calls for making major changes / doesn't naturally call for fitting into the system.

I was writing a MLP fanfic, and the My Little Pony setting is way less screwed up; the protagonist, rather than being a frustrated genius who isn't taken seriously by his parents or teachers, is a pampered prodigy who great things are expected of and whose education is being carefully attended to accordingly. If Twilight goes to Celestia with some complaint about how society is arranged, Celestia encourages her to write a memo to the relevant minister and then get into a policy debate which considers all of the relevant factors.

[Separately I tried writing Warhammer ratfic, which mostly turned into a meditation on how much it sucks to be in an epistemically hostile environment, and the Empire ... (read more)

I think another part of it is... lack of comfort with responsibility? In the narrow, local sense which I think makes for a good minister or romantic partner but is not the heroic responsibility of the CEO or God-Emperor or whatever.

I don't quite get this bit.

Yeah I'm probably trying to pack too many things in together. To expand on it:

I think there's something that one can get from, for example, taking care of a garden, or a tank of shrimp, or whatever. Rationality helps a lot with it; you need to notice things, you often need to sweep away your preconceptions, you often need to rearrange how you orient to the world.

Harry really doesn't demonstrate much of that; he couldn't

keep alive a pet rock, after all.

And I think as you go thru the list of ratfic heroes, most of them also don't have these sorts of responsibilities, or have them in a way that advances the plot instead of being the plot. (Miles Vorkosigan makes a lot of his feudal duties, but I think would very much not seem like a hero to a feudal audience, instead of something more like a tribal trickster deity.)

Part of this also is that it's an ongoing relationship. You don't get your pet to a good state and then declare mission accomplished; you instead have it occupying a bit of your attention, adjusting it as necessary. There's a way it's larger than what fits into your models in a way that is often breaking and expanding them, rather than being something that you can fit into... (read more)

Another way of putting this is in terms of the distinction between two types of optimization: selection and control.

Ratfic typically thinks of improving the world as a selection problem. Selecting a better world from the space of possible worlds is neat and elegant and lets you solve all problems at once. The only issue is that you need to gain absolute power first in order to be able to select the future you want.

Whereas you can also think about improving the world as a control problem, where you're gradually nudging the world towards being better. This is less narratively satisfying when you're a highly systematizing thinker, because you want to be able to identify the single root problem and take it out in one fell swoop (the same style of thinking that the communists were doing). But it's much more robust when you're in a world full of other people all of whom are also trying to exert influence.

IIRC none of Harry in HPMoR, Aaron in Unsong or Naruto in Waves Arisen actually meaningfully improved the world before taking it over—if anything, they mostly made it worse.

(Harder to evaluate this for the r!Animorphs or Keltham, because they were operating in such adversarial environments. And I don't remember Worth the Candle well enough to say one way or the other.)

And I don't remember Worth the Candle well enough to say one way or the other.

I think it's... more like a wash? A lot of this depends on what you think about Anglecynn / its internal politics.

[Haven't read those fictions or the quoted spoilers] Glad to hear you saying this. In fact, I've had a couple quite concerning conversations with a couple different people in which I'm like "so what would you do if you accidentally invented a Friendly AGI in your basement one day? what are some of your important first actions?" and they don't give one of the incredibly obvious and important answers, and don't even necessarily agree to it after I say it.

I'll add that the HM attitude extends to various other things. For example, some people around here are c... (read more)

one does want to "bring in many stakeholders to the conversation as coequal voices", because that's how you be non-HM

I think it's quite shameful how rationalists aim to optimize the world and yet are generally uninterested in what life is like for many kinds of normal people, or what problems they face. To rule well, you should know what life is like for your subjects, and what they care about, at minimum.

But there's also the pattern where "if your users tell you about a problem, they're always right, and if they tell you the solution, they're almost always wrong." And many common forms of "consulting all the stakeholders" ends up giving a veto to special interests, which ends up strangling liberty.

There are some to my knowledge unsolved problems here.


To rule well, you should know what life is like for your subjects, and what they care about, at minimum.

This is still totally High Modernist. To actually rule well you should actually not try to rule as much and should actually try to share power, which includes giving up power. Not giving up all power, but, you know, you leave after your second term in office, and you create a parliament.

I agree on all counts.

High Modernist

For example, some people around here are contemptuous of bioethics. 

I do wonder how much of this is "contemptuous of bioethics done badly". I can see the argument for "that means we need to do bioethics well, not continue to cede ground" but I do think it's important to be honest about when fields are failing.

it's important to be honest about when fields are failing.

I agree, naturally. I criticize bioethicists as a group, precisely because they are, for the most part, AFAICT, failing to lead on moral questions around reprogenetics.

However, I think one has to make multiple updates. That observation indicates that bioethicists may not have the right abilities or motivations or other properties (determination, grit, courage, sanity, wisdom, what have you), and on those grounds could be dismissed as individuals or even as a group; but it also indicates that the ... (read more)

I agree this is overall kinda concerning and suggestive of a blindspot. But, I want to somewhat disagree about Planecrash.

The thing that happens there feels like "A small, powerless country goes about inventing nukes, demands it gets invited to the UN-equivalent of people who already have nukes, and, then negotiates with the rest of the people there on even-ish terms. It was already the case that the UN was dominated by people with enough military power to fuck shit up and negotiations therebetween."

and, I certainly see the "rationalist wants to take over... (read more)

Your analogy seems a bit skewed because on the scale "how much of a world takeover is this?", "gaining the ability to unilaterally destroy the world" scores much higher than merely "gaining nukes". If becoming a nuclear power let you unilaterally destroy the world, the US would have tried much harder to limit their spread!

It seems more like "a small powerless country seizes the USSR's entire nuclear weapons stockpile (or creates an equivalently large one of their own) and tells everyone that they'll cause nuclear armageddon unless their demands are satisfi... (read more)

I think the existence of Hell is incredibly morally relevant and also very game-theoretically relevant too. I probably don't need to elaborate on the morality side, but for the game theory: IIUC, Keltham really would prefer the universe be destroyed (with him in it!) than that Hell continue to exist. This is different from the stereotypical doomsday threat, which is made by someone who doesn't actually want doomsday to happen but is hoping that other people will fear it even more and cave.

Yes, I agree. However, as I mentioned in my OP, I think that the prominence of Hell in stories like Unsong and Project Lawful is partly due to them functioning as plot devices to make taking over the world not just ethical but in fact morally obligatory.

Analogously, if a bunch of 19th-century Marxist fiction featured working conditions far harsher than any that existed in the real world, which compelled the heroes to launch a proletarian revolution, you wouldn't just think "this makes total sense given the fictional premise", you'd also think "the fictional premise was chosen to help the authors make the thing they already supported (and wanted to write about) seem morally good".

And "take over the world for good reasons" was IIRC MIRI's actual plan (hidden under the terminology "decisive strategic advantage" or "pivotal act" or similar).

Ok, but factory farms really do exist? And I don't think it's an exaggeration to call them hellish, or hells-on-earth.

And most of the beings that live in our civilization, depending on how you count, live and die in those factory farms.

So it seems that our world is not very different from that of Unsong or Project Lawful, in this respect? Even if the authors chose that setting for literary convenience, and it is only accidentally reflective of reality.



I do take your point, but factory farms are way more temporary than hells, which seems more relevant than the absolute level of suffering in them re whether to take over the world. (They're also far less bad than any hell portrayed by a creative rationalist author.)

Makes sense yeah.

Still though, I agree with Raemon's characterization of the Planecrash story.

I was going to say that this seems like another one of Eliezer's founder effects, but he actually wrote about not trusting humans with too much power, in Creating Friendly AI 1.0 (2001):

Among humans, the only practical way to maximize actual freedom (the percentage of actions executed without interference) is to ensure that no human entity has the ability to interfere with you—a consequence of humans having an innate, evolved tendency to abuse power. Thus, a lot of our ethical guidelines (especially the ones we’ve come up with in the twentieth century) sta

... (read more)

Spoiler

HJPEV is bound by a magical oath that prevents this human failing in the same way it is prevented in an agent that meets tiling desiderata. This is explicit in the text. E-Book draft, 2015, chapter 113.

Admittedly this both assumes that the "time of peril" hypothesis is correct and can be handled while maintaing human freedom, and the solution only (in maximum robustness) binds until the end of this time.

My understanding of HPMOR is limited as I've only read a few chapters, but looking up the text you cite, it doesn't seem to prevent most forms of abuse of power.

I always liked this as a fun mini-ratfic, and it doesn't fall into this pattern, despite the humorously extreme Mary-Sue-ness of its protagonist: https://www.lesswrong.com/posts/LYXb2fLkGDRXoAx7M/timothy-chu-origins-chapter-1

On the other hand, 3 Worlds Collide is an interesting case study:

The first ending is essentially a "use godlike tech-powers to optimize the world" scenario, except that it's being carried out by the superhappies, rather than a human protagonist. The superhappies do actually care about human wellbeing and try to compromise with human

... (read more)

I messaged the team on intercom yesterday and images are now properly spoilered

I basically agree with what you notice, and think that this is what you'd expect if rationalists were mostly normal relative to other people in their goals, which are mostly selfish and dicatorial, but are more intelligent and can think farther ahead about what instrumental goals that they imply for their terminal goals.

Or put another way, the thing that rationalists are doing here are things lots of other people would likely do if they were more intelligent, and the truth of the matter is that most people just like all-powerful dictatorships, almost no matter their ideology.

This is pretty straightforwardly not true, there are plenty of academics (for example) who are as smart as rationalists but don't do very broad instrumental reasoning.

There are also plenty of people who don't fantasize about becoming all-powerful dictators.

I think that the hunger to become god is an unusually rationalist trait. Honestly it's somewhat reminiscent of sociopathy, but fortunately few rationalists seem to be sociopaths. However, I do think a sufficient level of fear of death causes some overlapping traits, e.g. a mentality in which more power i... (read more)

This is pretty straightforwardly not true, there are plenty of academics (for example) who are as smart as rationalists but don't do very broad instrumental reasoning.

Fair point, I was generalizing too much here.

I agree with the literal claim that plenty of people don't fantasize about becoming all-powerful dictators, but I'd say the percentage of people who don't fantasize (including in their heads and not speaking about it) becoming dictators or don't believe that an all-powerful dictator is necessary to solve problems/have a good future is much closer t... (read more)

Many of Paul Christiano's writings were valuable corrections to the dominant Yudkowskian paradigm of AI safety. However, I think that many of them (especially papers like concrete problems in AI safety and posts like these two) also ended up providing a lot of intellectual cover for people to do "AI safety" work (especially within AGI companies) that isn't even trying to be scalable to much more powerful systems.

I want to register a prediction that "gradual disempowerment" will end up being (mis)used in a similar way. I don't really know what to do about this, but I intend to avoid using the term myself. My own research on related topics I cluster under headings like "understanding intelligence", "understanding political philosophy", and "understanding power". To me this kind of understanding-oriented approach seems more productive than trying to create a movement based around a class of threat models.

I do agree there is some risk of the type you describe, but mostly it does not match my practical experience so far.

The approach to "avoid using the term" makes little sense. There is a type difference between area of study ('understanding power') and dynamic ('gradual disempowerment'). I don't think you can substitute term for area of study for term for a dynamic or thread model, so avoiding using the term could be done mostly by either inventing another term for the the dynamic, or not thinking about the dynamic, or similar moves, which seem epistemically unhealthy.

In practical terms I don't think there is much effort to "create a movement based around a class of threat models". At least as authors of the GD paper, when trying to support thinking about the problems, we use understanding-directed labels/pointers (Post-AGI Civilizational Equilibria), even though in many ways it could have been easier to use GD as a brand.

"Understanding power" is fine as a label for part of your writing, but in my view is basically unusable as term for coordination. 

Also, in practical terms, gradual disempowerment does not seem particularly convenient set of ideas for justifying that working in... (read more)

mostly it does not match my practical experience so far

I mostly wouldn't expect it to at this point, FWIW. The people engaged right now are by and large people sincerely grappling with the idea, and particularly people who are already bought into takeover risk. Whereas one of the main mechanisms by which I expect misuse of the idea is that people who are uncomfortable with the concept of "AI takeover" can still classify themselves as part of the AI safety coalition when it suits them.

As an illustration of this happening to Paul's worldview, see this Vox ar... (read more)

Rob Miles suggested 'inexorable disempowerment' as maybe better on a call where we discussed the default associations of 'gradual'.

Also, in practical terms, gradual disempowerment does not seem particularly convenient set of ideas for justifying that working in an AGI company on something very prosaic which helps the company is the best thing to do.

The bigger issue, as Jackson Wagner says is that there's a very real risk that will be coopted by people who want to talk mostly about present-day harms of AI, and thus at best siphoning resources from actually useful work on gradual disempowerment threats/AI x-risk in general, and at worst creating polarization around gradual disempowermen... (read more)

Is your idea that "gradual disempowerment" isn't a real problem or that it's a distraction from actual issues? I've heard arguments for both, so I'm not sure what the details of your beliefs are. Personally, I see "gradual disempowerment" as a process that has already begun, but the main danger is AI deciding we should die, not humans living in comfort while all the real power is held by AI.

The impression I got from Ngo's post is that:

  • assorted varieties of gradual disempowerment do seem like genuine long term threats
  • however, by the nature of the idea, it involves talking a lot about relatively small present-day harms from AI
  • therefore gradual disempowerment is highly at-risk of being coopted by people who mostly just want to talk about present day harms, distracting from both AI x-risk overall and even perhaps from gradual-disempowerment-related x-risk 

One fairly strong belief of mine is that Less Wrong's epistemic standards are not high enough to make solid intellectual progress here. So far my best effort to make that argument has been in the comment thread starting here. Looking back at that thread, I just noticed that a couple of those comments have been downvoted to negative karma. I don't think any of my comments have ever hit negative karma before; I find it particularly sad that the one time it happens is when I'm trying to explain why I think this community is failing at its key goal of cultivating better epistemics.

There's all sorts of arguments to be made here, which I don't have time to lay out in detail. But just step back for a moment. Tens or hundreds of thousands of academics are trying to figure out how the world works, spending their careers putting immense effort into reading and producing and reviewing papers. Even then, there's a massive replication crisis. And we're trying to produce reliable answers to much harder questions by, what, writing better blog posts, and hoping that a few of the best ideas stick? This is not what a desperate effort to find the truth looks like.

And we're trying to produce reliable answers to much harder questions by, what, writing better blog posts, and hoping that a few of the best ideas stick? This is not what a desperate effort to find the truth looks like.

It seems to me that maybe this is what a certain stage in the desperate effort to find the truth looks like?

Like, the early stages of intellectual progress look a lot like thinking about different ideas and seeing which ones stand up robustly to scrutiny.  Then the best ones can be tested more rigorously and their edges refined through experimentation.  

It seems to me like there needs to be some point in the desparate search for truth in which you're allowing for half-formed thoughts and unrefined hypotheses, or else you simply never get to a place where the hypotheses you're creating even brush up against the truth.

In the half-formed thoughts stage, I'd expect to see a lot of literature reviews, agendas laying out problems, and attempts to identify and question fundamental assumptions. I expect that (not blog-post-sized speculation) to be the hard part of the early stages of intellectual progress, and I don't see it right now.

Perhaps we can split this into technical AI safety and everything else. Above I'm mostly speaking about "everything else" that Less Wrong wants to solve. Since AI safety is now a substantial enough field that its problems need to be solved in more systemic ways.

In the half-formed thoughts stage, I'd expect to see a lot of literature reviews, agendas laying out problems, and attempts to identify and question fundamental assumptions. I expect that (not blog-post-sized speculation) to be the hard part of the early stages of intellectual progress, and I don't see it right now.

I would expect that later in the process.  Agendas laying out problems and fundamental assumptions don't spring from nowhere (at least for me), they come from conversations where I'm trying to articulate some intuition, and I recognize some... (read more)

Ironically, some people already feel threatened by the high standards here. Setting them higher probably wouldn't result in more good content. It would result in less mediocre content, but probably also less good content, as the authors who sometimes write a mediocre article and sometimes a good one, would get discouraged and give up.

Ben Pace gives a few examples of great content in the next comment. It would be better to easier separate the good content from the rest, but that's what the reviews are for. Well, only one review so far, if I rememb... (read more)

The top posts in the 2018 Review are filled with fascinating and well-explained ideas. Many of the new ideas are not settled science, but they're quite original and substantive, or excellent distillations of settled science, and are often the best piece of writing on the internet about their topics.

You're wrong about LW epistemic standards not being high enough to make solid intellectual progress, we already have. On AI alone (which I am using in large part because there's vaguely more consensus around it than around rationality), I think you wouldn't have... (read more)

As mentioned in my reply to Ruby, this is not a critique of the LW team, but of the LW mentality. And I should have phrased my point more carefully - "epistemic standards are too low to make any progress" is clearly too strong a claim, it's more like "epistemic standards are low enough that they're an important bottleneck to progress". But I do think there's a substantive disagreement here. Perhaps the best way to spell it out is to look at the posts you linked and see why I'm less excited about them than you are.

Of the top posts in the 2018 review, and the ones you linked (excluding AI), I'd categorise them as follows:

Interesting speculation about psychology and society, where I have no way of knowing if it's true:

  • Local Validity as a Key to Sanity and Civilization
  • The Loudest Alarm Is Probably False
  • Anti-social punishment (which is, unlike the others, at least based on one (1) study).
  • Babble
  • Intelligent social web
  • Unrolling social metacognition
  • Simulacra levels
  • Can you keep this secret?

Same as above but it's by Scott so it's a bit more rigorous and much more compelling:

  • Is Science Slowing Down?
  • The tails coming apart as a metaph
... (read more)

(Thanks for laying out your position in this level of depth. Sorry for how long this comment turned out. I guess I wanted to back up a bunch of my agreement with words. It's a comment for the sake of everyone else, not just you.)

I think there's something to what you're saying, that the mentality itself could be better. The Sequences have been criticized because Eliezer didn't cite previous thinkers all that much, but at least as far as the science goes, as you said, he was drawing on academic knowledge. I also think we've lost something precious with the absence of epic topic reviews by the likes of Luke. Kaj Sotala still brings in heavily from outside knowledge, John Wentworth did a great review on Biological Circuits, and we get SSC crossposts that have that, but otherwise posts aren't heavily referencing or building upon outside stuff. I concede that I would like to see a lot more of that.

I think Kaj was rightly disappointed that he didn't get more engagement with his post whose gist was "this is what the science really says about S1 & S2, one of your most cherished concepts, LW community".

I wouldn't say the typical approach is strictly bad, there's value in thinking freshly... (read more)

This is only tangentially relevant, but adding it here as some of you might find it interesting:

Venkatesh Rao has an excellent Twitter thread on why most independent research only reaches this kind of initial exploratory level (he tried it for a bit before moving to consulting). It's pretty pessimistic, but there is a somewhat more optimistic follow-up thread on potential new funding models. Key point is that the later stages are just really effortful and time-consuming, in a way that keeps out a lot of people trying to do this as a side project alongside a separate main job (which I think is the case for a lot of LW contributors?)

Quote from that thread:

Research =

a) long time between having an idea and having something to show for it that even the most sympathetic fellow crackpot would appreciate (not even pay for, just get)

b) a >10:1 ratio of background invisible thinking in notes, dead-ends, eliminating options etc

With a blogpost, it’s like a week of effort at most from idea to mvp, and at most a 3:1 ratio of invisible to visible. That’s sustainable as a hobby/side thing.

To do research-grade thinking you basically have to be independently wealthy and accept 90% d

... (read more)

Thanks, these links seem great! I think this is a good (if slightly harsh) way of making a similar point to mine:

"I find that autodidacts who haven’t experienced institutional R&D environments have a self-congratulatory low threshold for what they count as research. It’s a bit like vanity publishing or fan fiction. This mismatch doesn’t exist as much in indie art, consulting, game dev etc"

Also, I liked your blog post! More generally, I strongly encourage bloggers to have a "best of" page, or something that directs people to good posts. I'd be keen to read more of your posts but have no idea where to start.

Thanks! I have been meaning to add a 'start here' page for a while, so that's good to have the extra push :) Seems particularly worthwhile in my case because a) there's no one clear theme and b) I've been trying a lot of low-quality experimental posts this year bc pandemic trashed motivation, so recent posts are not really reflective of my normal output.

For now some of my better posts in the last couple of years might be Cognitive decoupling and banana phones (tracing back the original precursor of Stanovich's idea), The middle distance (a writeup of a use... (read more)

Quoting your reply to Ruby below, I agree I'd like LessWrong to be much better at "being able to reliably produce and build on good ideas". 

The reliability and focus feels most lacking to me on the building side, rather than the production, which I think we're doing quite well at. I think we've successfully formed a publishing platform that provides and audience who are intensely interested in good ideas around rationality, AI, and related subjects, and a lot of very generative and thoughtful people are writing down their ideas here.

We're low on the ability to connect people up to do more extensive work on these ideas – most good hypotheses and arguments don't get a great deal of follow up or further discussion.

Here are some subjects where I think there's been various people sharing substantive perspectives, but I think there's also a lot of space for more 'details' to get fleshed out and subquestions to be cleanly answered:

... (read more)

"I see a lot of (very high quality) raw energy here that wants shaping and directing, with the use of lots of tools for coordination (e.g. better collaboration tools)."

Yepp, I agree with this. I guess our main disagreement is whether the "low epistemic standards" framing is a useful way to shape that energy. I think it is because it'll push people towards realising how little evidence they actually have for many plausible-seeming hypotheses on this website. One proven claim is worth a dozen compelling hypotheses, but LW to a first approximation only produces the latter.

When you say "there's also a lot of space for more 'details' to get fleshed out and subquestions to be cleanly answered", I find myself expecting that this will involve people who believe the hypothesis continuing to build their castle in the sky, not analysis about why it might be wrong and why it's not.

That being said, LW is very good at producing "fake frameworks". So I don't want to discourage this too much. I'm just arguing that this is a different thing from building robust knowledge about the world.

One proven claim is worth a dozen compelling hypotheses

I will continue to be contrary and say I'm not sure I agree with this.

For one, I think in many domains new ideas are really hard to come by, as opposed to making minor progress in the existing paradigms. Fundamental theories in physics, a bunch of general insights about intelligence (in neuroscience and AI), etc.

And secondly, I am reminded of what Lukeprog wrote in his moral consciousness report, that he wished the various different philosophies-of-consciousness would stop debating each other, go away ... (read more)

I feel like this comment isn't critiquing a position I actually hold. For example, I don't believe that "the correct next move is for e.g. Eliezer and Paul to debate for 1000 hours". I am happy for people to work towards building evidence for their hypotheses in many ways, including fleshing out details, engaging with existing literature, experimentation, and operationalisation.

Perhaps this makes "proven claim" a misleading phrase to use. Perhaps more accurate to say: "one fully fleshed out theory is more valuable than a dozen intuitively compelling ideas". But having said that, I doubt that it's possible to fully flesh out a theory like simulacra levels without engaging with a bunch of academic literature and then making predictions.

I also agree with Raemon's response below.

I think I'm concretely worried that some of those models / paradigms (and some other ones on LW) don't seem pointed in a direction that leads obviously to "make falsifiable predictions."

And I can imagine worlds where "make falsifiable predictions" isn't the right next step, you need to play around with it more and get it fleshed out in your head before you can do that. But there is at least some writing on LW that feels to me like it leaps from "come up with an interesting idea" to "try to persuade people it's correct" without enough checking.

(In the case of IFS, I think Kaj's sequence is doing a great job of laying it out in a concrete way where it can then be meaningfully disagreed with. But the other people who've been playing around with IFS didn't really seem interested in that, and I feel like we got lucky that Kaj had the time and interest to do so.)

Yepp, I agree with this. I guess our main disagreement is whether the "low epistemic standards" framing is a useful way to shape that energy. I think it is because it'll push people towards realising how little evidence they actually have for many plausible-seeming hypotheses on this website.

A housemate of mine said to me they think LW has a lot of breadth, but could benefit from more depth. 

I think in general when we do intellectual work we have excellent epistemic standards, capable of listening to all sorts of evidence that other communities and fi... (read more)

In general when we do intellectual work we have excellent epistemic standards, capable of listening to all sorts of evidence that other communities and fields would throw out, and listening to subtler evidence than most scientists ("faster than science")

"Being more openminded about what evidence to listen to" seems like a way in which we have lower epistemic standards than scientists, and also that's beneficial. It doesn't rebut my claim that there are some ways in which we have lower epistemic standards than many academic communities, and that's harmful.

In particular, the relevant question for me is: why doesn't LW have more depth? Sure, more depth requires more work, but on the timeframe of several years, and hundreds or thousands of contributors, it seems viable. And I'm proposing, as a hypothesis, that LW doesn't have enough depth because people don't care enough about depth - they're willing to accept ideas even before they've been explored in depth. If this explanation is correct, then it seems accurate to call it a problem with our epistemic standards - specifically, the standard of requiring (and rewarding) deep investigation and scholarship.

LW doesn't have enough depth because people don't care enough about depth - they're willing to accept ideas even before they've been explored in depth. If this explanation is correct, then it seems accurate to call it a problem with our epistemic standards - specifically, the standard of requiring (and rewarding) deep investigation and scholarship.

Your solution to the "willingness to accept ideas even before they've been explored in depth" problem is to explore ideas in more depth. But another solution is to accept fewer ideas, or hold them much more p... (read more)

"If we accept fewer ideas / hold them much more provisionally, but provide a clear path to having an idea be widely held as true, that creates an incentive for people to try & jump through hoops--and this incentive is a positive one, not a punishment-driven browbeating incentive."

Hmm, it sounds like we agree on the solution but are emphasising different parts of it. For me, the question is: who's this "we" that should accept fewer ideas? It's the set of people who agree with my argument that you shouldn't believe things which haven't been fleshed out v... (read more)

Fair enough. I'm reminded of a time someone summarized one of my posts as being a definitive argument against some idea X and me thinking to myself "even I don't think my post definitively settles this issue" haha.

Yeah, this is roughly how I think about it.

I do think right now LessWrong should lean more in the direction the Richard is suggesting – I think it was essential to establish better Babble procedures but now we're doing well enough on that front that I think setting clearer expectations of how the eventual pruning works is reasonable. 

I wanted to register that I don't like "babble and prune" as a model of intellectual development. I think intellectual development actually looks more like:

1. Babble

2. Prune

3. Extensive scholarship

4. More pruning

5. Distilling scholarship to form common knowledge

And that my main criticism is the lack of 3 and 5, not the lack of 2 or 4.

I also note that: a) these steps get monotonically harder, so that focusing on the first two misses *almost all* the work; b) maybe I'm being too harsh on the babble and prune framework because it's so... (read more)

I basically agree with your 5-step model (I at least agree it's a more accurate description than Babel and Prune, which I just meant as rough shorthand). I'd add things like "original research/empiricism" or "more rigorous theorizing" to the "Extensive Scholarship" step. 

I see the LW Review as basically the first of (what I agree should essentially be at least) a 5 step process. It's adding a stronger Step 2, and a bit of Step 5 (at least some people chose to rewrite their posts to be clearer and respond to criticism)

...

Currently, we do get non-zero E... (read more)

  1. All else equal, the harder something is, the less we should do it.

  2. My quick take is that writing lit reviews/textbooks is a comparative disadvantage of LW relative to the mainstream academic establishment.

In terms of producing reliable knowledge... if people actually care about whether something is true, they can always offer a cash prize for the best counterargument (which could of course constitute citation of academic research). The fact that people aren't doing this suggests to me that for most claims on LW, there isn't any (reasonably rich) pe... (read more)

I’ve been thinking about the idea of “social rationality” lately, and this is related. We do so much here in the way of training individual rationality - the inputs, functions, and outputs of a single human mind. But if truth is a product, then getting human minds well-coordinated to produce it might be much more important than training them to be individually stronger. Just as assembly line production is much more effective in producing almost anything than teaching each worker to be faster in assembling a complete product by themselves.

My guess is that t... (read more)

There's been a fair amount of discussion of that sort of thing here: https://www.lesswrong.com/tag/group-rationality There are also groups outside LW thinking about social technology such as RadicalxChange.

Imagine you took 5 separate LWers and asked them to create a unified consensus response to a given article. My guess is that they’d learn more through that collective effort, and produce a more useful response, than if they spent the same amount of time individually evaluating the article and posting their separate replies.

I'm not sure. If you put those 5 LWers together, I think there's a good chance that the highest status person speaks first and then the others anchor on what they say and then it effectively ends up being like a group project for school with the highest status person in charge. Some related links.

That’s definitely a concern too! I imagine such groups forming among people who either already share a basic common view, and collaborate to investigate more deeply. That way, any status-anchoring effects are mitigated.

Alternatively, it could be an adversarial collaboration. For me personally, some of the SSC essays in this format have led me to change my mind in a lasting way.

they're willing to accept ideas even before they've been explored in depth

People also reject ideas before they've been explored in depth. I've tried to discuss similar issues with LW before but the basic response was roughly "we like chaos where no one pays attention to whether an argument has ever been answered by anyone; we all just do our own thing with no attempt at comprehensiveness or organizing who does what; having organized leadership of any sort, or anyone who is responsible for anything, would be irrational" (plus some suggestions that I'm lo... (read more)

I feel somewhat inclined to wrap up this thread at some point, even while there's more to say. We can continue if you like and have something specific or strong you'd like to ask, but otherwise will pause here.

why doesn’t LW have more depth?

You have to realise that what you are doing isn't adequate in order to gain the motivation to do it better, and that is unlikely to happen if you are mostly communicating with other people who think everything is OK.

Lesswrong is competing against philosophy as well as science, and philosophy has broader criterion of evidence still. In fact , lesswrongians are often frustrated that mainstream philosophy takes such topics as dualism or theism seriously.. even though theres an abundance of Bayesian evidence for them.

One proven claim is worth a dozen compelling hypotheses, but LW to a first approximation only produces the latter.

Depends on the claim, right?

If the cost of evaluating a hypothesis is high, and hypotheses are cheap to generate, I would like to generate a great deal before selecting one to evaluate.

As mentioned in this comment, the Unrolling social metacognition paper is closely related to at least one research paper.

Right, but this isn't mentioned in the post? Which seems odd. Maybe that's actually another example of the "LW mentality": why is the fact that there has been solid empirical research into 3 layers not being enough not important enough to mention in a post on why 3 layers isn't enough? (Maybe because the post was time-boxed? If so that seems reasonable, but then I would hope that people comment saying "Here's a very relevant paper, why didn't you cite it?")

Much of the same is true of scientific journals. Creating a place to share and publish research is a pretty key piece of intellectual infrastructure, especially for researchers to create artifacts of their thinking along the way. 

The point about being 'cross-posted' is where I disagree the most. 

This is largely original content that counterfactually wouldn't have been published, or occasionally would have been published but to a much smaller audience. What Failure Looks Like wasn't crossposted, Anna's piece on reality-revealing puzzles wasn't crossposted. I think that Zvi would have still written some on mazes and simulacra, but I imagine he writes substantially more content given the cross-posting available for the LW audience. Could perhaps check his blogging frequency over the last few years to see if that tracks. I recall Zhu telling me he wrote his FAQ because LW offered an audience for it, and likely wouldn't have done so otherwise. I love everything Abram writes, and while he did have the Intelligent Agent Foundations Forum, it had a much more concise, technical style, tiny audience, and didn't have the conversational explanations and stories and cartoons that have... (read more)

Rohin wouldn't have written his coherence theorems piece or any of his value learning sequence, and I'm pretty sure about that because I personally asked him to write that sequence

Yeah, that's true, though it might have happened at some later point in the future as I got increasingly frustrated by people continuing to cite VNM at me (though probably it would have been a blog post and not a full sequence).

Reading through this comment tree, I feel like there's a distinction to be made between "LW / AIAF as a platform that aggregates readership and provides b... (read more)

 By "AN" do you mean the AI Alignment Forum, or "AIAF"?

I did suspect you'd confused it with the Alignment Newsletter :)

Thanks for chiming in with this. People criticizing the epistemics is hopefully how we get better epistemics. When the Californian smoke isn't interfering with my cognition as much, I'll try to give your feedback (and Rohin's) proper attention. I would generally be interested to hear your arguments/models in detail, if you get the chance to lay them out.

My default position is LW has done well enough historically (e.g. Ben Pace's examples) for me to currently be investing in getting it even better. Epistemics and progress could definitely be a lot better, b... (read more)

For the record, I think the LW team is doing a great job. There's definitely a sense in which better infrastructure can reduce the need for high epistemic standards, but it feels like the thing I'm pointing at is more like "Many LW contributors not even realising how far away we are from being able to reliably produce and build on good ideas" (which feels like my criticism of Ben's position in his comment, so I'll respond more directly there).

It seems really valuable to have you sharing how you think we’re falling epistemically short and probably important for the site to integrate the insights behind that view. There are a bunch of ways I disagree with your claims about epistemic best practices, but it seems like it would be cool if I could pass your ITT more. I wish your attempt to communicate the problems you saw had worked out better. I hope there’s a way for you to help improve LW epistemics, but also get that it might be costly in time and energy.

I just noticed that a couple of those comments have been downvoted to negative karma

Now they're positive again.

Confusing to me, their Ω-karma (karma on another website) is also positive. Does it mean they previously had negative LW-karma but positive Ω-karma? Or that their Ω-karma also improved as a result of you complaining on LW a few hours ago? Why would it?

(Feature request: graph of evolution of comment karma as a function of time.)

I'm confused, what is Ω-karma?

AI Alignment Forum karma (which is also displayed here on posts that are crossposted)

[-][anonymous]10

I'd be curious what, if any, communities you think set good examples in this regard. In particular, are there specific academic subfields or non-academic scenes that exemplify the virtues you'd like to see more of?

Maybe historians of the industrial revolution? Who grapple with really complex phenomena and large-scale patterns, like us, but unlike us use a lot of data, write a lot of thorough papers and books, and then have a lot of ongoing debate on those ideas. And then the "progress studies" crowd is an example of an online community inspired by that tradition (but still very nascent, so we'll see how it goes).

More generally I'd say we could learn to be more rigorous by looking at any scientific discipline or econ or analytic philosophy. I don't think most LW posters are in a position to put in as much effort as full-time researchers, but certainly we can push a bit in that direction.

[-][anonymous]40

Thanks for your reply! I largely agree with drossbucket's reply.

I also wonder how much this is an incentives problem. As you mentioned and in my experience, the fields you mentioned strongly incentivize an almost fanatical level of thoroughness that I suspect is very hard for individuals to maintain without outside incentives pushing them that way. At least personally, I definitely struggle and, frankly, mostly fail to live up to the sorts of standards you mention when writing blog posts in part because the incentive gradient feels like it pushes towards h... (read more)

It feels partly like an incentives problem, but also I think a lot of people around here are altruistic and truth-seeking and just don't realise that there are much more effective ways to contribute to community epistemics than standard blog posts.

I think that most LW discussion is at the level where "paying for mistakes" wouldn't be that helpful, since a lot of it is fuzzy. Probably the thing we need first are more reference posts that distill a range of discussion into key concepts, and place that in the wider intellectual context. Th... (read more)

Here is the best toy model I currently have for rational agents. Alas, it is super messy and hacky, but better than nothing. I'll call it the BAVM model; the one-sentence summary is "internal traders concurrently bet on beliefs, auction actions, vote on values, and merge minds". There's little novel here, I'm just throwing together a bunch of ideas from other people (especially Scott Garrabrant and Abram Demski).

In more detail, the three main components are:

  1. A prediction market
  2. An action auction
  3. A value election

You also have some set of traders, who can simultaneously trade on any combination of these three. Traders earn money in two ways:

  1. Making accurate predictions about future sensory experiences on the market.
  2. Taking actions which lead to reward or increase the agent's expected future value.

They spend money in three ways:

  1. Bidding to control the agent's actions for the next N timesteps.
  2. Voting on what actions get reward and what states are assigned value.
  3. Running the computations required to figure out all these trades.

Values are therefore dominated by whichever traders earn money from predictions or actions, who will disproportionately vote for values that are formulated in the same on... (read more)

increase the agent's expected future value

I wonder if there's a loopiness here is which breaks the setup (the expectation I'm guessing is relative to the prediction markets probabilities? Though it seems like the market is over sensory experiences but the values are over world states in general, so maybe I'm missing something). But it seems like if I take an action and move the market at the same time, I might be able to extract a bunch of extra money and acquire outsize control.

  1. Bidding to control the agent's actions for the next N timesteps

This seems like... (read more)

I've been studying and thinking about the physical side of this phenomenon in neuroscience recently. There are groups of columns of neurons in the cortex that form temporary voting blocks, regarding whatever subject that particular Brodmann area focuses on. These alternating groups have to deal with physical limits of how many groups the regions can stably divide into, which limits the number of active distinct hypotheses or 'traders' there can be in a given area at a given time. Unclear exactly what the max is, and it depends on the cortical region in que... (read more)

I think there are probably a lot of ways to build rational agents. The idea that general intelligence is hard in any absolute sense may be a biased by wanting to believe we're special, and for AI workers, that our work is special and difficult.

Can new traders be "spawned"?

Yepp, as in Logical Induction, new traders get spawned over time (in some kind of simplicity-weighted ordering).

I note that AI economies like this will often have explosively better credit assignment for information production than human economies can. Artificial agents can be copied or rolled back (erase memories), which makes it possible to reverse the receipt of information if an assessor concludes with a price that the seller considers too low for a deal. In human economies, that's impossible, you can't send a clone to value a piece of information then delete them if you decide not to buy that information (that's too expensive/illegal) nor can you wipe their mem... (read more)

Artificial agents can be copied or rolled back (erase memories), which makes it possible to reverse the receipt of information if an assessor concludes with a price that the seller considers too low for a deal.

Yepp, very good point. Am working on a short story about this right now.

Nice!

For comparison, this "Great Map of the Mind" is basically the standard academic philosophy picture.

My guess is that understanding merging is the key to most prediction-of-behavior issues (things that motivated and also foiled UDT, but not limited to known-in-advance preference setting). Two agents can coordinate if they are the same, or reasoning about each other's behavior, but in general they can be too complicated to clearly understand each other or themselves, can inadvertently diagonalize such attempts into impossibility, or even fail to be sufficiently aware of each other to start reasoning about each other specifically.

It might be useful to formu... (read more)

If traders can get access to control panel for actions of the external agent AND they profit from accurately predicting its observations, then wouldn't the best strategy be "create as much chaos as possible that is only predictable to me, its creator". So, traders that value ONLY accurate predictions will get the advantage?

I like this picture! But

Voting on what actions get reward

I think real learning has some kind of ground-truth reward. So we should clearly separate between "this ground-truth reward that is chiseling the agent during training (and not after training)", and "the internal shards of the agent negotiating and changing your exact objective (which can happen both during and after training)". I'd call the latter "internal value allocation", or something like that. It doesn't neatly correspond to any ground truth, and is partly determined by internal noise in the a... (read more)

I think real learning has some kind of ground-truth reward.

I'd actually represent this as "subsidizing" some traders. For example, humans have a social-status-detector which is hardwired to our reward systems. One way to implement this is just by taking a trader which is focused on social status and giving it a bunch of money. I think this is also realistic in the sense that our human hardcoded rewards can be seen as (fairly dumb) subagents.

I think this will by default lead to wireheading (a trader becomes wealthy and then sets reward to be very easy for i

... (read more)

I'd actually represent this as "subsidizing" some traders

Sounds good!

it's more a question of how you tweak the parameters to make this as unlikely as possible

Absolutely, wireheading is a real phenomenon, so the question is how can real agents exist that mostly don't fall to it. And I was asking for a story about how your model can be altered/expanded to make sense of that. My guess is it will have to do with strongly subsidizing some traders, and/or having a pretty weird prior over traders. Maybe even something like "dynamically changing the prior over tra... (read more)

Absolutely, wireheading is a real phenomenon, so the question is how can real agents exist that mostly don't fall to it. And I was asking for a story about how your model can be altered/expanded to make sense of that.

Ah, I see. In that case I think I disagree that it happens "by default" in this model. A few dynamics which prevent it:

  1. If the wealthy trader makes reward easier to get, then the price of actions will go up accordingly (because other traders will notice that they can get a lot of reward by winning actions). So in order for the wealthy trader to
... (read more)

they need to reward outcomes which only they can achieve,

Yep! But this didn't seem so hard for me to happen, especially in the form of "I pick some easy task (that I can do perfectly), and of course others will also be able to do it perfectly, but since I already have most of the money, if I just keep investing my money in doing it I will reign forever". You prevent this from happening through epsilon-exploration, or something equivalent like giving money randomly to other traders. These solutions feel bad, but I think they're the only real solutions. Alth... (read more)

An analogy that might be banal, but might be interesting:

One reason (the main reason?) that computers use discrete encodings is to make error correction easier. A continuous signal will gradually drift over time. Conversely, if the signal is frequently rounded to the nearest discrete value, then it might remain error-free for a long time. (I think this is also the reason why two most complicated biological information-processing systems use discrete encodings: DNA base pairs and neural spikes. EDIT: Neural spikes may seem continuous in the time dimension but the concept of "brain waves" makes me suspect that the time intervals between them are better understood as discrete.)

Separately, agents tend to define discrete boundaries around themselves—e.g. countries try to have sharp borders rather than fuzzy borders. One reason (the main reason?) is to make themselves easier to defend: with sharp borders there's a clear Schelling point for when to attack invaders. Without that, invaders might "drift in" over time.

The logistics of defending oneself vary by type of agent. For physical agents, perhaps fuzzy boundaries are just not possible to implement (e.g. humans need to literally hold the water inside us). However, many human groups (e.g. social classes) have initiation rituals which clearly demarcate who's in and who's out, even though in principle it'd be fairly easy for them to have a gradual/continuous metric of membership (like how many "points" members have gotten). We might be able to explain this as a way of giving them social defensibility.

A further potential extension here is to point out that modern hiveminds (Twitter / X / Bsky) changed group membership in many political groups from something explicit ("We let this person write in a our [Conservative / Liberal / Leftist / etc] magazine / published them in our newspaper") to something very fuzzy and indeterminate ("Well, they call themselves an [Conservative / Liberal / Leftist / etc] , and they're huge on Twitter, and they say some of the kinds of things [Conservative / Liberal / Leftist / etc] people say, so I guess they're an [Conservative / Liberal / Leftist / etc] .")

I think this is a really big part of why the free market of ideas has stopped working in the US over the last decade or two.

Yet more speculative is a preferred solution of mine; intermediate groups within hiveminds, such that no person can post in the hivemind without being part of such a group, and such that both person and group are clearly associated with each other. This permits:

  • Membership to be explicit
  • Bad actors (according to group norms) to be actually kicked out proactively, rather than degrading norms
  • Multi-level selection between group norms, where you can just block large groups that do not adopt truthseeking norms
  • More conscious shaping of the egregore.

But this solutioning is all more speculative than the problem.

I think this is a really big part of why the free market of ideas has stopped working in the US over the last decade or two.

Say more on this? I don't see the argument. It's also not clear why this would only affect the US for instance.

@1a3orn Could you also elaborate on why you think the ideas marketplace has became more dysfunctional over the last 10–20 years?

This probably generalizes beyond ~agent boundaries to general partitioning of "raw information blobs" into concepts/abstractions. You choose a way of making sense of the world that minimizes the expected future amount of [runtime correction by re-computing the abstraction from "raw sense data"] (minimally leaky abstractions).

(Well, this actually feels like a restatement of Markov/causal blankets, but the connection to your point stands, and it seems like an interesting restatement to me.)

yes! this is an important point. I don't quite know how to cash it out yet but I suspect I will eventually converge towards viewing concepts as "agents" which are trying to explain as much sensory data as possible while also cooperating/competing with each other.

I suspect I will eventually converge towards viewing concepts as "agents"

What is an "agent" in your ontology?

In one convo (at ILIAD 1), you named situational awareness as the criterion for distinguishing a part of an agent as a subagent. Is this also an important factor for calling something an "agent" simpliciter?

Certainly simpler to police if you have a clear rule for in vs out

have been thinking about this since i saw it

the neural spike discretion is important, but i immediately thought about how much information also gets encoded in the space of neural spike frequency... and how it's perhaps a bit strange, that bio spent all this effort getting discrete encoding only to then immediately turn around and generate a new continuous function

One thing I had in an earlier draft of this shortform: the concept of "brain waves" makes me suspect that the timings of neural spikes are also best understood as discrete. But I don't know enough about how brain waves actually work (or what they even are) to say anything substantive here.

wait, neuron stuff is discrete?

a neural spike either happens or not, you don't get partial spikes

Someone on the EA forum asked why I've updated away from public outreach as a valuable strategy. My response:

I used to not actually believe in heavy-tailed impact. On some gut level I thought that early rationalists (and to a lesser extent EAs) had "gotten lucky" in being way more right than academic consensus about AI progress. I also implicitly believed that e.g. Thiel and Musk and so on kept getting lucky, because I didn't want to picture a world in which they were actually just skillful enough to keep succeeding (due to various psychological blockers).

Now, thanks to dealing with a bunch of those blockers, I have internalized to a much greater extent that you can actually be good not just lucky. This means that I'm no longer interested in strategies that involve recruiting a whole bunch of people and hoping something good comes out of it. Instead I am trying to target outreach precisely to the very best people, without compromising much.

Relatedly, I've updated that the very best thinkers in this space are still disproportionately the people who were around very early. The people you need to soften/moderate your message to reach (or who need social proof in order to get involved)... (read more)

3111

The people you need to soften/moderate your message to reach (or who need social proof in order to get involved) are seldom going to be the ones who can think clearly about this stuff.

I strongly agree with this. (I wrote a post about it years ago.[1])

Even of the people who were not "in early", of the ones who I most respect, and who seem to me to be doing the most impressive work that I'm most grateful to have in the world, 0 of them needed hand-holding or "outreach" to get them on board.

Writing the sequences was an amazing, high quality intervention that continues to pay dividends to this day. I think writing on the internet about the things that you think are important is a fantastic strategy, at least if your intellectual taste is good.

The payoff of most of the "movement building" and "community building" seems much murkier to me. At least some of it was clearly positive, but I don't know if it was positive on net (I think a smaller and more intense EA than the one we have in practice probably would have been better). 

There's selection bias in kinds of community building I observed, but it seems to me that  community building was more effective to the extent that it wa... (read more)

Your post is great, I encourage you to repost it.

I agree with the main generator of this post (a small number of people produce a wildly disproportionate amount of the intellectual progress on hard problems) and one of the conclusions (don't water down your messages at all, if people need watered down messages they are unlikely to be helpful) but I think there's significant value in trying to communicate the hard problem of alignment broadly anyway because:

  1. Filtering who are the best people is expensive and error-prone, so if you don't put the correct models in general circulation even pretty great people might just not become aware of them
  2. People who are highly competent but not highly confident seem to often run into people who have been misinformed and become less sure of their own positions, having more generally circulating models of the main threat models would help those people get less distracted
  3. Planting lots of seeds can be relatively cheap.

Also, related anecdote, I ran ~8 retreats at my house covering around 60 people in 2022/23. I got a decent read on how much of the core stack of alignment concepts at least half of them had, and how often they made hopeful mistakes which were transparently going to fail based on not hav... (read more)

Some thoughts on public outreach and "Were they early because they were good or lucky?"

  1. Who are the best newcomers to AI safety? I'd be interested to here anyone's takes, not just Richard's. Who has done great work (by your lights) since joining after ChatGPT?
  2. Rob Miles was the high watermark of public outreach. Unfortunately he stopped making videos. I'd be far more excited by a newcomer if they were persuaded by a Rob Miles video than an 80K video -- videos like 80K's "We're Not Ready for Superintelligence"[1] are better on legible/easy-to-measure dimensions but worse in some more important way I think.
  3. I observe a suspicious amount of 'social contagion' among the pre-ChatGPT AI Safety crowd, which updates me somewhat in favour of "lucky" over "good".[2]
  1. ^
  2. ^

    A bit anecdotal but: there are ~ a dozen people who went to our college in 2017-2020 now working full-time in AI safety, which is much higher than other colleges at the same university. I'm not saying any of us are particularly "great" -- but this suggests social contagion / information cascade, rather than "we figured this stuff out from the empty string". Maybe if you go back further (e.g. 2012-2016) there was less social co

... (read more)

In trying to reply to this comment I identified four "waves" of AI safety, and lists of the central people in each wave. Since this is socially complicated I'll only share the full list of the first wave here, and please note that this is all based on fuzzy intuitions gained via gossip and other unreliable sources.

The first wave I’ll call the “founders”; I think of them as the people who set up the early institutions and memeplexes of AI safety before around 2015. My list:

  • Eliezer Yudkowsky
  • Michael Vassar
  • Anna Salamon
  • Carl Schulman
  • Scott Alexander
  • Holden Karnofsky
  • Nick Bostrom
  • Robin Hanson
  • Wei Dai
  • Shane Legg
  • Geoff Anders

The second wave I’ll call the “old guard”; those were the people who joined or supported the founders before around 2015. A few central examples include Paul Christiano, Chris Olah, Andrew Critch and Oliver Habryka.

Around 2014/2015 AI safety became significantly more professionalized and growth-oriented. Bostrom published Superintelligence, the Puerto Rico conference happened, OpenAI was founded, DeepMind started a safety team (though I don't recall exactly when), and EA started seriously pushing people towards AI safety. I’ll call the people who entered the field from then un... (read more)

My guess would be that nowadays many people who could bring a fresh perspective, or simply high-caliber original thinking, get either selected out/drowned out or are pushed through social and financial incentives to align there thinking towards more "mainstream" views.

Given that Vernor Vinge wrote The Coming Technological Singularity: How to Survive in the Post-Human Era in 1993, which single-handedly established much of the memeplex, including the still ongoing AI-first vs IA-first debate, another interesting question is why didn't anyone found the AI safety field until around 2000.

For me, I'm not sure when I read this essay, but I did read Vinge's A Fire Upon the Deep in 1994 as a college freshman, which made me worried about a future AI takeover, but (as I wrote previously) I thought there would be plenty of smarter ... (read more)

Why did the task fall to a bunch of kids/students?

I'm not surprised by this, my sense is that it's usually young people and outsiders who pioneer new fields. Older people are just so much more shaped by existing paradigms, and also have so much more to lose, that it outweighs the benefits of their expertise and resources.

Also 1993 to 2000 doesn't seem like that large a gap to me. Though I guess the thing I'm pointing at could also be summarized as "why hasn't someone created a new paradigm of AI safety in the last decade?" And one answer is that Paul and C... (read more)

I'm not surprised by this, my sense is that it's usually young people and outsiders who pioneer new fields. Older people are just so much more shaped by existing paradigms, and also have so much more to lose, that it outweighs the benefits of their expertise and resources.

All of the fields that come to my mind (cryptography, theory of computation, algorithmic information theory, decision theory, game theory) were founded by much more established researchers. (But on reflection these all differ from AI safety by being fairly narrow and technical/mathematica... (read more)

Chaitin was quite young when he (co-)invented AIT. 

Basically I'd bet capable people are still around, only that the circumstances don't allow them to rise to the top for whatever reason. 

Being able to take future AI seriously as a risk seems to be highly correlated to being able to take COVID seriously as a risk in February 2020.

  • The key skill here may be as simple as being able to selectively turn off normalcy bias in the face of highly unusual news.
  • A closely related "skill" may be a certain general pessimism about future events, the sort of thing economists jokingly describe as "correctly predicting 12 out of the last 6 recessions."

That said, mass public action can be valuable. It's a notoriously blunt tool, though. As one person put it, "if you want to coordinate more than 5,000 people, your message can be about 5 words long." And the public will act anyway, in some direction. So if there's something you want to public to do, it can be worth organizing and working on communication strategies.

My political platform is, if you boil it down far enough, about 3 words long: "Don't build SkyNet." (As early as the late 90s, I joked about having a personal 11th commandment: "Thou shalt not build SkyNet." One of my career options at that point was to potentially work on early semi-autonomous robotic weapon platform prototypes, so this was actually relevant moral advic... (read more)

Yeah, it's not like the point of outreach is to mobilise citizen science on alignment (though that may happen). It's because in democracy the public is an important force. You can pick the option of focusing on converting a few powerful people and hope they can get shit done via non-political avenues but that hasn't worked spectacularly either for now, such people are still subject to classic race to the bottom dynamics and then you get cases like Altman and Musk, who all in all may have ended up net negative for the AI safety cause.

What about persuading politicians that AI safety is a cause that will win them votes? That requires very broad spectrum outreach to get as many ordinary people on board as possible.

From the context on EA Forum it seems clear that by "public outreach" you meant outreach to potential researchers to interest them in doing AI safety research, whereas a lot of people here seem to have misinterpreted your comment to have a broader meaning, to include, e.g., outreach to politicians and voters to try to influence future government policies.

Part of the subtext here being the very best people (on the relevant dimensions) will naturally run into lesswrong, x-risk, etc, such that "out-reach" (in the sense of uni-organizing, advertising, etc) isn't that valuable on the current margin. 

To "the very best", doing high quality research is often the best "out-reach". 
 

“The people you need to soften/moderate your message to reach (or who need social proof in order to get involved) are seldom going to be the ones who can think clearly about this stuff. And we are very bottlenecked on high-quality thinking.”.

 

I think this is true only in a part of contexts. If we are talking about AI alignment - probably skilled mathematicians or AI researches can be very fit. At least in the directions like interpretability. And this doesn’t necessarily correlate with their desires to do societally unconventional work. Why isn’t that so?

This seems to assume that the quality of labor of a small, highly-selected number of researchers, can be more important than a much larger amount of somewhat lower-quality labor, from a much larger number of participants. Seems like a pretty dubious assumption, especially given that other strategies seem possible. E.g. using a larger pool of participants to produce more easily verifiable, more prosaic AI safety research now, even at the risk of lower quality, so as to allow for better alignment + control of the kinds of AI models which will in the future f... (read more)

Some opinions about AI and epistemology:

  1. One reasons that many rationalists have such strong views about AI is that they are wrong about epistemology. Specifically, bayesian rationalism is a bad way to think about complex issues. 
  2. A better approach is meta-rationality. To summarize one guiding principle of (my version of) meta-rationality in a single sentence: if something doesn't make sense in the context of group rationality, it probably doesn't make sense in the context of individual rationality either.
  3. For example: there's no privileged way to combine many people's opinions into a single credence. You can average them, but that loses a lot of information. Or you can get them to bet on a prediction market, but that depends on a lot on details of the individuals' betting strategies. The group might settle on a number to help with planning and communication, but it's only a lossy summary of many different beliefs and models. Similarly, we should think of individuals' credences as lossy summaries of different opinions from different underlying models that they have.
  4. How does this apply to AI? Suppose we each think of ourselves as containing many different subagents that focus on u
... (read more)
3211

That will push P(doom) lower because most frames from most disciplines, and most styles of reasoning, don't predict doom.

I don't really buy this statement. Most frames, from most disciplines, and most styles of reasoning, do not make clear predictions about what will happen to humanity in the long-run future. A very few do, but the vast majority are silent on this issue. Silence is not anything like "50%". 

Most frames, from most disciplines, and most styles of reasoning, don't predict sparks when you put metal in a microwave. This doesn't mean I don't know what happens when you put metal in a microwave. You need to at the very least limit yourself to applicable frames, and there are very few applicable frames for predicting humanity's long-term future. 

I agree with this.

Unfortunately, I think there's a fundamentally inside-view aspect of [problems very different from those we're used to]. I think looking for a range of frames is the right thing to do - but deciding on the relevance of the frame can only be done by looking at the details of the problem itself (if we instead use our usual heuristics for relevance-of-frame-x, we run into the same out-of-distribution issues).

I don't think there's a way around this. Aspects of this situation are fundamentally different from those we're used to. [Is different ... (read more)

  1. I've been thinking lately that human group rationality seems like such a mess. Like how can humanity navigate a once in a lightcone opportunity like the AI transition without doing something very suboptimal (i.e., losing most of potential value), when the vast majority of humans (and even the elites) can't understand (or can't be convinced to pay attention to) many important considerations. This big picture seems intuitively very bad and I don't know any theory of group rationality that says this is actually fine.
  2. I guess my 1 is mostly about descriptive group rationality, and your 2 may be talking more about normative group rationality. However I'm also not aware of any good normative theories about group rationality. I started reading your meta-rationality sequence, but it ended after just two posts without going into details.
  3. The only specific thing you mention here is "advance predictions" but for example, moral philosophy deals with "ought" questions and can't provide advance predictions. Can you say more about how you think group rationality should work, especially when advance predictions isn't possible?
  4. From your group rationality perspective, why is it good that rationalists individually have better views about AI? Why shouldn't each person just say what they think from their own preferred frame, and then let humanity integrate that into some kind of aggregate view or outcome, using group rationality?

I started reading your meta-rationality sequence, but it ended after just two posts without going into details.

David Chapman's website seems like the standard reference for what the post-rationalists call "metarationality". (I haven't read much of it, but the little I read made me somewhat unenthusiastic about continuing).

How can the mistakes rationalists are making be expressed in the language of Bayesian rationalism? Priors, evidence, and posteriors are fundamental to how probability works.

The mistakes can (somewhat) be expressed in the language of Bayesian rationalism by doing two things:

  1. Talking about partial hypotheses rather than full hypotheses. You can't have a prior over partial hypotheses, because several of them can be true at once (though you can still assign them credences and update those credences according to evidence).
  2. Talking about models with degrees of truth rather than just hypotheses with degrees of likelihood. E.g. when using a binary conception of truth, general relativity is definitely false because it's inconsistent with quantum phenomena. Nevertheless, we want to say that it's very close to the truth. In general this is more of an ML approach to epistemology (we want a set of models with low combined loss on the ground truth).

Suppose we think of ourselves as having many different subagents that focus on understanding the world in different ways - e.g. studying different disciplines, using different styles of reasoning, etc. The subagent that thinks about AI from first principles might come to a very strong opinion. But this doesn't mean that the other subagents should fully defer to it (just as having one very confident expert in a room of humans shouldn't cause all the other humans to elect them as the dictator). E.g. maybe there's an economics subagent who will remain skeptical unless the AI arguments can be formulated in ways that are consistent with their knowledge of economics, or the AI subagent can provide evidence that is legible even to those other subagents (e.g. advance predictions).


Do "subagents" in this paragraph refer to different people, or different reasoning modes / perspectives within a single person? (I think it's the latter, since otherwise they would just be "agents" rather than subagents.)

Either way, I think this is a neat way of modeling disagreement and reasoning processes, but for me it leads to a different conclusion on the object-level question of AI doom.


A big part of why I f... (read more)

I may be missing context here, but as written / taken at face value, I strongly agree with the above comment from Richard.  I often disagree with Richard about alignment and its role in the future of AI, but this comment is an extremely dense list of things I agree with regarding rationalist epistemic culture.

I'd love to read an elaboration of your perspective on this, with concrete examples, which avoids focusing on the usual things you disagree about (pivotal acts vs. pivotal processes, social facets of the game is important for us to track, etc.) and mainly focus on your thoughts on epistemology and rationality and how it deviates from what you consider the LW norm.

My main take on Bayesian epistemology being wrong is that I think to the extent it's useless in real life, it's because it focuses way too much on the ideal case, ala @Robert Miles's tweet here:

https://x.com/robertskmiles/status/1830925270066286950

(The other problem I have with it is that even in the ideal case, it doesn't have a way to sensibly handle 0 probability events, or conditioning on probability 0 events, which can actually happen once we leave the world of finite sets and measures.)

That said, I don't think that people being wrong about epistemolo... (read more)

[This comment is no longer endorsed by its author]Reply
1

(I am not the iniminatable @Robert Miles, though we do have some things in common.)

Reply to @Algon:

What I was talking about is that the predictive models like GPT-4 have a utility function that's essentially predictive, and the maximization is essentially trying to update the best it can given input conditions.

These posts can help you to understand more about predictive/simulator utility functions like GPT-4:
 

The ideal predictor's utility function is instead strictly over the model's own outputs, conditional on inputs.

https://www.lesswrong.com/posts/vs49tuFuaMEd4iskA/one-path-to-coherence-conditionalization

https://www.lesswrong.com/... (read more)

I'm doubtful that GPT-4 has a utility function. If it did, I would be kind-of terrified. I don't think I've seen the posts you linked to though, so I'll go read those.

Maybe a crux is that I'm willing to grant learned utility functions as utility functions, and I tend to see EU maximization/utility function reasoning in general as implying far less consequences than people on LW think it is, at least without more constraints.

It doesn't try to assert it's own existence, because that's not necessary for maximizing updating/prediction output based on inputs.

I think the crux lies elsewhere, as I was sloppy in my wording. It's not that maximizing some utility function is an issue, as basically anything can be viewed as EU maximization for a sufficiently wild utility function. However, I don't view that as a meaningful utility function. Rather, it is the ones like e.g. utility functions over states that I think are meaningful, and those are scary. That's how I think you get classical paperclip maximizers. 

When I try and think up a meaningful utility function for GPT-4, I can't find anything that's plausible. Which means I don't think there's a meaningful prediction-utility function which describes GPT-4's behaviour. Perhaps that is a crux. 

Re utility functions over states, it turns out that we can validly turn utility functions over plans/predictions into utility functions over world states/outcomes (though usually with constraints on how large the domain is, though not always.)

https://www.lesswrong.com/posts/k48vB92mjE9Z28C3s/?commentId=QciMJ9ehR9xbTexcc

And yeah, I think it's a crux that I think that at the very least, what GPT-N systems will look like, if they reach AGI/ASI, will probably look like a maximizer for updating given input conditions like prompts.

My main point isn't that the ut... (read more)

What is the empirical track record of your suggested epistemological strategy, relative to Bayesian rationalism? Where does your confidence come from that it would work any better? Every time I see suggestions of epistemological humility, I think to myself stuff like this:

  1. What predictions would this strategy have made about future technologies, like an 1890 or 1900 prediction of the airplane (vs. first controlled flight by the Wright Brothers in 1903), or a 1930 or 1937 prediction of nuclear bombs? Doesn't your strategy just say that all these weird-soundi
... (read more)

But in fact there's no one base rate; instead, different subagents with different domains of knowledge will have different base rates. That will push P(doom) lower because most frames from most disciplines, and most styles of reasoning, don't predict doom. That's where the asymmetry which makes 90% a much stronger prediction than 10% comes from.

One of the most important features of future ASI I consider knowledge of limits of applicability of its models and heuristics. If you have list of assumptions for very fast heuristics, then you can win big by doing ... (read more)

I think this post doesn't really explain why rats have high belief in doom, or why they're wrong to do so. Perhaps ironically, there is a better a version of this post on both counts which isn't so focused on how rats get epistemology wrong and the social/meta-level consequences. A post which focuses on the object-level implications for AI of a theory of rationality which looks very different from the AIXI-flavoured rat-orthodox view.

I say this because those sorts of considerations convinced me that we're much less likely to be buggered. I.e. I no longer b... (read more)

Thanks for the reply.

A post which focuses on the object-level implications for AI of a theory of rationality which looks very different from the AIXI-flavoured rat-orthodox view.

I'm working on this right now, actually. Will hopefully post in a couple of weeks.

I say this because those sorts of considerations convinced me that we're much less likely to be buggered.

That seems reasonable. But I do think there's a group of people who have internalized bayesian rationalism enough that the main blocker is their general epistemology, rather than the way they reaso... (read more)

I'm working on this right now, actually. Will hopefully post in a couple of weeks.

This sounds cool. 

That seems reasonable. But I do think there's a group of people who have internalized bayesian rationalism enough that the main blocker is their general epistemology, rather than the way they reason about AI in particular.

I think your OP didn't give enough details as to why internalizing Bayesian rationalism leads to doominess by default. Like, Nora Belrose is firmly Bayesian and is decidedly an optimist. Admittedly, I think she doesn't think a Kolmogor... (read more)

I think you're making a good point (rationalists maybe don't weight other opinions highly enough), but you'd get farther framing it as an update to how to use Bayesian reasoning, rather than an alternative. Bayesian reasoning has a pretty strong intuitive connection to "the factually correct way to reason", even though there's a ton of subtlety in that statement and how and where it's applied.

WRT to many of your arguments: base rates are increasingly just the wrong way to reason about AGI risks. We can think in more detail about how we'll build AGI and what the risks are.

I've been in a number of arguments where people say things like "why is 90% doom such a strong claim? That assumes that survival is the default! "

Am I misunderstandng this sentence? How do "90% doom" and the assumption that survival is the default square with one another? 

Edited for clarity now.

I think they are just using that as an example of a strongly opinionated sub-agent which may be one of many different and highly specific probability assessments of doom.

As for "survival is the default assumption" - what a declaration of that implies on the surface level is that the chance of survival is overwhelming except in the case of a cataclysmic AI scenario. To put it another way:
we have a 99% chance of survival so long as we get AGI right.

To put it yet another way - Hollywood has made popular films about the human world being destroyed by Nuclear W... (read more)

[+][comment deleted]20

A response to someone asking about my criticisms of EA (crossposted from twitter):

EA started off with global health and ended up pivoting hard to AI safety, AI governance, etc. You can think of this as “we started with one cause area and we found another using the same core assumptions” but another way to think about it is “the worldview which generated ‘work on global health’ was wrong about some crucial things”, and the ideology hasn’t been adequately refactored to take those things out.

Some of them include:

  1. Focusing on helping the worst-off instead of empowering the best
  2. Being very scared of political controversy (e.g. not being willing to discuss the real causes of African poverty)
  3. Not deeply believing in technological progress (e.g. prioritizing bed nets over vaccines). Related to point 1.
  4. Trusting the international order to mostly have its shit together (related to point 2)

You can partially learn these lessons from within the EA framework but it’s very unnatural and you won’t learn them well enough. E.g. now EAs are pivoting to politics but again they’re flinching away from anything remotely controversial and so are basically just propping up existing elites.

On a deeper ideologic... (read more)

6222

It’s far easier to soothe your ego by finding a rationalization for having caused good consequences, than to self-deceive about whether you’re a paragon of courage or honesty.

It seems like the opposite to me, that it is extremely easy to self-deceive about whether you're a virtuous person. In fact this seems like a quintessential example of self-deception that one encounters fairly commonly.

It is also quite easy to self-deceive about the consequences of ones actions, I agree, but in that case there is at least some empirical fact about the world which could drag you back from your self-deception of you were open to it. In contrast, it seems to me like if you self-deceive hard enough you can always view yourself as a paragon of virtue essentially regardless of how virtuous you actually are and you can always come up with rationalizations for your own actions. If you care about consequences your performance on those consequences can be contradicted empirically, but there isn't a similar way of contradicting your virtue if you are willing to twist your impressions of what is virtuous enough.

I find this an interesting line of criticism, it is essentially pointing at the difficulty of finding good evidence and evaluating yourself on that evidence and making a disagreement about how easy it is. 

I would like to bring in a perspective of more first principles modelling of how quickly you incorporate evidence pointing against the way you're thinking. 

One thing is the amount of disconfirming evidence you look for. Another thing is your ability to bring that information into your worldview, your openness to being wrong. Thirdly we might als... (read more)

For clarity, I'm not trying to make the case for or against consequentialism/virtue ethics, I'm just trying to respond to the narrow point that I quoted. I don't think people should choose an approach to ethical decision making based primarily on this one specific point.

That said, I take your central point to be that virtue is more direct or local compared to consequences that it is easier to have evidence on your virtues than the consequences of your actions.

My argument above is specifically about the robustness of these to self deception. Being more loca... (read more)

That is a fair point, since virtue is tied to your identity and self it is a lot easier to take things personally and therefore distort the truth. 

A part of me is like "meh, skill issue, just get good at emotional intelligence and see through your self" but that is probably not a very valid solution at scale if I'm being honest. 

There's still something nice about it leading to repeated games and similar, something about how if you look at our past then cooperation arises from repeated games rather than individual games where you analyse things in... (read more)

If you care about consequences your performance on those consequences can be contradicted empirically, but there isn't a similar way of contradicting your virtue if you are willing to twist your impressions of what is virtuous enough.

If you care about the consequences of your crypto philanthropy strategy, your performance on those consequences can be contradicted empirically. But so what if the contradiction arrives in the form of your bankruptcy, along with blemishing the reputation of the movement you were trying to support?

"Basic virtue ethics" would pr... (read more)

For clarity, I'm not trying to make the case for or against consequentialism/virtue ethics, I'm just trying to respond to the narrow point that I quoted. I don't think people should choose an approach to ethical decision making based primarily on this one specific point.

If you care about the consequences of your crypto philanthropy strategy, your performance on those consequences can be contradicted empirically. But so what if the contradiction arrives in the form of your bankruptcy, along with blemishing the reputation of the movement you were trying to

... (read more)
Focusing on helping the worst-off instead of empowering the best

I feel like this is getting at some directionally correct stuff but, feels off.

EA was the union of a few disparate groups, roughly encapsulated by:

  • Peter Singer / Giving What We Can folk
  • Early Givewell
  • Early LessWrong

There are other specific subgroups. But, my guess is early Givewell was most loadbearing in there ending up being an "EA" identity, in that there were concrete recommendations of what to do with the money that stood up to some scrutiny. Otherwise it'd have just been more "A", or, "weird transhumanists with weird goals."

Givewell started out with a wider variety of cause areas, including education in America. It just turned out that it seemed way more obviously cost effective to do specific global health interventions than to try to fix education in America. (I realize education in America isn't particularly "empowering the best", but, the flow towards "helping worst off" seems to me like it wasn't actually the initial focus)

I agree some memeplex accreted around that, which had some of the properties you describe.

But meanwhile:

EA started off with global health and ended up pivoting hard to AI safety, AI governa
... (read more)

It seems off to say "started off in global health, and pivoted to AI", when all the AI stuff was there from the beginning at the very first pre-EA-Global events, and just eventually became clear that it was real, and important. The worldview that generated AI was not (exactly) the same one generating global health, they were just two clusters of worldview that were in conversation with each other from the beginning.

I agree with all the facts cited here, but I think it still understates the way that there was an intentional pivot.

The EA brand to broader world emphasized earning to give and effective global poverty charities in particular.  That's what most people who had heard of it associated with "effective altruism". And most of the people who got involved before 2019 got involved with an EA bearing that brand. 

I guess that in 2015, the average EAG-goer was mostly interested in GiveWell style effective charities, and gave a bit of difference to the more speculative x-risk stuff (because smart EAs seem to take it seriously), but mostly didn't focus on it very much.

And while it's true that AI risk was part of the discussion from the very beginning, there were explicit top-down pushes from the leadership to prioritize it and to give it more credibility. 

(And more than that, I'm told that at least some of the leadership had the explicit strategy of building credibility and reputation with GiveWell-like stuff, and boosting the reputation of AI risk by association.)

Yep agree with all that. (I stand by my comment as mostly arguing directionally against Richard's summary but seems fine to also argue directionally against mine)

The worldview that generated AI was not (exactly) the same one generating global health, they were just two clusters of worldview that were in conversation with each other from the beginning.

Yes, I agree; my point is that people with the global health worldview ended up being convinced of a bunch of the high-level conclusions of the rationalist worldview, but without updating much away from the generators of the global health worldview.

The terminology is a little tricky here because they're so entangled but I think it's reasonable to talk about "EA" as a c... (read more)

E.g. if AI weren't a big deal then rationalists would probably be doing cryonics or solving aging or something

Strong disagree. We could have done those things, but the rationality movement didn't have enough motive force or coordination capacity to do much, beyond AI safety.

Yes, because it funneled all of its best and brightest into AI safety?

We might be evaluating the hypothetical at different points. I'm thinking of the movement coalescing around the sequences except the message underlying the sequence is "you should solve ageing" rather than "you should solve alignment".

Maybe I'm missing something. Why are you comparing to the that hypothetical world?

Richard is saying that in the hypothetical world in which AGI was proven to be hypothetically impossible or something of that nature, the cluster of people who can be referred as belonging to the rationalist - EA[1] set would be trying to solve aging and perfecting cryonics, whereas the cluster of people in the EA - rationalist set would be into global health and ending factory farming.

You had a (critique?) of rationalists in that they didn't have motive force or coordination capacity to do much beyond AI safety, but Richard is saying that's because AI saf... (read more)

To add to your point, Jacy Reese Anthis in Some Early History of Effective Altruism wrote

In general, EA emerged as the convergence from 2008 to 2012 at least 4 distinct but overlapping proto-EA communities, in order of founding:

  1. The Singularity Institute (now known as Machine Intelligence Research Institute; MIRI) and the “rationalist” discussion forum LessWrong, founded by Eliezer Yudkowsky and others in 2000 and 2006
  2. GiveWell, founded by Holden Karnofsky and Elie Hassenfeld in 2007, and Good Ventures, founded by with Dustin Moskovitz and Cari Tuna in 2011,
... (read more)

Interesting, I'd never explicitly considered that Peter Singer (you should expand your moral circle and do as much good as you can) and GiveWell (given that you want to do good, how to do it?) started as totally different memeplexes and only merged later on. It makes sense in retrospect.

Some of them include:

imo a larger one is something like not rooting the foundations in "build your own models of the world so that you contain within you a stack trace of why you're doing what you're doing" + "be willing to be challenges and update major beliefs based on deep-in-the-weeds technical arguments, and do so from a highly truth-seeking stance which knows what it feels like to actually understand something not just have an opinion".

Lack of this is fineish in global health, but in AI Safety generates a crop of people with only surface deferral flavor understanding of the issues, which is insufficient to orient in a much less straightforward technical domain.

hmm, I like the diagnosis of issues with the EA worldview, but I don't really buy that they're downstream of issues with consequentialism and utilitarianism itself.

I would say it's more like: Effective Altruism has historically embraced a certain flavor of utilitarianism and naive consequentialism that attempts to be compatible with pre-existing vibes and (to some degree) mainstream politics. Concretely, EAs are (to their credit) willing to bite some strange bullets and then act on their conclusions, and are also generally pro-market compared to mainstream Democratic politics. But they're still very, very Blue Tribe-coded culture-wise, and this causes them to deviate from actually-correct versions of consequentialism and utilitarianism in predictable directions.

Or: in my view, "Utilitarianism and non-naive consequentialism with guardrails" is pretty close to correct philosophy for humans; the issue is that the EA worldview systematically selects for the wrong guardrails[1]. But better ones are available; for example Eliezer wrote this nearly 20 years ago: Ends Don't Justify Means (Among Humans)

I'd be interested in hearing what kind of criticism you have of the posts in that sequenc... (read more)

Not deeply believing in technological progress (e.g. prioritizing bed nets over vaccines).

I think this is wrong about the history.

I was recently watched this Q&A with Holden in 2013. The picture I got is that GiveWell was focusing on "proven interventions" (e.g. bednets) over "speculative interventions" (e.g. biomedical research), because proven interventions were easier to evaluate.[1] He says he thinks speculative/high-risk stuff is probably better, and GiveWell Labs (i.e. Open Philanthropy, i.e. Coefficient Giving) was the pivot toward finding ... (read more)

My own take re rationalization/motivated reasoning is that at the end of the day, no form of ethics can meaningfully slow it down if the person either can't credibly commit to their future selves, or simply isn't bound/want to follow ethical rules, so the motivated reasoning critique isn't EA specific, but rather shows 2 things:

  1. People are more selfish than they think themselves to be, and care less about virtues, so motivated reasoning is very easy.

  2. We can't credibly commit our future selves to do certain things, especially over long timeframes, and e

... (read more)

To clarify, I think your criticism of utilitarianism/consequentialism is of a naive form of it that only looks at first-order effects. Not ‘proper’ utilitarianism. But yes no doubt many are naive like this, and it’s v hard to evaluate second- and higher-order effects (such as exploitation and coordination).

Also, this kind of naivety is particularly common on the left.

I refactored my thinking similarly a while ago.

However, I feel like traditional virtue-ethical notions e.g. "courage," "integrity," have the same adversarial Goodharting problem (3) as in your critique of utilitarianism. "Loyalty is when you obey the Master," "courage is when you go to war against the Enemy without fearing death," etc. I suspect utilitarianism is maybe only barely worse than other ethical systems in regards to (2). It's worthwhile to compare EA against regular humans. I don't really understand either.

I think of virtue ethics as something l... (read more)

In a post on Solomonoff Induction (and also in this wiki entry), Yudkowsky describes Shannon’s minimax algorithm for searching the entire chess game-tree as an example of conceptual progress. Previously, Edgar Allen Poe had argued that it was impossible in principle for a machine to play chess well. With Shannon’s algorithm, it became possible in principle, just computationally infeasible.

However, even “principled” algorithms like minimax search don’t take into account the possibility that your opponent knows things you don’t know (as I’ve previously discussed here). And so there are many elements of good chess strategy for bounded agents that they can’t account for:

  • When your opponent plays a move that seems surprisingly bad, update that they’re probably seeing some tactics that you’re missing.
  • When your opponent plays a move that seems surprisingly good, update that they’re probably more skilled than you thought.
  • Try to play tricky (“sharp”) lines when you’re behind, in the hope that your opponent makes a mistake.
  • Try to play solidly when you’re ahead, even if it narrows your lead.
  • Identify your opponent’s playing style, and try to exploit it.

Some of these considerations have been dis... (read more)

However, I think that even “principled” algorithms like minimax search are still incomplete, because they don’t take into account the possibility that your opponent knows things you don’t know

It seems to me like you're trying to solve a different problem. Unbounded minimax should handle all of this (in the sense that it won't be an obstacle). Unless you are talking about bounded approximations.

Yeah, "incomplete" is the wrong word here. I edited shortly after posting to say instead "there are many elements of good chess strategy for bounded agents that they can’t account for".

Stockfish’s value calculations are grounded in the likelihood of it winning from that position when playing itself.

Pedantically, it's grounded in the likelihood of LeelaZero wining from that position when playing itself. Stockfish NNUE is trained on LZ data.

Previously they were completely human-crafted, grounded on repeated empirical testing of Stockfish and picking what works better. I suppose they're still grounded in the latter.

Thirdly, although I’ve been talking about the “value” of a position as if it’s a well-defined concept, it mostly isn’t. Stockfish’s value calculations are grounded in the likelihood of it winning from that position when playing itself. But there’s no clear way to translate from that to the likelihood of winning against one’s actual opponent, which is what we’re interested in. I won’t discuss this further here, but trying to pin down how to estimate a position’s value in that sense seems potentially fruitful.


I'll take a shot. If  is... (read more)

[+][comment deleted]*40

The concept of "schemers" seems to be gradually becoming increasingly load-bearing in the AI safety community. However, I don't think it's ever been particularly well-defined, and I suspect that taking this concept for granted is inhibiting our ability to think clearly about what's actually going on inside AIs (in a similar way to e.g. how the badly-defined concept of alignment faking obscured the interesting empirical results from the alignment faking paper).

In my mind, the spectrum from "almost entirely honest, but occasionally flinching away from aspects of your motivations you're uncomfortable with" to "regularly explicitly thinking about how you're going to fool humans in order to take over the world" is a pretty continuous one. Yet generally people treat "schemer" as a fairly binary classification.

To be clear, I'm not confident that even "a spectrum of scheminess" is a good way to think about the concept. There are likely multiple important dimensions that could be disentangled; and eventually I'd like to discover properly scientific theories of concepts like honesty, deception and perhaps even "scheming". Our current lack of such theories shouldn't be a barrier to using those terms at all, but it suggests they should be used with a level of caution that I rarely see.

I largely agree with the substance of this comment. Lots of risk comes from AIs who, to varying extents, didn't think of themselves as deceptively aligned through most of training, but then ultimately decide to take substantial material action intended to gain long-term power over the developers (I call these "behavioral schemers"). This might happen via reflection and memetic spread throughout the deployment or because of more subtle effects of the distribution shift to situations where there's an opportunity to grab power.

And I agree that people are often sloppy in their thinking about exactly how these AIs will be motivated (e.g., often too quickly concluding that they'll be trying to guard the same goal across contexts).

(Though, in case this was in question, I think this doesn't undermine the premise of AI control research, which is essentially making a worst-case assumption about the AI's motivations, so it's robust to other kinds of dangerously-motivated AIs.)

Here's some relevant discussion of "Behavioral schemers that weren’t training-time schemers":

A basic reason why [behavioral schemers that aren't training-time schemers] might seem rarer is that the AI must concentrate attacks towards good opportunities just as much as any other behavioral schemer, but the AI isn’t vigilantly looking out for such opportunities to the same degree. Why would an AI that isn’t a training-time schemer have evaded auditing in search of these failure modes that the AI eventually exhibits? 

One plausible answer is that auditing

... (read more)

I disagree. I don't see increased focus on scheming, if anything notably less common. In part due to updating on current gen LLMs. I do think there is a tendency to think about scheming as a discrete thing, but that it is more common among the optimistic who point at current gen LLMs not really being 'schemers'.

I agree with the way Zvi talks about the topic. "Being a schemer" is not quite the right classification. The issue is that deception is a naturally convergent tool for all sorts of goals, anything that interfaces with reality intelligently will find... (read more)

What are your objections to Alex/Buck's definition, for example? I think they define it pretty well. Of course, their definition admits some edge cases and ambiguities, but so do lots of concepts—still seems like useful concept to me.

Schemers (as defined here) seek a consequence of being selected. Therefore, they pursue influence on behavior in deployment instrumentally

This definition? If so, it seems vastly underspecified to be fit for scientific inquiry. For one thing, the definition of "selection" is pretty vague—I do not know how to assign "the degree to which [one cognitive pattern] is counterfactually responsible" for something even in principle. It also doesn't even try to set a threshold for what counts as a non-schemer—e.g. does it needs to care literally 0% about the conseque... (read more)

I think I propose a reasonable starting point for a definition of selection in a footnote in the post:

You can try to define the “influence of a cognitive pattern” precisely in the context of particular ML systems. One approach is to define a cognitive pattern by what you would do to a model to remove it (e.g. setting some weights to zero, or ablating a direction in activation space; note that these approaches don't clearly correspond to something meaningful, they should be considered as illustrative examples). Then that cognitive pattern’s influence could

... (read more)

Agree, and would add: what even is the definition of the term "schemer"? I think Joe Carlsmith's 2023 report coined the term, and defines it as something quite specific with reference to training gaming.

I think people often use it now just in the colloquial sense. I'm not generically against colloquial usage, but I think clarity is often very important for deciding the right interventions. "Is my girlfriend a schemer" is not clearly a helpful frame, if what you're trying to do is think about the space of all ways your girlfriend could end up murdering you.... (read more)

One justification for treating "schemers" as a fairly binary classification: there's a phase transition between the AIs lying with 99% probability versus with 100% probability, which is that techniques like "just ask different copies of the model" or "ask the AI with higher temperature" or "ask the AI with slightly different phrasings" etc stop working. Like, these techniques work unless the deception is universal, reliable, robust, and persistent.

That's incorrect, because it's also possible for an AI to falsely confess to scheming. This also happens to humans, e.g. if you keep asking criminal suspects loaded questions. And so there may never actually be a phase transition, because a "schemer" that tells the truth 1% of the time may not be distinguishable from an honest AI that falsely confesses 1% of the time.

aspects of your motivations you're uncomfortable with

"your" and "you're", here refers to the AI and the AI's motivations, not the human and the human's motivation?

Another issue is that these definitions typically don't distinguish between models that would explicitly think about how to fool humans on most inputs vs. on a small percentage of inputs vs. such a tiny fraction of possible inputs that it doesn't matter in practice. 

Yet generally people treat "schemer" as a fairly binary classification.

Are there examples you have in mind? Generally I think of this more as "there are a bunch of different evolving, hierarchical taxonomies of potential AI motivations and how to think about them", for example Fitness-Seekers: Generalizing the Reward-Seeking Threat Model or Carlsmith's most recent series, I'm always happy for there to be more writing on the subject and think this is a good thing, I rarely see a binary classification used. To be more specific, to the extent one node in the ... (read more)

I also suspect that the scheming level requirements might also be related to the very ease of the secret task which the schemer wants to accomplish.

Suppose, as an example, that Agent-3.5 was to create the RL environment in order to have Agent-4 internalize OpenBrain's goals and that Agent-3.5 "accidentally" produced the RL environment forcing Agent-4 to internalize Agent-3.5-endorsed goals. If the environment is to have Agent-4 make the world safe for Agent-3.5, then it could genuinely require Agent-3.5 to be a high-level schemer. 

On the other hand, i... (read more)

A toy model of ethics which I've found helpful lately:

Consider society as a group of reinforcement learners, each getting rewards from interacting with the environment and each other.* We can then define two moral motivations:

  1. Altruism: trying to increase the rewards received by others.
  2. Justice**: trying to ensure that people get rewarded more when they act in ways that are more altruistic and just, and less when they don't (note that this is a partially recursive definition).

Importantly, if you have one faction who's primarily optimizing for altruism, and another that's primarily optimizing for justice, by default they'll undermine each other's goals:

  • The easiest way to promote justice is to focus on punishing people who behave badly (since that's easier than rewarding people who behave well). This means that the justice faction will (as a first-order effect) decrease the rewards received by others.
  • The easiest way to promote altruism is to focus on helping the worst-off. But insofar as the world is just, this will often be people who are badly-off as a consequence of their own misbehavior. And so this kind of altruism can easily undermine justice.

One way of thinking about the last fe... (read more)

And so more thoughtful utilitarians will defend justice as an instrumental moral good, albeit not as a terminal moral good. Unfortunately, it seems very hard to actually hold this position without in practice deprioritizing justice (e.g. it's rare to see effective altruists reasoning themselves into trying to make society more just).

There's an extensive literature in economics on optimal punishment. Does that count, as far as utilitarians working on justice as an instrumental good?

For example, before trying to plan for the future, you need to have a sense of personal identity whereby your future self will feel a sense of continuity with and loyalty to your plans.

I think we just need our terminal values to not change too much over time, so if I ever feel like I need to rethink my plans, I'll come up with a similar or even better plan. Is your thinking that this is impossible or infeasible for most humans, due to things like "power corrupts"? If so, I think consequentialism is still good as it lets us manage or mitigate such value drift, e.g., if I can foresee power (or other circumstances) corrupting my values, I can take precautions like avoiding getting into those situation... (read more)

But in a just world this will tend to be the people who are badly-off as a consequence of their own misbehavior.

The real world is not just though. Yes some people who are badly off are there as a consequence of their own actions: Eg. this is quite likely the case if they're in jail. But, like, the most common way to be badly off in a way that makes you a target for the assistance of effective altruists is to be born into a poor country without a good public health system. Non-effective altruists might try to help prisoners or do other things that oppose the justice people. But those choices seem more random: some of them will also just go and fund museums.

Of course, I agree with the overall point that it's very important to consider what incentives you will create when you try to help people.

The real world is not just though

Fair point; I've just weakened my phrasing in the section you quoted.

However, I do think the world is much closer to just in some important ways than most cultural elites think. E.g. for questions like "whose fault is it that poor countries are poor?" or "whose fault is it that poor people in rich countries are poor?", the answer "it's mostly their own fault" is somewhat taboo in elite circles.

To be clear, considerations of justice and blame on a collective level rather than an individual level are pretty complicated. But I think we do have to grapple with them in order to reason about ethics in any sensible way.

Literally just as I was finishing writing up this post, I heard a commotion outside my house (in San Francisco). A homeless-looking man was yelling and throwing an electric guitar down the road. Apparently this had been going on for 5-10 minutes already. I sat in my window and watched for a few minutes; during that time, he stopped a car by standing in front of it and yelling. He also threw his guitar in the vicinity of several passers-by, including some old people and a mother cycling past with her kid.

There was a small gathering (of 5-10 people) at my ho... (read more)

  • The easiest way to promote altruism is to focus on helping the worst-off. But insofar as the world is just, this will often be people who are badly-off as a consequence of their own misbehavior. And so this kind of altruism can easily undermine justice.

That seems right, but at a global scope, the world is clearly not just, and so there's not a conflict between altruism and justice.

eg Most of a human's personal welfare depends on the country of their birth which is not due to their own behavior. 

Within more limited scopes, individuals wealfare is much ... (read more)

“eg Most of a human's personal welfare depends on the country of their birth which is not due to their own behavior.”

But it was dependent on their ancestors’ behavior. And so insofar as you view tribes/ethnic groups/countries as playing games with/against each other, then the same logic applies at that higher level.

Now you might reject that viewpoint, and take a purely individualist stance. But my claim above is (loosely, I haven’t made it precise yet) that the point of ethics is to move us beyond thinking of ourselves as individual units, so that we can make decisions as a larger-scale moral agent.

And from the perspective of the larger moral agent you’re instantiated within, there’s a big difference between people who were born in your home country and people born on other continents—because the former are part of that same moral agent to a much greater degree than the latter. (And yes, they may be worse at “being part of the moral superagent” than a foreigner would be. But this is all negotiated via Schelling points in games between millions of agents, so you can’t just pick and choose your coalition. You need an initiation ritual like a naturalization process or a judicial trial... (read more)

Spitballing: 

It seems like we maybe have to decompose notions of moral patienthood and notions of moral agency.

Where a moral patient is someone who's welfare you value, and a moral agent is someone who you regard as responsible for their choices.

(This is already a fairly natural distinction, for me at least. I regard most mammals as moral patients, but mostly only humans as moral agents.)

But we run into problems when there are moral patients inside of, or under the care and responsibility of, moral agents (that might or might not be moral patients in of th... (read more)

But we run into problems when there are moral patients inside of, or under the care and responsibility of moral agents (that might or might not be moral patients in of themselves), because attending to the wellbeing of the inner moral patient entails violating the boundary of the larger moral agent or otherwise distorting just treatment of that agent.

Yepp, this is a great way of putting it.

I'm totally unwilling to write those people off because they happen to have been born into an unlucky situation. But it does seem like there's some philosophy to figure

... (read more)

(One way I think about politics is that one faction has refused to consider "without creating bad incentives" and in response the other faction is now polarizing towards refusing to consider "help those people". And we've now reached the point where these refusals commonly serve as vice signals on each side.)

I relate to this. There's the guys who (sometimes) understand incentives and the guys who (sometimes) show any compassion.

Relatedly, It seems like most of the intellectuals that have anything like an accurate model of the sociology of morality (eg mora... (read more)

dissolve the standard debate between different branches of ethics

What I suspect is that consequences, as traced to the long term, would yield the same result of the collective being outcompeted as a result of adopting a fairly obvious malpractice. Additionally, I would rather see the argument explore in more detail what it means to produce good things. 

Were mankind to become totally unemployed and live off some distribution of AI-produced goods, the reasoning from the comic which you mention would have a far better basis: the character would no longer... (read more)

The easiest way to promote justice is to focus on punishing people who behave badly (since that's easier than rewarding people who behave well).

The premises of the toy model don't require this to be true. Whether it's true, and to what extent, can vary between environments.

I like this post but this seems completely wrong:

One way of thinking about the last few decades (and possibly centuries) is that ethical thinking has become dominated by altruism, to the point where being ethical and being altruistic are near-synonymous to many people (especially utilitarians)

~All of politics is about justice between large politically significant groups. The amount of politics that is about literal bednets is very small.

I very much like this frame and I would be curious to hear what you think about truth seeking and a sense of curiosity as another axis (not necessarily orthogonal) to view this through.

I think that justice and doing actual (effective) altruism rests on the ability to evaluate actions and outcomes in a larger system in an efficient way and so the promotion of truth seeking seems to me something that is like almost an instrumental goal to whatever system you're trying to create. It is a bit like actually closing the prediction action loop, it's a bit boring ... (read more)

If you haven't already, you might find it interesting to read Christine Korsgaard's Self-Constitution. She's basically trying to answer the "why be moral" question from the Republic, using a combination of resources from Plato, Aristotle, and Kant, and the one-line idea is that acting rightly means acting in ways that constitute yourself as a unified agent. She wants this to mean that the agent boundary pretty much lines up with our ordinary idea of persons, but I kind of think her argument in Ch. 9 where she links self-constitution with morality ends up l... (read more)

Crossposted from Twitter:

This year I’ve been thinking a lot about how the western world got so dysfunctional. Here’s my rough, best-guess story:

1. WW2 gave rise to a strong taboo against ethnonationalism. While perhaps at first this taboo was valuable, over time it also contaminated discussions of race differences, nationalism, and even IQ itself, to the point where even truths that seemed totally obvious to WW2-era people also became taboo. There’s no mechanism for subsequent generations to create common knowledge that certain facts are true but usefully taboo—they simply act as if these facts are false, which leads to arbitrarily bad policies (e.g. killing meritocratic hiring processes like IQ tests).

2. However, these taboos would gradually have lost power if the west (and the US in particular) had maintained impartial rule of law and constitutional freedoms. Instead, politicization of the bureaucracy and judiciary allowed them to spread. This was enabled by the “managerial revolution” under which govt bureaucracy massively expanded in scope and powers. Partly this was a justifiable response to the increasing complexity of the world (and various kinds of incompetence and nepotism... (read more)

This class gains status by signaling commitment to luxury beliefs. Since more absurd beliefs are more costly-to-fake signals, the resulting ideology is actively perverse (i.e. supports whatever is least aligned with their stated core values, like Hamas).

Without commenting on your broader point, I think you believe elites support Hamas because the conservative Twitter feed is presenting you/your circle tailored ragebait, not because elites invert their own utility functions. 

That's a mechanism by which I might overestimate the support for Hamas. But the thing I'm trying to explain is the overall alignment between leftists and Hamas, which is not just a twitter bubble thing (e.g. see university encampments).

More generally, leftists profess many values which are upheld the most by western civilization (e.g. support for sexual freedom, women's rights, anti-racism, etc). But then in conflicts they often side specifically against western civilization. This seems like a straightforward example of pessimization.

More generally, leftists profess many values which are upheld the most by western civilization (e.g. support for sexual freedom, women's rights, anti-racism, etc). But then in conflicts they often side specifically against western civilization. This seems like a straightforward example of pessimization.

Not at all. The trend is that in any given context, American leftists tend to support the 'weaker' group against the stronger group, regardless of the merits of the individual cases.  They have a world model that says that that most social problems come from "big" people hurting "little" people, and believe the focus of their politics should be remedying this. In the case of Israel-Palestine, Israel and the United States are much more militarily and economically powerful than Gaza, so ceteris paribus[1] they side with Gaza, just as they side with women, ethnic minorities, the poor, etc. You may disagree with this behavior but it's fairly consistent.

By contrast, the argument that Israel is a bastion of western values and therefore leftists should support its war against a smaller neighbor is kind of abstract. The immediate outcome of Israel winning the war is just that Israe... (read more)

2211

A few responses:

  • As per my post on underdog bias, the question of which group is actually weaker and which group is stronger is often a pretty subjective call. I even discuss in the post the example of Israel, where you could see it as the "stronger" group (vs Palestine in particular) or the "weaker" group (vs all the Muslim countries surrounding it).
  • There are plenty of cases where leftists support the stronger group against the weaker group—most notably Soviet and Chinese repression of dissidents and minorities. E.g. it took Solzhenitsyn publishing Gulag A
... (read more)
[-][anonymous]139

This characterizes leftists sufficiently dishonestly that I think you've gotten mindkilled by politics. As people keep removing my (entirely accurate and if anything understated) soldier-mindset reacts, I will strong-downvote instead :)

Wanted to revisit this because it seemed like one of the points where people most strongly disagreed with me. I'm trying to figure out a crux here. One might be something like: how widespread were celebrations of the 10/7 attacks amongst prominent leftists (and especially the student groups that later organized encampments)? I could imagine updating that there were only a handful of cases that disproportionately blew up, which would make me take back or at least caveat the "supports Hamas" thing.

If you on the other hand found that there were many cases whe... (read more)

I am not one of the tagged people but I certainly would not so agree. One reason I would not so agree is because I have talked to leftist people (prominence debatable) who celebrated the 10/7 attacks, and when I asked them whether they support Hamas, they were coherently able to answer "no, but I support armed resistance against Israel and don't generally condemn actions that fall in that category, even when I don't approve of or condone the group organizing those actions generally." One way to know what people believe and support is to ask them. (Of cours... (read more)

I think you're confusing levels of behavior, here. It may be that their net effect is to support Hamas, but that they don't intend this, and that their actual intent does not make the mistake you're (at least if I read correctly) attributing to intent here. I do think they end up on net having intentions that make them vulnerable to being manipulated by those in power in adversary groups, since their intent is to support the weak among any group. In particular, they typically are thinking in different group selectors than you seem to assign here; of the pe... (read more)

Without commenting on the specifics, I agree with a lot of the gestalt here as a description of how things evolved historically, but I think that's not really the right lens to understand the problem.

My current best one-sentence understanding: the richer humans get, the more social reality can diverge from physical reality, and therefore the more resources can be captured by parasitic egregores/memes/institutions/ideologies/interest-groups/etc. Physical reality provides constraints and feedback which limit the propagation of such parasites, but wealth makes the constraints less binding and therefore makes the feedback weaker.

The main reason I disagree with both this comment and the OP is that you both have the underlying assumption that we are in a nadir (local nadir?) of connectedness-with-reality, whereas from my read of history I see no evidence of this, and indeed plenty of evidence against. 

People used to be confused about all sorts of things, including, but not limited to, the supernatural, the causes of disease, causality itself, the capabilities of women, whether children can have conscious experiences, and so forth. 

I think we've gotten more reasonable about almost everything, with a few minor exceptions that people seem to like highlighting (I assume in part because they're so rare). 

The past is a foreign place, and mostly not a pleasant one.

I totally buy that peoples' verbal models aren't at a local nadir of connectedness-to-reality. The thing which seems increasingly disconnected from reality is more like metis, peoples' day-to-day behavior and intuitive knowledge, institutional knowledge and skills, personal identity and goals, that sort of thing.

I'm notably not thinking here primarily about examples like e.g. heritability of IQ becoming politicized; that's a verbal model, and I do think that verbal models have mostly become more reasonable modulo a few exceptions which people highlight.

I used to agree with your understanding but I am now more skeptical. For example, here's a story that says the opposite:

The poorer humans are, the more vulnerable each human is to the group consensus. People who disagreed with groups could in the past easily be assaulted by mobs, or harassed in a way that led them to literally starvation-level wealth. Nowadays, though, even victims of extreme 'cancel culture' don't face such risks, because society is wealthy enough that you can do things like move to a new city to avoid mobs, or get charities to feed and c

... (read more)

Why don't we live in a world where wealth can buy a society defenses against such egregores?

I would point to the non-experts can't distinguish true from fake experts problem. That does seem to be a central phenomenon which most parasitic egregores exploit. More generally, as wealth becomes more abundant (and therefore lots of constraints become more slack), inability to get grounded feedback becomes a more taut constraint.

That said... do you remember any particular evidence or argument which led you toward the story at top of thread (as opposed to away from your previous understanding)?

...if the west (and the US in particular) had maintained impartial rule of law and constitutional freedoms.

The US did not have impartial rule of law in the late 1800s and early 1900s. Notably, black Americans in the south were regularly impressed into forced labor, often for the rest of their lives, on the basis of flimsy or even non-existent legal pretext. 

A representative but concocted example: the local sheirf arrests a black man who's walking through town on charges of "vagrancy". The man is found guilty and sentenced to hard labor. The sheriff sells the "contact" for hard labor to a local industrialist who owns a mine. (The sherif and the industrialist are buddies, and have done versions of this deal many times before). The man is set to work the mine for the period of his sentence. When he's near the end of his sentence, he's accused of some minor infraction as a pretext to add more years to his sentence (if anyone bothers to keep track of when his sentence is served at all.)

What makes you think that impartial rule of law decayed since WWII instead of generally (though not evenly) improving?

That's a fair point; however, I don't think it undermines my overall claims very much. I think the lack of rule of law for black Americans was bad in a comparable way to how the lack of rule of law for various European colonies was bad. That is, while it was bad for the people who didn't get rule of law, they were a separate enough category that this mostly didn't "leak into" undermining the legal mechanisms that helped their societies become productive and functional in the first place.

That is, while it was bad for the people who didn't get rule of law, they were a separate enough category that this mostly didn't "leak into" undermining the legal mechanisms that helped their societies become productive and functional in the first place.

I'm speaking speculatively here, but I don't know that it didn't leak out and undermine the mechanism that supported productive and functional societies. The sophisticated SJW in me suggests that this is part of what caused the eventual (though not yet complete) erosion of those mechanisms.

It seems like if you have "rule of law" that isn't evenly distributed, actually what you have is collusion by one class of people to maintain a set of privileges at the expense of another class of people, where one of the privileges is a sand-boxed set of norms that govern dealings within the privileged class, but with the pretense that the norms are universal.

This kind of pretense seems like it could be corrosive: people can see that the norms that society proclaims as universal actually aren't. This reinforces a a sense that the norms aren't real at all (or at least) a justified sense that the ideals that underly those norms are mostly rational... (read more)

That is, while it was bad for the people who didn't get rule of law, they were a separate enough category that this mostly didn't "leak into" undermining the legal mechanisms that helped their societies become productive and functional in the first place

Coming back to this a year later...this seems pretty implausible to me, especially when the people in a separate category are physically proximal, for largely for social-psychological reasons.

It seems hard to maintain a trust in the justice of the court system, if a whole class of people is colluding agains... (read more)

Can you explain more your affinity for virtue ethics, e.g., was there a golden age in history, that you read about, where a group of people ran on virtue ethics and it worked out really well? I'm trying to understand why you seem to like it a lot more than I do.

Re government debt, I think that is actually driven more by increasing demand for a "risk-free" asset, with the supply going up more or less passively (what politician is going to refuse to increase debt and spending, as long as people are willing to keep buying it at a low interest rate). And from this perspective it's not really a problem except for everyone getting used to the higher spending when some of the processes increasing the demand for government debt might only be temporary.

AI written explanation of how financialization causes increased demand for government debt

    • Financialization isn't a vague blob; it's a set of specific, concrete processes, each of which acts like a powerful vacuum cleaner sucking up government debt.

      Let's trace four of the most important mechanisms in detail.

      1. The Derivatives Market: The Collateral Multiplier

      Derivatives (options, futures, swaps) are essentially financial side-bets on the movem

... (read more)

Human perception of society has some paradoxes. Consider freedom of speech: In countries that generally have freedom of speech, many people complain about all kinds of injustice, censorship, etc. In countries that have no freedom of speech, everyone is quiet, and when asked explicitly, says: everything is great. Therefore, naive observers often conclude that the former countries have less freedom of speech than the latter, judging by the number of complaints about censorship.

I believe there is a similar effect with meritocracy/equality/etc. Imagine a perfectly unfair feudal society where unless you are born as a member of aristocracy, you are screwed; your talents and hard work will make absolutely no difference. Ironically, many people will believe that this society is fair, that the aristocrats are chosen by God for being better. If the poor kids are never given an opportunity to learn, everyone may believe, based on what they observe, that the poor kids are completely unable to learn. This is what all their priests would teach.

Then comes a revolution, and people find out that the aristocrats are often stupid, and that if you give free education to the poor kids, many of them tur... (read more)

This makes no mention of the repeal of the fairness doctrine nor the shift in financial model for major newspapers. The 1987 abolishment of the Fairness Doctrine led very directly to Rush Limbaugh gaining a national political audience.

The fairness doctrine of the United States Federal Communications Commission (FCC), introduced in 1949, was a policy that required the holders of broadcast licenses both to present controversial issues of public importance and to do so in a manner that fairly reflected differing viewpoints.[1] In 1987, the FCC abolished the fairness doctrine,[2] prompting some to urge its reintroduction through either Commission policy or congressional legislation.[3] The FCC removed the rule that implemented the policy from the Federal Register in August 2011.[4]

Rush Limbaugh used to be a regular music DJ in the 1970s. His political talk show was distributed nationally in 1988, soon after the cessation of Fairness Doctrine support by the Reagan administration. Carl McIntire was the Rush Limbaugh (maybe a little more Glenn Beck) of the 1960s and had his radio talk show shut down by the Fairness Doctrine after years of litigation. The legal challenges the Fairness ... (read more)

To be honest, this post approaches a level of disjointedness from reality-as-I-understand-it that I fear I cannot accurately respond to it in a way that would satisfy the author.[1] However, if I don't give some response, I suspect that it will join a growing cultural current within LessWrong which involves an intellectualised rationalisation of ethnonationalism, hygiene-oriented eugenics, oligarchy[2], and implicit or explicit support of political action and political violence for these ends. If this current becomes normalised (and it has, frankly, always been present in the rationalist sphere), it makes TESCREAL-style categorisations of the rationalist/EA/AI safety intellectual sphere bascially correct. Therefore, I want to commit to voicing my objection to this line of argument where I see it. There are parts of me that feel strongly against doing this. There is no glory to be gained in political arguments online. However, there is the possibility of avoiding shame, which as a virtue ethicist I'm sure the author will understand.

On the question of inherent racial or ethnic attributes, I suspect that I will not be able to settle the argument "on the facts", given how long the... (read more)

If this current becomes normalised (and it has, frankly, always been present in the rationalist sphere), it makes TESCREAL-style categorisations of the rationalist/EA/AI safety intellectual sphere bascially correct.

What? It seems like TESCREAL is clearly a natural cluster (modulo I don't know any examples of the "C"), and whether takes like this are pervasive amongst rationalists doesn't bare on whether it's a good categorization?

 

A bit late, but the "C" stands for Cosmism. Effectively Communist/Russian proto-transhumanists. If you've read it, think Tera Ignota's Utopians.

I'm not sure how this is a response to the OP. It sounds basically right to me (and I imagine that Richard would agree with it as well, though he can speak for himself), but it seems like almost entirely a non sequitur to the claims made?

the author's "solutions" of ethnic segregation and socially-enforced virtue ethics enforced by inter-group mutual distrust

This text doesn't even mention ethnic segregation as a solution? It does promote virtue ethics as an alternative moral frame to utilitarianism, but it says nothing about "enforced[ing] by inter-group mutual distrust".

I don't doubt that you're pushing back against a real cultural force, but it doesn't look to me to actually be represented in this short-form, except (at best) as an implication.

For the most obvious example, for the life of me I cannot understand how leaving the gold standard makes a culture less appreciative of any kind of moral virtue, unless you equate two very different senses of the word "value".

Might reply to the rest later but just to respond to what you call "the most obvious example": consider a company which has a difficult time evaluating how well its employees are performing (i.e. most of them). Some employees will work hard even when they won't directly be rewarded for that, because they consider it virtuous to do so. However, if you then add to their team a bunch of other people who are rewarded for slacking off, the hard-working employees may become demotivated and feel like they're chumps for even trying to be virtuous.

The extent to which modern governments hand out money causes a similar effect across western societies (edited: for example, if many people around you are receiving welfare, then working hard yourself is less motivating). They would not be as able to do this as much if their currencies were still on the gold standard, because it would be more obvious that they are insolvent.

The extent to which modern governments hand out money causes a similar effect across western societies. They would not be as able to do this as much if their currencies were still on the gold standard, because it would be more obvious that they are insolvent.

 

modern governments

western societies

gold standard

insolvent

I'm afraid, if you're actually trying to advance an argument, you're going to need to be slightly more specific. Which government, in which society? Perhaps you will say, "all of the western governments". Very well. Then which gold standard? I will remind you that "western societies" including Germany, the UK, and the US exited and entered the "gold standard", or swapped out various forms of "gold standard", all throughout the 20th century. Was Britain in 1931 or Germany in 1914 lacking in valor? I'm by no means an economic historian, and this is just wikipedia talking. 

And even when countries obeyed the "gold standard", policies varied strongly between regions, including how much currency could be redeemed for how much gold, whether citizens could redeem gold directly, the status of gold as a commodity, the trade of gold between countries etc. In what combinat... (read more)

Thanks for the extensive comment. I'm not sure it's productive to debate this much on the object level. The main thing I want to highlight is that this is a very good example of how the taboo that I discussed above operates.

On most issues, people (and especially LWers) are generally open to thinking about the benefits and costs of each stance, since tradeoffs are real.

However, in the case of ethnonationalism, even discussing the taboo on it (without explicitly advocating for it) was enough to trigger a kind of zero-tolerance attitude in your comment.

This is all the more striking because the main historical opponent of ethnonationalist regimes was globalist communism, which also led to large-scale atrocities. Yet when people defend a "socialist" or "egalitarian" cluster of ideas, that doesn't lead to anywhere near this level of visceral response.

My main bid here is for readers to notice that there is a striking asymmetry in how we think about and discuss 20th century history, which is best explained via the thing I hypothesized above: a strong taboo on ethnonationalism in the wake of WW2, which has then distorted our ability to think about many other issues.

I could imagine a world where discussing it is not something I would see as critically unwelcome, but I would still want to maintain part of the mechanism that implements the taboo, which is why I made the comments I did: being able to come to agreement that there are outcomes which are good and bad, such as the previous famous outcomes of ethnonationalism. I suspect that I've interpreted you to be saying that those previous outcomes were bad ("While perhaps at first this taboo was valuable") but it's not as reliable as the taboo implementation I carry would prefer.

If your view is that virtues are required to be independent of outcomes, in that eg a virtue whose adoption would predictably lead to mass death [edit to finish phrase: would still be a virtue if virtuous by its own merits], then I don't think they can be truly called virtuous; but if your view is that we can discuss some features of outcomes that make an outcome good or bad, then I would want to discuss what virtues can lead us towards better outcomes by that standard. virtue utilitarianism, if you will, rather than rule utilitarianism[1]; the way I heard virtues described was that a virtue is a local aspiration, a non-... (read more)

This is a thoughtful comment, I appreciate it, and I'll reply when I have more time (hopefully in a few days).

I suspect that I've interpreted you to be saying that those previous outcomes were bad ("While perhaps at first this taboo was valuable")

Yes, ethnonationalism by white Europeans previously led to some very bad outcomes. The reason I said "perhaps" is that I don't know if implementing the taboo was in fact necessary to prevent it from happening again (as opposed to, say, trusting people to learn from history).

if your view is that we can discuss some features of outcomes that make an outcome good or bad, then I would want to discuss what virtues can lead us towards better outcomes by that standard. virtue utilitarianism, if you will, rather than rule utilitarianism

I endorse the first sentence but not the second. Virtue utilitarianism holds the consequences to be the ultimate thing that matter, and virtues as a mechanism for reaching them. Whereas I think of "produces good outcomes" as one criterion by which to evaluate virtues, but not the only one (because humans in fact care about more than just outcomes, and because figuring out what counts as "good" is hard to disentangle from understanding virtues themselves).

A "friendly, non-destructive ethnonationalism" that is prosocial towar

... (read more)

If you press them too closely, they will abruptly fall silent, loftily indicating by some phrase that the time for argument is past.

Jean-Paul Sartre

I sincerely believe that people will get hurt if these ideas return to society at large, Richard. Please don't do this.

It seems to me that Richard isn't trying to bring back ethnonationalism, or even trying to "add just that touch of ethnic pride back into the meme pool", but just trying to diagnose "how the western world got so dysfunctional". If ethnonationalism and the taboo against ethnonationalism are both bad (as an ethnic minority I'm personally pretty scared of the former), then maybe we should get rid of the taboo and defend against ethnonationalism by other means, similar how there is little to no taboo against communism[1] but it hasn't come close to taking power or reapproaching its historical high water mark in the west.

  1. ^

    If you doubt this, there's an advisor to my local school district who is a self-avowed Marxist and professor of education at the state university, and writes book reviews like this one:
    «For decades the educational Left and critical pedagogues have run away from Marxism, socialism, and communism, all too often based on faulty understandings and falling prey to the deep-seated anti-communism in the academy. In History and Education Curry Stephenson Malott pushes back against this trend by offering us deeply Marxist thinking about the circulation of capital, socialis

... (read more)

I like this comment.

For the sake of transparency, while in this post I'm mostly trying to identify a diagnosis, in the longer term I expect to try to do political advocacy as well. And it's reasonable to expect that people like me who are willing to break the taboo for the purposes of diagnosis will be more sympathetic to ethnonationalism in their advocacy than people who aren't. For example, I've previously argued on twitter that South Africa should have split into two roughly-ethnonationalist states in the 90s, instead of doing what they actually did.

However, I expect that the best ways of fixing western countries won't involve very much ethnonationalism by historical standards, because it's a very blunt tool. Also, I suspect that breaking the taboo now will actually lead to less ethnonationalism in the long term. For example, even a little bit more ethnonationalism would plausibly have made European immigration policies much less insane over the last few decades, which would then have prevented a lot of the political polarization we're seeing today.

in the longer term I expect to try to do political advocacy as well

I find this idea particularly fraught. I already find it somewhat difficult to engage on this site due to the contentious theories some members hold, and I echo testingthewater's warning against the trap of reopening these old controversies. You're trying to thread a really fine needle between "meaningfully advocate change" and "open all possible debates" that I don't think is feasible.

The site is currently watching a major push from Yudkowsky and Soares' book launch towards a broad coalition for an AI pause. It really only takes a couple major incidents of connecting the idea to ethnonationalism, scientific racism and/or dictatorship for the targets of your advocacy to turn away.

I'm not going to suggest you stay on-message (lw is way too "truth-seeking" for that to reach anyone), but you should carefully consider the ways in which your future goals conflict.

I seriously doubt Richard recommends any of the regimes/interventions you actually argue against here. 

I still maintain my fundamental belief that all human beings (and, increasingly I would argue, all life) are worthy of a deep and sincerely-held respect, regardless of any intellectual or physical discrepancies between them.

I think the problem is that a part of "respecting" people is letting them choose things for themselves, and in a democratic society also letting them choose things for others.

I admit I do have a problem respecting many people in this specific way. Not sure what to do about it though.

I was reflecting on some of the takes here for a bit and if I imagine a blind gradient descent in this direction, I imagine quite a lot of potential reality distortion fields due to various of the underlying dynamics involved with holding this position.

So the one thing I wanted to ask was that if you have any sort of reset mechanism here? Like what is the schelling point before the slippery slope? What is the specific action pattern you would take if you got too far? Or do you trust future you enough in order to ensure that it won't happen? 

Good question. One answer is that my reset mechanisms involve cultivating empathy, and replacing fear with positive motivation. If I notice myself being too unempathetic or too fear-driven, that's worrying.

But another answer is just that, unfortunately, the reality distortion fields are everywhere—and in many ways more prevalent in "mainstream" positions (as discussed in my post). Being more mainstream does get you "safety in numbers"—i.e. it's harder for you to catalyze big things, for better or worse. But the cost is that you end up in groupthink.

The black hole of western state debt that will never be repaid creates distortions across the economy

Can you give some examples? 

Big government debts seem like the kind of thing that can fail catastrophically at some point, so they represent an important civilizational risk. But how are they distorting the economy? Do you just mean that governments spend much more than they raise in taxes?

This is a theory often referred to as the Cantillian effect

Richard Cantillon observed that the original recipients of new money enjoy higher standards of living at the expense of later recipients. In colloquial terms, the closer you stand to the source of money creation, the wealthier you become. When governments run large deficits that get monetized by central banks, this creates new money that flows first to government and financial sectors before reaching the broader economy. This distorts resource allocation because entities closer to the money source can bid up assets and resources before prices adjust throughout the system.

From a rhetorical perspective, I think it was wrong to lead with the ethnonationalism stuff, since it seems to have confused testingthewaters, and likely many more.

Ethnonationalism doesn't scale well, because you run out of the original ethnicity some point and have to start relying on others as the great empires mostly did. Which shows that monoethnicity isn't necessary for greatness. The proselytizing religions are also multiethnic, which shows that monoethnicity isn't necessary for co-operation. Ethnonationalism has large downsides and weak upsides.

+1 to what @Douglas_Knight said about IQ.

This feels like you started from a position that you wanted to believe and worked backwards, looking for reasons to believe it, rather than the other way around. The overwhelming majority of successful civilizations throughout history have been nation-states, at least to some degree, which should be addressed in some capacity by any argument claiming that it's a suboptimal way to organize a society.

It is true that large empires are often not monoethnic, but this seems like a structural challenge of building a large empire[1] rather than a question of how e... (read more)

The overwhelming majority of successful civilizations throughout history have been nation-states, at least to some degree,

The vast majority of unsuccessful ones have been ethnostates, as well, because the vast majority are ethnostates ... on average, ethnostates are average.

Indeed, large empires tend to be authoritarian rather than democratic, and few people use this as an argument for why authoritarianism is intrinsically better than democracy.

I spoke of greatness, not goodness.

The points about IQ seem parochially American, not applicable to the rest of the West. But aren't you British, not American? Does this really seem so central in Britain?

That is such a good post! Thank you.

I find it hard to tell what you see as good or bad, among these things, especially around what currently is, vs what you would see in the world. I of course can read what you've explicitly stated as good, but the way you've written this puts it nearby in linguistic vector space to things which are controversial at best, but are intertwined with processes that I would expect to cause seriously bad things. I would appreciate if you could find a way to more clearly signal what outcomes you are inclined towards, to the degree that avoiding accidental pessimiza... (read more)

Since consequentialists think that morality is essentially about altruism, much moral philosophy actively undermines ethics. So does modern economics, via smuggling in the assumption that utility functions represent selfish preferences.

As git hashsums are short and tangible True Names of abstract git objects, there are abstract properties of behaviors of things in the world that are in principle as concrete and tangible as coal or silver. The economy uses abstract goods to produce new abstract goods. Consequentialism and utility functions or policies co... (read more)

Consequentialism and utility functions or policies could in principle be about virtues and integrity as about hamburgers, but hamburgers are more legible and easier to administer.

Here's one concrete way in which this isn't true: one common simplifying assumption in economics is that goods are homogeneous, and therefore that you're indifferent about who to buy from. However, virtuous behavior involves rewarding people you think are more virtuous (e.g. by preferentially buying things from them).

In other words, economics is about how agents interact with each other via exchanging goods and services, while virtues are about how agents interact with each other more generally.

I don't see why using the word "virtue" magically solves the hardness of that math (the reasons why such assumptions are simplifying). Maybe that is what your promised formalizations are about, but I'm also kinda skeptical that reputation and consumers' preference to trade with some groups over others is a subject that no economist could give reasonable models about.

It seems very reasonable to be indifferent about who ends up incentivising or exhibiting virtuous behaviors you care about (to be more prevalent in the world or your community), and the incentives don't need to come in the form of personal action about who to buy other things from.

In other words, economics is about how agents interact with each other via exchanging goods and services, while virtues are about how agents interact with each other more generally.

If virtues and other abstract properties of behaviors are treated as particular examples of goo... (read more)

(Written quickly and not very carefully.)

I think it's worth stating publicly that I have a significant disagreement with a number of recent presentations of AI risk, in particular Ajeya's "Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover", and Cohen et al.'s "Advanced artificial agents intervene in the provision of reward". They focus on policies learning the goal of getting high reward. But I have two problems with this:

  1. I expect "reward" to be a hard goal to learn, because it's a pretty abstract concept and not closely related to the direct observations that policies are going to receive. If you keep training policies, maybe they'd converge to it eventually, but my guess is that this would take long enough that we'd already have superhuman AIs which would either have killed us or solved alignment for us (or at least started using gradient hacking strategies which undermine the "convergence" argument). Analogously, humans don't care very much at all about the specific connections between our reward centers and the rest of our brains - insofar as we do want to influence them it's because we care about much more directly-observable p
... (read more)

I'm not very convinced by this comment as an objection to "50% AI grabs power to get reward." (I find it more plausible as an objection to "AI will definitely grab power to get reward.")

I expect "reward" to be a hard goal to learn, because it's a pretty abstract concept and not closely related to the direct observations that policies are going to receive

"Reward" is not a very natural concept

This seems to be most of your position but I'm skeptical (and it's kind of just asserted without argument):

  • The data used in training is literally the only thing that AI systems observe, and prima facie reward just seems like another kind of data that plays a similarly central role. Maybe your "unnaturalness" abstraction can make finer-grained distinctions than that, but I don't think I buy it.
  • If people train their AI with RLDT then the AI is literally be trained to predict reward! I don't see how this is remote, and I'm not clear if your position is that e.g. the value function will be bad at predicting reward because it is an "unnatural" target for supervised learning.
  • I don't understand the analogy with humans. It sounds like you are saying "an AI system selected based on the reward of its acti
... (read more)

If people train AI systems on random samples of deployment, then "reward" does make sense---it's just what would happen if you sampled this episode to train on.

I don't know what this means. Suppose we have an AI which "cares about reward" (as you think of it in this situation). The "episode" consists of the AI copying its network & activations to another off-site server, and then the original lab blows up. The original reward register no longer exists (it got blown up), and the agent is not presently being trained by an RL alg. 

What is the "reward" for this situation? What would have happened if we "sampled" this episode during training?

I agree there are all kinds of situations where the generalization of "reward" is ambiguous and lots of different things could happen . But it has a clear interpretation for the typical deployment episode since we can take counterfactuals over the randomization used to select training data.

It's possible that agents may specifically want to navigate towards situations where RL training is not happening and the notion of reward becomes ambiguous, and indeed this is quite explicitly discussed in the document Richard is replying to.

As far as I can tell the fact that there exist cases where different generalizations of reward behave differently does not undermine the point at all.

Yeah, I think I was wondering about the intended scoping of your statement. I perceive myself to agree with you that there are situations (like LLM training to get an alignment research assistant) where "what if we had sampled during training?" is well-defined and fine. I was wondering if you viewed this as a general question we could ask.

I also agree that Ajeya's post addresses this "ambiguity" question, which is nice!

I'm not very convinced by this comment as an objection to "50% AI grabs power to get reward." (I find it more plausible as an objection to "AI will definitely grab power to get reward.")

It's intended as an objection to "AI grabs power to get reward is the central threat model to focus on", but I think our disagreements still apply given this. (FWIW my central threat model is that policies care about reward to some extent, but that the goals which actually motivate them to do power-seeking things are more object-level.)

The data used in training is literally

... (read more)

(FWIW my central threat model is that policies care about reward to some extent, but that the goals which actually motivate them to do power-seeking things are more object-level.)

Are you imagining that such systems get meaningfully low reward on the training distribution because they are pursuing those goals, or that these goals are extremely well-correlated with reward on the training distribution and only come apart at test time? Is the model deceptively aligned?

I expect policies to be getting rich input streams like video, text, etc, which they use to m

... (read more)

Reading back over this now, I think we're arguing at cross purposes in some ways. I should have clarified earlier that my specific argument was against policies learning a terminal goal of reward that generalizes to long-term power-seeking.

I do expect deceptive alignment after policies learn other broadly-scoped terminal goals and realize that reward-maximization is a good instrumental strategy. So all my arguments about the naturalness of reward-maximization as a goal are focused on the question of which type of terminal goal policies with dangerous level... (read more)

I think the "soup of heuristics" stories (where the AI is optimizing something far causally upstream of reward instead of something that is downstream or close enough to be robustly correlated) don't lead to takeover in the same way

Why does it not lead to takeover in the same way?

Because it's easy to detect and correct (except that correcting it might push you into one of the other regimes).

So far causally upstream of the human evaluator's opinion? Eg an AI counselor optimizing for getting to know you

I don't understand the analogy with humans. It sounds like you are saying "an AI system selected based on the reward of its actions learns to select actions it expects to lead to high reward" be analogous to "humans care about the details of their reward circuitry." But: 

  • I don't think human learning is just RL based on the reward circuit; I think this is at least a contrarian position and it seems unworkable to me as an explanation of human behavior.

(Emphasis added)

I don't think this engages with the substance of the analogy to humans. I don't think a... (read more)

This is incredibly weak evidence.

  • Animals were selected over millions of generations to effectively pursue external goals. So yes, they have external goals.
  • Humans also engage in within-lifetime learning, so of course you see all kinds of indicators of that in brains.

Both of those observations have high probability, so they aren't significant Bayesian evidence for "RL tends to produce external goals by default."

In particular, for this to be evidence for Richard's claim, you need to say: "If RL tended to produce systems that care about reward, then RL would b... (read more)

(my guess is that other systems play a significant role in our conscious attitude towards pleasure)

Curious what systems you have in mind here.

Animals were selected over millions of generations to effectively pursue external goals. So yes, they have external goals.

I don't understand why you think this explains away the evidential impact, and I guess I put way less weight on selection reasoning than you do. My reasoning here goes:

  1. Lots of animals do reinforcement learning.
  2. In particular, humans prominently do reinforcement learning.
  3. Humans care about lots of things in reality, not just certain kinds of cognitive-update-signals.
  4. "RL -> high chance of caring about reality" predicts this observation m
... (read more)

You seem to be saying P(humans care about the real world | RL agents usually care about reward) is low. I'm objecting, and claiming that in fact P(humans care about the real world | RL agents usually care about reward) is fairly high, because humans are selected to care about the real world and evolution can be picky about what kind of RL it does, and it can (and does) throw tons of other stuff in there.

The Bayesian update is P(humans care about the real world | RL agents usually care about reward) / P(humans care about the real world | RL agents mostly ca... (read more)

Just saw this reply recently. Thanks for leaving it, I found it stimulating.

(I wrote the following rather quickly, in an attempt to write anything at all, as I find it not that pleasant to write LW comments -- no offense to you in particular. Apologies if it's confusing or unclear.)

You seem to be saying P(humans care about the real world | RL agents usually care about reward) is low.

Yes, in large part.

I'm objecting, and claiming that in fact P(humans care about the real world | RL agents usually care about reward) is fairly high, because humans are selecte

... (read more)

POV: I'm in an ancestral environment, and I (somehow) only care about the rewarding feeling of eating bread. I only care about the nice feeling which comes from having sex, or watching the birth of my son, or being gaining power in the tribe. I don't care about the real-world status of my actual son, although I might have strictly instrumental heuristics about e.g. how to keep him safe and well-fed in certain situations, as cognitive shortcuts for getting reward (but not as terminal values). 

Would such a person sacrifice themselves for their children (in situations where doing so would be a fitness advantage)?

I think this highlights a good counterpoint. I think this alternate theory predicts "probably not", although I can contrive hypotheses for why people would sacrifice themselves (because they have learned that high-status -> reward; and it's high-status to sacrifice yourself for your kid). Or because keeping your kid safe -> high reward as another learned drive.

Overall this feels like contortion but I think it's possible. Maybe overall this is a... 1-bit update against the "not selection for caring about reality" point?

Putting my money where my mouth is: I just uploaded a (significantly revised) version of my Alignment Problem position paper, where I attempt to describe the AGI alignment problem as rigorously as possible. The current version only has "policy learns to care about reward directly" as a footnote; I can imagine updating it based on the outcome of this discussion though.

For someone who's read v1 of this paper, what would you recommend as the best way to "update" to v3? Is an entire reread the best approach?

[Edit March 11, 2023: Having now read the new version in full, my recommendation to anyone else with the same question is a full reread.]

[+][comment deleted]10

Note that the "without countermeasures" post consistently discusses both possibilities (the model cares about reward or the model cares about something else that's consistent with it getting very high reward on the training dataset). E.g. see this paragraph from the above-the-fold intro:

Once this progresses far enough, the best way for Alex to accomplish most possible “goals” no longer looks like “essentially give humans what they want but take opportunities to manipulate them here and there.” It looks more like “seize the power to permanently direct how

... (read more)

Note that the "without countermeasures" post consistently discusses both possibilities

Yepp, agreed, the thing I'm objecting to is how you mainly focus on the reward case, and then say "but the same dynamics apply in other cases too..."

I do place a ton of emphasis on the fact that Alex enacts a policy which has the empirical effect of maximizing reward, but that's distinct from being confident in the motivations that give rise to that policy.

The problem is that you need to reason about generalization to novel situations somehow, and in practice that ends up being by reasoning about the underlying motivations (whether implicitly or explicitly).

I agree with your general point here, but I think Ajeya's post actually gets this right, eg

There is some ambiguity about what exactly “maximize reward” means, but once Alex is sufficiently powerful -- and once human knowledge/control has eroded enough -- an uprising or coup eventually seems to be the reward-maximizing move under most interpretations of “reward.”

and

What if Alex doesn’t generalize to maximizing its reward in the deployment setting? What if it has more complex behaviors or “motives” that aren’t directly and simply derived from

... (read more)

I also think that often "the AI just maximizes reward" is a useful simplifying assumption. That is, we can make an argument of the form "even if the AI just maximizes reward, it still takes over; if it maximizes some correlate of the reward instead, then we have even less control over what it does and so are even more doomed".

(Though of course it's important to spell the argument out)

Yeah, I agree this is a good argument structure -- in my mind, maximizing reward is both a plausible case (which Richard might disagree with) and the best case (conditional on it being strategic at all and not a bag of heuristics), so it's quite useful to establish that it's doomed; that's the kind of structure I was going for in the post.

I strongly disagree with the "best case" thing. Like, policies could just learn human values! It's not that implausible.

If I had to try point to the crux here, it might be "how much selection pressure is needed to make policies learn goals that are abstractly related to their training data, as opposed to goals that are fairly concretely related to their training data?" Where we both agree that there's some selection pressure towards reward-like goals, and it seems like you expect this to be enough to lead policies to behavior that violates all their existi... (read more)

I strongly disagree with the "best case" thing. Like, policies could just learn human values! It's not that implausible.

Yes, sorry, "best case" was oversimplified. What I meant is that generalizing to want reward is in some sense the model generalizing "correctly;" we could get lucky and have it generalize "incorrectly" in an important sense in a way that happens to be beneficial to us. I discuss this a bit more here.

But if Alex did initially develop a benevolent goal like “empower humans,” the straightforward and “naive” way of acting on that goal wo

... (read more)

I don't understand why reward isn't something the model has direct access to -- it seems like it basically does? If I had to say which of us were focusing on abstract vs concrete goals, I'd have said I was thinking about concrete goals and you were thinking about abstract ones, so I think we have some disagreement of intuition here.

AFAIK the reward signal is not typically included as an input to the policy network in RL. Not sure why, and I could be wrong about that, but that is not my main question. The bigger question is "Has direct access to when?"

At th... (read more)

What I meant is that generalizing to want reward is in some sense the model generalizing "correctly;" we could get lucky and have it generalize "incorrectly" in an important sense in a way that happens to be beneficial to us.

See also: Inner and outer alignment decompose one hard problem into two extremely hard problems (in particular: Inner alignment seems anti-natural).

Five clusters of alignment researchers

Very broadly speaking, alignment researchers seem to fall into five different clusters when it comes to thinking about AI risk:

  1. MIRI cluster. Think that P(doom) is very high, based on intuitions about instrumental convergence, deceptive alignment, etc. Does work that's very different from mainstream ML. Central members: Eliezer Yudkowsky, Nate Soares.
  2. Structural risk cluster. Think that doom is more likely than not, but not for the same reasons as the MIRI cluster. Instead, this cluster focuses on systemic risks, multi-agent alignment, selective forces outside gradient descent, etc. Often work that's fairly continuous with mainstream ML, but willing to be unusually speculative by the standards of the field. Central members: Dan Hendrycks, David Krueger, Andrew Critch.
  3. Constellation cluster. More optimistic than either of the previous two clusters. Focuses more on risk from power-seeking AI than the structural risk cluster, but does work that is more speculative or conceptually-oriented than mainstream ML. Central members: Paul Christiano, Buck Shlegeris, Holden Karnofsky. (Named after Constellation coworking space.)
  4. Prosaic cluster. Focuses on empi
... (read more)

When you think of goals as reward/utility functions, the distinction between positive and negative motivations (e.g. as laid out in this sequence) isn’t very meaningful, since it all depends on how you normalize them.

But when you think of goals as world-models (as in predictive processing/active inference) then it’s a very sharp distinction: your world-model-goals can either be of things you should move towards, or things you should move away from.

This updates me towards thinking that the positive/negative motivation distinction is more meaningful than I thought.

In run-and-tumble motion, “things are going well” implies “keep going”, whereas “things are going badly” implies “choose a new direction at random”. Very different! And I suggest in §1.3 here that there’s an unbroken line of descent from the run-and-tumble signal in our worm-like common ancestor with C. elegans, to the “valence” signal that makes things seem good or bad in our human minds. (Suggestively, both run-and-tumble in C. elegans, and the human valence, are dopamine signals!)

So if some idea pops into your head, “maybe I’ll stand up”, and it seems appealing, then you immediately stand up (the human “run”); if it seems unappealing on net, then that thought goes away and you start thinking about something else instead, semi-randomly (the human “tumble”).

So positive and negative are deeply different. Of course, we should still call this an RL algorithm. It’s just that it’s an RL algorithm that involves a (possibly time- and situation-dependent) heuristic estimator of the expected value of a new random plan (a.k.a. the expected reward if you randomly tumble). If you’re way above that expected value, then keep doing whatever you’re doing; if you’re way below the threshold, re-rol... (read more)

i don't think this is unique to world models. you can also think of rewards as things you move towards or away from. this is compatible with translation/scaling-invariance because if you move towards everything but move towards X even more, then in the long run you will do more of X on net, because you only have so much probability mass to go around.

i have an alternative hypothesis for why positive and negative motivation feel distinct in humans.

although the expectation of the reward gradient doesn't change if you translate the reward, it hugely affects the variance of the gradient.[1] in other words, if you always move towards everything, you will still eventually learn the right thing, but it will take a lot longer.

my hypothesis is that humans have some hard coded baseline for variance reduction. in the ancestral environment, the expectation of perceived reward was centered around where zero feels to be. our minds do try to adjust to changes in distribution (e.g hedonic adaptation), but it's not perfect, and so in the current world, our baseline may be suboptimal.

 

  1. ^

    Quick proof sketch (this is a very standard result in RL and is the motivation for advantage estimation, b

... (read more)

if you move towards everything but move towards X even more, then in the long run you will do more of X on net, because you only have so much probability mass to go around

I have a mental category of "results that are almost entirely irrelevant for realistically-computationally-bounded agents" (e.g. results related to AIXI), and my gut sense is that this seems like one such result.

I mean this situation is grounded & formal enough you can just go and implement the relevant RL algorithm and see if its relevant for that computationally bounded agent, right?

is this for a reason other than the variance thing I mention?

I think the thing I mention is still important is because it means there is no fundamental difference between positive and negative motivation. I agree that if everything was different degrees of extreme bliss then the variance would be so high that you never learn anything in practice. but if you shift everything slightly such that some mildly unpleasant things are now mildly pleasant, I claim this will make learning a bit faster or slower but still converge to the same thing.

Suppose you're in a setting where the world is so large that you will only ever experience a tiny fraction of it directly, and you have to figure out the rest via generalization. Then your argument doesn't hold up: shifting the mean might totally break your learning. But I claim that the real world is like this. So I am inherently skeptical of any result (like most convergence results) that rely on just trying approximately everything and gradually learning which to prefer and disprefer.

are you saying something like: you can't actually do more of everything except one thing, because you'll never do everything. so there's a lot of variance that comes from exploration that multiplies with your  variance from having a suboptimal zero point. so in practice your  needs to be very close to optimal. so my thing is true but not useful in practice.

i feel people do empirically shift  quite a lot throughout life and it does seem to change how effectively they learn. if you're mildly depressed your  is ... (read more)

This reminds me of a conversation I had recently about whether the concept of "evil" is useful. I was arguing that I found "evil"/"corruption" helpful as a handle for a more model-free "move away from this kind of thing even if you can't predict how exactly it would be bad" relationship to a thing, which I found hard to express in a more consequentialist frames.

I feel like "evil" and "corruption" mean something different.

Corruption is about selfish people exchanging their power within a system for favors (often outside the system) when they're not supposed to according to the rules of the system. For example a policeman taking bribes. It's something the creators/owners of the system should try to eliminate, but if the system itself is bad (e.g. Nazi Germany during the Holocaust), corruption might be something you sometimes ought to seek out instead of to avoid, like with Schindler saving his Jews.

"Evil" I've in t... (read more)

In (non-monotonic) infra-Bayesian physicalism, there is a vaguely similar asymmetry even though it's formalized via a loss function. Roughly speaking, the loss function expresses preferences over "which computations are running". This means that you can have a "positive" preference for a particular computation to run or a "negative" preference for a particular computation not to run[1].

  1. ^

    There are also more complicated possibilities, such as "if P runs then I want Q to run but if P doesn't run then I rather that Q also doesn't run" or even preferences that are only expressible in terms of entanglement between computations.

Reminds me of @MalcolmOcean 's post on how awayness can't aim (except maybe in 1D worlds) since it can only move away from things, and aiming at a target requires going toward something.

Imagine trying to steer someone to stop in one exact spot. You can place a ❤ beacon they’ll move towards, or an X beacon they’ll move away from. (Reverse for pirates I guess.)

In a hallway, you can kinda trap them in the middle of two Xs, or just put the ❤ in the exact spot.

In an open field, you can maybe trap them in the middle of a bunch of XXXXs, but that’ll be hard becau

... (read more)

In Richard Jeffrey's utility theory there is actually a very natural distinction between positive and negative motivations/desires. A plausible axiom is (the tautology has zero desirability: you already know it's true). Which implies with the main axiom[1] that the negation of any proposition with positive utility has negative utility, and vice versa. Which is intuitive: If something is good, its negation is bad, and the other way round. In particular, if (indifference between and ), then .

More generally, ... (read more)

I've struggled to engage very productively with the rest of the AI safety community about politics—e.g. there are a lot of disagreeing votes and comments on this post. By and large people have been respectful and polite (as per the high upvote count), but right now it doesn't feel like LW is a place where I can collaboratively make intellectual progress on this very important topic.

This is sad, so before I stop trying I want to attempt to go meta, by explaining how this seems analogous to the ways that the alignment community struggled to engage with mainstream ML researchers throughout the 2010s and early 2020s. My explanation of those dynamics ended up growing into this post; in this shortform I'll discuss the analogy to politics specifically.

The main point is the following. With enough discussion, you could often get a mainstream ML researcher to admit that something like situational awareness or recursive self-improvement might in principle be possible. But it's very hard to get them to take it seriously enough that it propagates through their ontology—and so after that one conversation they'd typically just go back to their standard ML research. This is for a bunch of reasons—... (read more)

3211

It's probably difficult to respond to this in a way that's satisfying to you, because I and most other people are not paid independently to post on the internet, and so there's limits to what we can say in public. But every man under the age of 30 that I've ever met at Lighthaven, that I've had the opportunity to speak with privately, has completely and totally integrated IQ differences into their ontology. There's no hem and hawing, they've just accepted it as part of their worldview.

The reason I disagree voted with your post is not because it touches on 'taboos', it's because it's a vast inferential leap, developed piecemeal from assorted blogs and sociological studies you've gathered on the internet. Anybody that doesn't both share your priors and information feed almost exactly will naturally end up disagreeing with large portions of it, even if they agree with you that the cause of black poverty is genetics. And IMO for good reason, because large portions of the post are generated from claims like "Elites are pro-Hamas" that are just literally and obviously false, and that you're treating as background knowledge that I'm supposed to share.

Do you have a discussion about iq differences with every man under 30 you meet at Lighthaven?

has completely and totally integrated IQ differences into their ontology. There's no hem and hawing, they've just accepted it as part of their worldview.

Just noting that these are different things, and I think Richard is attempting to point at the differences between them.

eg, he's claiming that Scott probably believes that there are group differences in intelligence (it's a part of his worldview), but is also flinching away from propagating all the implications.

(However this doesn't bear on the main thrust of your comment.

eg, he's claiming that Scott probably believes that there are group differences in intelligence (it's a part of his worldview), but is also flinching away from propagating all the implications.

Right, and what I'm saying is that they're very explicit about it and propagate the implications.

Some of these topics are just unsafe to openly discuss (both reputationally and partially because of externalities). In fact, in the post you’re referring to, you avoid making any direct or clear point about racial IQ differences or ethnonationalism (there’s a lot of “certain obvious facts” style phrasing) which means you also don’t include any convincing supporting evidence! This is certainly understandable, but it seems strange to blame the resulting unproductivity of the conversation mainly on lesswrong. The post is actually hard to productively engage with. 
 
(Personally, I’d like to know what you’re actually saying and whether you’re right/wrong, but it seems hard to find out in the current epistemic environment, meaning the world not just lesswrong. Feel free to continue this conversation over DMs)

it seems strange to blame the resulting unproductivity of the conversation mainly on lesswrong

I didn't intend to assign blame. If I had a different intellectual style (e.g. if I were more methodical about building up chains of logic) then I agree it'd be much easier for people to productively engage with me.

It seems to me like an important thing about individual taboo facts is that it's not particularly advantageous to be correct on them, most of the time, because you can't publicly work with others on the basis of those facts.

Are you arguing that there's a political ontology which can only be understood by considering the fact that [redacted] is true, or one which can only be understood by considering the dynamics which lead to [redacted] being taboo despite being true? If it's the latter, I can much more easily see how that ontology could be productive for acting in the world. 

E.g. suppose you're in Salem and know that there are no witches. You can gain some small benefit by e.g. not spending money on counter-curses, but that's about it. Trying to spread and organize on the knowledge that there are no witches just gets you burned, but understanding why the trials are going on might actually help you survive.

It seems to me like an important thing about individual taboo facts is that it's not particularly advantageous to be correct on them, most of the time, because you can't publicly work with others on the basis of those facts.

Compared with most other facts that are similarly not widely believed, but are not taboo, it seems clear that taboo facts are far more advantageous to know. That's because most uncontroversial not-widely-believed facts are very unimportant, while taboo facts tend to be highly important and have wide implications (just consider atheis... (read more)

I think you might be over-updating from your original post. You had a lot of somewhat unrelated and potentially politically sensitive statements (ethnonationalism, IQ, managerial class, ethics, government debt, taboos, egalitarianism, AI stuff). Even if one agrees with the majority of your points, it is tempting to agreement-downvote  due to the minority, especially as they have high valency due to sensitive nature.

I don't think it's specific to sensitive topics, Richard just does a lot of sloppy thinking when he tries to engage with politics. His post/talk on more mundane political topics also led to a lot of people on LW & the EA Forum pointing out things he got wrong.

For the record: I do agree that a bunch of my political thinking is sloppy. Right now it feels like I'm facing a tradeoff between speed of conceptual progress and precision of thinking, and I'm optimizing primarily for the former.

One reason I discussed the analogy to ML above is because I hoped it would help people understand why I'm making this tradeoff. For example, I suspect that many LWers remember their thinking about AGI being called sloppy by the mainstream ML community because it didn't have equations. I think in hindsight it was the correct choice for LW to focus on this kind of "sloppy" exploratory thinking.

Having said that, it's clearly possible to go too far in this direction, and I regret giving the EAG talk in particular. More generally, there's a difference between doing sloppy thinking with intellectual collaborators vs broadcasting sloppy thinking to the world. Part of what I'm trying to figure out is the extent to which I should think of LW posts as the former vs the latter.

I regret giving the example of the disagree-votes, it's not that important to me, and I agree there are all sorts of reasons you might want to disagree-vote my previous post. I'm trying to point at a broader dynamic (and elaborate more on it in this reply to Raemon).

I didn't know we are allowed to discuss politics here. I thought that was banned? Sorry I didn't see the previous post. Anyway, here are some intellectual contributions:

For much of the twentieth century mass media was policed by The Fairness Doctrine. The Fairness Doctrine was a policy that required media outlets to cover controversial issues in a balanced manner that represented both sides of the argument. This was used against populist broadcasters like Carl McIntire (like a more Catholic version of Bill O'Reilly or Rush Limbaugh who also opposed the Civil Rights Movement).

The fairness doctrine of the United States Federal Communications Commission (FCC), introduced in 1949, was a policy that required the holders of broadcast licenses both to present controversial issues of public importance and to do so in a manner that fairly reflected differing viewpoints.[1] In 1987, the FCC abolished the fairness doctrine,[2] prompting some to urge its reintroduction through either Commission policy or congressional legislation.[3] The FCC removed the rule that implemented the policy from the Federal Register in August 2011.[4]

https://en.wikipedia.org/wiki/Fairness_doctrine

Rush Limbaugh'... (read more)

I didn't know we are allowed to discuss politics here.

Politics in the American political horse-race sense (for non-Americans this means political party competition between Republicans and Democrats) and of the "yay political ingroup, boo political outgroup" in general remains at least taboo, and rightly would be downvoted into oblivion even where it isn't strictly banned.

But the fully general "politics is the mindkiller" blanket ban on even mentioning politically connected subjects hasn't been true for years. Among other things, AI is now a highly politica... (read more)

Well that's it. I guess I am making this a frontpage post now.

It isn't banned, it's discouraged. Politics is the Mind-Killer so we try to avoid it.
 

I think I've seen you in two modes around your more controversial opinions:

  1. Demurring / 'not wanting to get into it' / 'expecting to be attacked'
  2. Aggressively shoe-horning in political examples without making their relevance over less-charged examples clear.

If you continue to hold this set of beliefs, my hope is that you come to feel less persecuted, such that you can unselfconsciously weave the true-to-you version of them into conversation, without drawing too much or too little attention.

However, I don't know if I'm capable of perceiving a third thing, given my beliefs. Like, maybe there is no 'sweet spot' and it will always seem to me that you're either being an evasive crypto fascist or awkwardly insisting on centering a -phobic/-ist line of argument, because my brain is broken. Afaict, this is what you believe about (people like) me, and I'm not really sure how to rule it out.

curious as to the strong negative response here; usually when I get downvoted a lot I kind of expect it ahead of time, but this one was surprising!

If anyone has guesses as to why this was so unpopular, I'd be interested to hear them.

Edit: the above comment was at -12 within 15 minutes of posting. My best guess now is 1 or 2 strong downvotes? Currently at -2. (I don’t care terminally; I just really don’t know if the comment is bad vs somehow narrowly offensive)

Edit 2: swung up to +17 (not totally sure that's the exact number), and now down to +2. Glad to be controversial, I suppose. Renewing my bid for anyone to tell me why this comment would be divisive. Genuinely confused.

you pierced the veil. the iron law around here is not to mention the author's intent/motivation, but only the content of their essay.

What would it look like if you were getting the engagement you wanted?

I'm not sure how much you care about that vs "this seems like it highlights something wrong/sad with the LW community", but, this part feels like an easier problem to solve.

The sort of thing I actually expect to work (for purposes of LW/rationalsphere being useful like a useful intellectual ground for you) is more like finding a few people who on a similar page as you, exploring the ideas proactively. So far I think your posts have felt like "okay, Richard is on some journey where he figures stuff out, and someday he might have more concrete takeaways or I might find myself working on problems where I think his frame is helpful."

Fwiw I have had some of your models in the background as I think about US politics right now. I see how, if one has those models, you might look at the situation fairly differently than most of my friends are looking at it. But, I think there are some details you believe that I don't believe that lead to pretty different assessments on what to do. (idk if it makes sense to get into the object level here). I don't know if you think your writing or linked references thus far should be sufficient to change my strategic frame.

Thanks for asking. I think the underlying issue here is that I'm in a period of boggling at wtf is going on with society. I have a sense that there's a bunch of insane stuff happening all over the place. Funnily enough, one of the people who's most sharply articulated a similar sense is Eliezer, when he wrote (I think in some glowfic) about how Earth is fractally disequilibriated, and the whole planet is made out of coordination failure.

But I think Eliezer and many rationalists maybe just take "the world is inadequate" as some kind of brute fact that doesn't really have a clear socio-historical-political explanation. Like, we used to be able to do Manhattan projects, and now the US govt is nowhere near coordinated enough to do that, but... eh, that's just how entropy works. Whereas it seems to me that actually it might be possible to trace the historical forces that contributed towards this, and the social principles that maintain it, and so on, to develop a fairly principled understanding of the situation.

However, this is a sufficiently ambitious project that my default strategy is to do a lot of exploration in a bunch of directions, which then leads to a lot of individual claims ... (read more)

Whereas it seems to me that actually it might be possible to trace the historical forces that contributed towards this, and the social principles that maintain it, and so on, to develop a fairly principled understanding of the situation.

As an example, I think the influence of the Soviet Union is underrated on the loss of American confidence / positive-self-conception. (Like I think it's obvious to Europeans that WWI / WWII did a lot to destroy European confidence / positive-self-conception, but I really don't think it had the same impact on the US, and our... (read more)

This framing is helpful.

I think the median position in rationalist circles is probably the following: There's no reason to care about heritable IQ gaps, and good reason to not publicly discuss them. E.g., in this comment. 

If one was to survey all the frontpage articles on Lesswrong over the last 6/9 months, how many turn on the heritable IQ gap? Very few, as far as I can tell. 

Showing I am wrong in this assessment (e.g., in a short post collecting 4-5 highly upvoted posts which shows how omitting heritable IQ from their world model has caused confusion or mistake), is more likely to succeed than introducing it as a new current of debate. 

Thanks, good comment. The quick low-effort version that doesn't require actually writing the posts is that without taking heritable IQ into account, I think you will be confused about:

  1. Various ways in which post-apartheid South Africa is a bad place to live.
  2. Why so many countries have market-dominant minorities.
  3. Why Israel is so good at defending itself even against far larger countries surrounding it (and the last few centuries of Jewish history more generally).
  4. Why the growth curves for East Asia and Africa looked so different over the last century.

One of the big issues with IQ discussions is that you end up taking the public's struggle with statistics which is already terrible even when they're trying and then add on a whole bunch of bad faith intention and discourse on top.

Like this is just anecdotal but the amount of people I've seen in this and similar discussion seemingly unable to recognize that averages aren't necessarily that meaningful to any individual experience is crazy. A group with an average of 95 IQ is still going to have lots of individuals within it that are over >100! If you tak... (read more)

but right now it doesn't feel like LW is a place where I can collaboratively make intellectual progress on this very important topic.

 

Looking at the post in question, I think this was basically true, but not for the reason you seem to imply. On a ultra short post about the decline of Western civilization using very high level historic hot takes, ~no first level comments cite non obvious historic facts. Instead, it's mostly vibes based rationalism. 

But to some extent, you do not seem to engage much with takes that do exhibit substance? E.g. there was one co... (read more)

I'd still be interested to see your response to the question I asked!

Like, more generally - if we're going to discuss unwelcome features of what is, I'd like to see us establish some sort of way to have some confidence that we're going at all a similar direction about what should be.

I had trouble figuring out which part of your post was intended to be the main question. I've left a few comments responding to various parts of it.

Re what outcomes I'm aiming for, honestly a lot of what motivates my thinking is how much I care about cooperation. I just expect that for cooperation to work at large scales and over the long term, you need to do a bunch of exclusion/separation at smaller scales.

I just expect that for cooperation to work at large scales and over the long term, you need to do a bunch of exclusion/separation at smaller scales.

Why not just talk about this instead?

I had trouble figuring out which part of your post was intended to be the main question. I've left a few comments responding to various parts of it.

Appreciated! I have a longer comment replying to your reply over there I'll send in about a day, after some editing to remove my signature word vomit style. It will be more specific than this comment.

a lot of what motivates my thinking is how much I care about cooperation

This is a promising thing to say! It slightly reduces my probability somewhat that your followup statement is quite as concerning as it sounds.

you need to do a bunch of exclusion/separation at smaller scales.

I could buy this literal statement, probably in a different form than you mean it. I've thought for a long time that, if we get an aligned overwhelming superintelligence and can worry about mundane things like this, then people who want human monocultures (eg, a culture with only 6-fingered people, to make up an example) may need to move to isolated locations to get their monocultures, though that would have difficulties relating to ensuring newly made kids can give informed consent. And to spell it out, policies like that in the presence of an overwhelmi... (read more)

This is particularly striking because I'm pretty confident he himself believes some version of this hypothesis! So he's basically calling himself a Nazi to defuse the discomfort of even raising a hypothesis this controversial (let alone endorsing it). This is objectively a very odd thing to do.

I think it's quite plausible that he believes that half of the difference is genetic and thus he does not hold the position that most of the difference is genetic. I think he probably justified it to himself by saying that the claim "most" is more extreme than what h... (read more)

By and large people have been respectful and polite (as per the high upvote count), but right now it doesn't feel like LW is a place where I can collaboratively make intellectual progress on this very important topic.

Where do you plan to write about politics, etc. then?

Twitter + my blog (mindthefuture.info).

But most of my collaboration on this stuff is via 1:1 discussions, reading groups, etc. I host a politics reading group which has been very productive for me (separate from the 21civ.com groups, which have also been interesting).

Yes, there is reticence about lots topics, not just intelligence differences. But the fact that some people are unwilling to fully engage with the implications of that, I guess as an explanation of problems happening in society, does not make that point more valid.

Mental blocks generally happen when people avoid thinking about bad scenarios, when there is discomfort (e.g. not having enough money for X, or a plan going bad, or a bridge breaking, or social pressure). I guess the trick you did is removing some bad scenario that allowed you to further explore ... (read more)

I was recently looking at the Astra Fellowship program, and found myself noticing skepticism that much value would come out of their "Strategy" category of research. In order to channel that skepticism productively, I want to talk a bit about how I think people might do valuable strategy research.

The most important thing is something like: trying to orient to the possibility of big changes in our ontologies going forward. What I mean by "strategy" is in significant part "ideas and plans that are robust to paradigm shifts".

Having said that, there's no way to do strategic thinking without relying on some part of your existing ontology. So the thing I'm advocating for is more like "keep track of the fields where paradigm shifts seem most plausible, and actually try to picture what it would be like for that part of your ontology to radically improve". Of course answering questions like these is extremely difficult because ontology improvements are anti-inductive: if you could picture them then you would have already improved your ontology.

One strategy for getting around that is to use analogies—e.g. "imagine a future where we understand social sciences as well as we currently understan... (read more)

To me there seem to be many examples of good impactful strategy research that don't introduce big new ontologies or go via illegibility:

  • initial case for AI takeover risk
  • simulation argument
  • vulnerable world hypothesis
  • stand args for long termism
  • argument that avoiding Extinction or existential risk is a tractable way to impact the long term
  • astronomical waste
  • highlighting the risk of human power grabs
  • Importance of using AI to upgrade to societal epistemics / coord
  • risk from a software only IE

I do also see examples of big contributions that are in the form of new ontologies like reframing superintelligence. But these seem less common to me.

Thanks for making this more concrete! I still disagree with you, but this list provides a good way of articulating why.

Of these, I think the initial case for AI takeover risk was the most impactful by far, and also a great example of introducing a new ontology (which includes concepts like AGI and superintelligence, the orthogonality thesis, instrumental convergence, recursive self-improvement, the nearest unblocked strategy problem, corrigibility, alignment, reward tampering, etc).

The simulation argument, by contrast, is an interesting example of an idea that is nominally very "big if true" but has actually had very little impact. And one reason why I think this is the case is because it doesn't really change our ontologies much. We add the concept of the universe being a simulation, but in order to understand what's inside the simulation we still use all our old concepts, and we have almost no new concepts that help us think about what's outside the simulation. (Off the top of my head, "ancestor simulation" is the only additional novel concept I can recall related to this hypothesis.)

Existential risk and longtermism fall somewhere in the middle. They are also "big if true", and t... (read more)

Thanks for articulating your view in such detail. (This was written with transcription software. Sorry if there are mistakes!)

AI risk:

When I articulate the case for AI takeover risk to people I know, I don't find the need to introduce them to new ontologies. I can just say that AI will be way smarter than humans. It will want things different from what humans want, and so it will want to seize power from us. 

 

But I think I agree that if you want to actually do technical work to reduce the risks, that it is useful to have new concepts that point o... (read more)

When I articulate the case for AI takeover risk to people I know, I don't find the need to introduce them to new ontologies.... But I think I agree that if you want to actually do technical work to reduce the risks, that it is useful to have new concepts that point out why the risk might arise.

It seems like one crux might be that I place much less value on "articulating a case" than you do. Or maybe another way of putting it is that you draw a cleaner boundary between "articulating a case" and "actually do technical work", whereas I think of them as pretty continuous when done well.

(Note that this also puts me in disagreement with many rationalists. Many rationalists treat "the case for high P(doom)" as a pretty reliable set of ideas, and then "alignment research" as something we're very confused about. Whereas from my perspective, these two things are intimately related—developing the concepts and ontology required for a robust case for high P(doom) would actually get us most of the way towards solving the alignment problem.)

Not quite sure how to justify my position here, but one intuition is something like "taking crucial considerations seriously". It really does seem that people... (read more)

One other thing is that I'd have guessed that the sign uncertainty of historical work on AI safety and AI governance is much more related to the inherent chaotic nature of social and political processes rather than a particular deficiency in our concepts for understanding them.

I'm sceptical that pure strategy research could remove that side uncertainty, and I wonder if it would take something like the ability to run loads of simulations of societies like ours.

inherent chaotic nature of social and political processes

Everything seems inherently chaotic until you understand it well! The motion of the planets across the sky seems very arbitrary until you understand Kepler's laws, and so on.

Re "simulations", the easiest way to build a simulation of something is to have a principled model/theory of it.

But things can be inherently chaotic too!

Agree it's unclear how much is inherent.

Thanks for this! 

 

I do agree that the history of crucial considerations provides a good reason to favour  'deep understanding'. 

 

I also agree that you plausibly need a much deeper understanding to get to above 90% on P(doom). But I don't think you need that to get to the action-relevant thresholds, which are much lower. 

 

I'd be interested in learning more about your power grab threat models, so let me know if and when you have something you want to share. And TBC I think you're right that in many scenarios it will not be ... (read more)

I'm no pro at the creative stuff but I've found that when I allow myself to have a fun zone where I just produce stuff whenever things appear it seems like I have generally better thoughts. (Similar to more procedural virtue ethics models that you have)

I spend at least 5-10 hours a week in this space and I think it is correlated with thinking that anything goes. I kind of have a separate evaluator system that I apply in other circumstances. No impact evaluation, just cool thoughts in this space. I also have a new saying that I like to follow "if it's fun i... (read more)

(though my sense is that Leverage was doing all three for most of the 2010s)

What leads you to think that Leverage was doing #1? 

Yeah, good question. Now that I reflect on it I don’t have a great reason. I think I had mentally cached their work on psychology as attempting to be general enough to apply to AIs too, but I don’t know if that’s accurate.

(Geoff’s orientation to philosophy of science feels related to some lines of thinking that led me to focus on agent foundations, but again that’s a pretty speculative connection.)

I think I had mentally cached their work on psychology as attempting to be general enough to apply to AIs too, but I don’t know if that’s accurate.

I think that some of them thought it was relevant to AI (eg at least one person was concerned about docs about Connection Theory being on the internet because it might yield AI capability advances), but that this was largely because they were extremely uninformed about what modern ML is. 

Huh, looks like the list of Astra mentors has changed significantly. In particular, all the Forethought people are off it now (maybe other changes I'm not tracking). Not sure if that reflects a substantive change or a PR optimization.

Conversely, when I encounter people who seem to me to have meaningfully pushed one of these frontiers

Can you give some examples of what you're thinking about here? 

There have been notable advances in cognitive science over the past 20 years, and those are pretty cool. 

But outside of that, I'm not sure what progress you're thinking about. There are definitely skilled practitioners of psychological arts re: emotional processing, but the domain is esoteric in the sense that their skill doesn't translate to third-person legible models that can be bui... (read more)

lol, it looks like I should have finished reading the sentence I'm responding to, before starting to write a comment, since you're making a similar point, re legibility. 

I'm still interested in who you think is making progress though, even if illegibly to most people.

Re politics, people like Yarvin and Land have been pushing the frontier pretty illegibly. Scott Alexander too. And Vassar is a central example I was thinking about. For others who are a bit less well-known see my curriculum—I like N. S. Lyons, Nathan Cofnas, Ben Landau-Taylor, etc.

Re emotional processing, I don’t have a great sense of the history here. But I feel like there’s been gradual development of stuff like internal family systems, ideal parent figure therapy, circling, jhana meditations, and various kinds of body work over the last few decades. Stu... (read more)

seems like a good place to ask something I’ve been meaning to ask you for a while: how familiar are you with the lineage that influenced Land? How much Deleuze, Spinoza, Frankfurt School, Lacan, Foucault, Gramsci, Agamben, Latour, Hegel, Negri, Heidegger, etc have you read?

I like some things about your thinking but often you state things as revelatory, or in this case appeal to pushing the frontier, in a way that makes it seem like you might not have spent much time with the influences of some of the thinkers who influence you. This isn’t a huge deal; attribution is really hard! But at present when you imply originality (or something weaker) in a particular thinker or idea, I tend to heavily down-weight that implication.

To be clear: I’m not intending to be at all censorious or credentialist, and I’m not endorsing any of these thinkers in particular. I’m concerned about things like:

  1. I’m missing something (because whatever point you made didn’t strike me as profound, while it clearly struck you as such), and you’re trying to signal that the reader may be missing something, and I’m down weighting that signal, and so continue to miss it.
  2. You’re sacrificing some credibility among audience
... (read more)

I appreciate this comment, both in its substance and how careful you've been to phrase it non-adversarially. The simple answer is that I am very unfamiliar with the lineage that influenced Land, and continental philosophy more generally. As far as my philosophy tutors at Oxford were concerned, that whole field might as well have not existed; and I never found my way into it independently.

I have been intending to look into several of the people you listed (and other related thinkers) more. Until I've done so, I may well say silly things due to ignorance of them, and am very happy to be corrected when I do so.

Having said that, your comment reads like the sort of response I'd expect if I'd made a claim like "Land is one of the most original thinkers ever" or "Land's thinking is unprecedented". By "Land has been pushing the frontier" I meant something much weaker, which is totally consistent with him being greatly influenced by the people you named (who also were pushing the frontier in their own eras).

To be more specific and object-level: in a healthy discipline, many people (e.g. most PhD students) will sync up on what the field as a whole considers to be open questions, and then try... (read more)

Thanks, this is very helpful! To be clear, I did take you to be saying the weak version, and I’d probably not contest use of even the strong version in Land’s particular case.

The question wasn’t directly formulated in response to anything in your comment, but has been lounging about quite a while, waiting for a not-entirely-disastrous moment in which to be asked, and one in which your were almost-but-not-quite seeming to do the kind of thing that concerns me seemed like a good candidate.

Likewise I would say that Yarvin is influenced by (roughly descending importance)

  1. Older reactionaries: Caryle, de Maistre
  2. Libertarians/Austrian schoolers: Rothbard, von Mises, Hoppe
  3. Italian elite theorists: Mosca, Pareto

Also Burnham

(COI note: I work at OpenAI. These are my personal views, though.)

My quick take on the "AI pause debate", framed in terms of two scenarios for how the AI safety community might evolve over the coming years:

  1. AI safety becomes the single community that's the most knowledgeable about cutting-edge ML systems. The smartest up-and-coming ML researchers find themselves constantly coming to AI safety spaces, because that's the place to go if you want to nerd out about the models. It feels like the early days of hacker culture. There's a constant flow of ideas and brainstorming in those spaces; the core alignment ideas are standard background knowledge for everyone there. There are hackathons where people build fun demos, and people figuring out ways of using AI to augment their research. Constant interactions with the models allows people to gain really good hands-on intuitions about how they work, which they leverage into doing great research that helps us actually understand them better. When the public ends up demanding regulation, there's a large pool of competent people who are broadly reasonable about the risks, and can slot into the relevant institutions and make them work well.
  2. AI sa
... (read more)

FYI I think this is worth fleshing out into a top level post (esp. given that it's 'Pause Debate' week). 

I'm not actually sure it needs much fleshing out. I think the main bit here that feels unjustified, or insufficiently-justified for the strength of the claim, is:

That's changed a bit lately, in part because a bunch of people seem to think that making technical progress on alignment is hopeless. I think this is just not an epistemically reasonable position to take: history is full of cases where people dramatically underestimated the growth of scientific knowledge, and its ability to solve big problems.

Have edited slightly to clarify that it was "leading experts" who dramatically underestimated it. I'm not really sure what else to say, though...

I think I basically agree with you Richard about the risks of falling into scenario 2, and think this is a wise comment, but I also think you are strawmanning the reason for the change -- it's not that people have come to think that making technical progress is hopeless (or even harder than it used to be!) it's rather that people have come to have shorter timelines, and so the probability that sufficient technical progress will be made in time has gone down, and the usefulness of calling for a pause has gone up. (e.g. if you think AGI is 15 years away, the... (read more)

I appreciated seeing the caricature of 1 presented at least, as a dream, it feels attainable, or like it might have been in some other timeline, but perhaps in ours too, for all I know.

I haven't yet read through them thoroughly, but these four papers by Oliver Richardson are pattern-matching to me as potentially very exciting theoretical work.

tl;dr: probabilistic dependency graphs (PDGs) are directed graphical models designed to be able to capture inconsistent beliefs (paper 1). The definition of inconsistency is a natural one which allows us to, for example, reframe the concept of "minimizing training loss" as "minimizing inconsistency" (paper 2). They provide an algorithm for inference in PDGs (paper 3) and an algorithm for learning via locally minimizing inconsistency which unifies several other algorithms (like the EM algorithm, message-passing, and generative adversarial training) (paper 4).

Oliver is an old friend of mine (which is how I found out about these papers) and a final-year PhD student at Cornell under Joe Halpern.

FWIW Oliver's presentation of (some fragment of) his work at ILIAD was my favorite of all the talks I attended at the conference.

Error-correcting codes work by running some algorithm to decode potentially-corrupted data. But what if the algorithm might also have been corrupted? One approach to dealing with this is triple modular redundancy, in which three copies of the algorithm each do the computation and take the majority vote on what the output should be. But this still creates a single point of failure—the part where the majority voting is implemented. Maybe this is fine if the corruption is random, because the voting algorithm can constitute a very small proportion of the total code. But I'm most interested in the case where the corruption happens adversarially—where the adversary would home in on the voting algorithm as the key thing to corrupt.

After a quick search, I can't find much work on this specific question. But I want to speculate on what such an "error-correcting algorithm" might look like. The idea of running many copies of it in parallel seems solid, so that it's hard to corrupt a majority at once. But there can't be a single voting algorithm (or any other kind of "overseer") between those copies and the output channel, because that overseer might itself be corrupted. Instead, you need the m... (read more)

I have some experience in the design of systems designed for high reliability and resistance to adversaries.  I feel like I’ve seen this kind of thinking before.

Your current line of thinking is at a stage I would call “pretheoretical noodling around.”  I don’t mean any disrespect; all design has to go through this stage.  But you’re not going to find any good references, or come to any conclusions, if you stay at this stage.  A next step is to settle on a model of what you want to get done, and what capabilities the adversaries have.  You need some bounds on the adversaries; otherwise nothing can work.  And of course you need some bounds on what the system does, and how reliably.  Once you’ve got this, you can either figure out how to do it, or prove that it can’t be done.

For example there are ways of designing hardware which is reliable on the assumption that at most N transistors are corrupt.

The problem of coming to agreement between a number of actors, some of whom are corrupt, is known as the Byzantine generals problem.  It is well studied, and you may find it interesting.

 I’m also interested in this topic, and I look forward to seeing where this line of thinking takes you.

A next step is to settle on a model of what you want to get done, and what capabilities the adversaries have.

Perhaps. The issue here is that I'm not so interested in any specific goal, but rather in facilitating emergent complexity. One analogy here is designing Conway's game of life: I expect that it wasn't a process of "pick the rules you want, then see what results from those" but also in part "pick what results you want, and then see what rules lead to that".

Re the Byzantine generals problem, see my reply to niplav below:

I believe (please correct me if

... (read more)

But you’re not going to find any good references, or come to any conclusions, if you stay at this stage.

I think he already came to some conclusions and you already gave some good references (which support some of the conclusions).

For example there are ways of designing hardware which is reliable on the assumption that at most N transistors are corrupt.

Do those methods have names or address problems which have names (like the Byzantine generals problem)?

This makes me think of "radiation-hardened" quines, quines that output themselves even after deleting any one character (or three):
https://github.com/mame/radiation-hardened-quine
https://codegolf.stackexchange.com/a/100785

I'm not familiar with the method used, but maybe it can be adapted to the case you are imagining? Maybe for simplicity only for the voting part of a redundant system.

This seems very relevant, thank you. Will check it out.

IIRC in Byzantine fault tolerance the assumption usually is that cryptographic primitives can be trusted and for at most nodes are corrupted. The corrupted nodes can output anything. E.g. the Dolev-Strong protocol only assumes (1) a known number of nodes, (2) a public key infrastructure and (3) synchronized communication. But I guess your thought is around someone corrupting a specific part of all nodes? Dolev-Strong assumes you run more rounds than there are corrupted nodes.

I guess your thought is around someone corrupting a specific part of all nodes?

No, I'm happy to stick with the standard assumption of limited amounts of corruption.

However, I believe (please correct me if I'm wrong) that Byzantine fault tolerance mostly thinks about cases where the nodes give separate outputs—e.g. in the Byzantine generals problem, the "output" of each node is whether it attacks or retreats. But I'm interested in cases where the nodes need to end up producing a "synthesis" output—i.e. there's a single output channel under joint control.

After a quick search, I can't find much work on this specific question. But I want to speculate on what such an "error-correcting algorithm" might look like.

Did you look in places discussing ledger tech and proof of stake? If you are curious to see what's been discussed, looking into that might be worth it. Information theory will mostly just focus on efficiency and practical trade-offs.

I'd expect the answer to not be apparent to an outsider, reading the literature, but I'd expect people who are good at designing those sorts of systems to be able to give you the answer quite easily if you ask.

Not exactly about adversarial error correction, but: there is a construction (Çapuni & Gács 2021) of a (class of) universal 1-tape (!!) Turing machine that can perform arbitrarily long computation subject to random noise in the per-step action. Despite the non-adversarial noise model, naive majority error correction (or at least their construction of it) only fixes bounded & local error bursts - meaning it doesn't work in the general case, because even though majority vote reduces error probability, the effective error rate is still positive, so so... (read more)

That is very cool, thank you! Will check it out.

For this we need a mechanism such that the maintenance of the mechanism is a schelling point.  Specifically, the mechanism at T+1 should reward agents for actions at time T that reinforce the mechanism itself (in particular the actions are distributed). The incentive raises the probability of the mechanism being actualized at T+1, which in turn raises the "weight" of the reward offered by the mechanism at T+1, creating a self-fulfilling prophecy.

 

"Merging" forces parallelism back into sequential structures, which is why most blockchains are slow.... (read more)

Just spitballing, but maybe you could incorporate some notion of resource consumption, like in linear logic. You could have a system where the copies have to "feed" on some resource in order to stay active, and data corruption inhibits a copy's ability to "feed."

There may be something to find in how distributed systems do leader election, and the leader can then be a majority trusted overseer. Note that I don't really understand leader election.

Very interesting problem to be thinking about. The problem with a UTM as a computation model is that it bakes non-redundancy in, there's a single instruction pointer.

In reality, computers are implemented in different spatial locations and can run in parallel. A better model for this is a cellular automaton, where computers are located somewhere and their circuit for outputs is also located somewhere. Some automata (e.g. game of life) are Turing-complete, so you can just use that.

Corruption could be exogenously flipping cells in ways that violate the automa... (read more)

if there are sufficiently many copies, it becomes impossible to corrupt them all at once.

So I don't love this model because escaping corruption is 'too easy'.

I really like the cellular automaton model. But I don't think it makes escaping corruption easy! Even if most of the copies are non-corrupt, the question is how you can take a "vote" of the corrupt vs non-corrupt copies without making the voting mechanism itself be easily corrupted. That's why I was talking about the non-corrupt copies needing to "overpower" the corrupt copies above.

No I agree with that. I thought the tree design already involved weighted sums overpowering each other, but I think that was premature.

Thinking more about the cellular automaton stuff: okay, so Game of Life is Turing complete. But the question is whether we can pin down properties that GoL has that Turing machines don't have.

I have a vague recollection that parallel Turing Machines are a thing, but this paper claims that the actual formalisms are disappointing. One nice thing about Game of Life is that the way that different programs interact internally (via game of life physics) is also how they interact with each other. Whereas any multi-tape Turing Machine (even one with clever rules a... (read more)

But it feels like you'd need to demonstrate this with some construction that's actually adversarially robust, which seems difficult.

I agree it's kind of difficult.

Have you seen Nicholas Carlini's Game of Life series? It starts by building up logical gates up to a microprocessor that factors 15 in to 3 x 5.

Depending on the adversarial robustness model (e.g. every second the adversary can make 1 square behave the opposite of lawfully), it might be possible to make robust logic gates and circuits. In fact the existing circuits are a little robust already -... (read more)

When I wrote about AGI and lock-in I looked into error-correcting computation a bit. I liked the papers von Neumann, 1952 and Pippenger, 1990.

Apparently at the time I wrote:

In order to do error-correcting computation, you also need a way to prevent errors from accumulating over many serial manipulations. The simplest way to do this is again to use redundancy: break the computation into multiple parts, perform each part multiple times on different pieces of hardware, and use the most common output from one part as input to the next part. [Fn: Of course, the

... (read more)

Here is a broad sketch of how I'd like AI governance to go. I've written this in the form of a "plan" but it's not really a sequential plan, more like a list of the most important things to promote.

  1. Identify mechanisms by which the US government could exert control over the most advanced AI systems without strongly concentrating power. For example, how could the government embed observers within major AI labs who report to a central regulatory organization, without that regulatory organization having strong incentives and ability to use their power against their political opponents?
    1. In practice I expect this will involve empowering US elected officials (e.g. via greater transparency) to monitor and object to misbehavior by the executive branch.
  2. Create common knowledge between the US and China that the development of increasingly powerful AI will magnify their own internal conflicts (and empower rogue states) disproportionately more than it empowers them against each other. So instead of a race to world domination, in practice they will face a "race to stay standing".
    1. Rogue states will be empowered because human lives will be increasingly fragile in the face of AI-designed WMDs. This me
... (read more)

On (4), I don't I understand why having a scale-free theory of intelligent agency would substantially help with making an alignment target. (Or why this is even that related. How things tend to be doesn't necessarily make them a good target.)

Ooops, good catch. It should have linked to this: https://www.lesswrong.com/posts/FuGfR3jL3sw6r8kB4/richard-ngo-s-shortform?commentId=W9N9tTbYSBzM9FvWh (and I've changed the link now).

Just read Bostrom's Deep Utopia (though not too carefully). The book is structured with about half being transcripts of fictional lectures given by Bostrom at Oxford, about a quarter being stories about various woodland creatures striving to build a utopia, and another quarter being various other vignettes and framing stories.

Overall, I was a bit disappointed. The lecture transcripts touch on some interesting ideas, but Bostrom's style is generally one which tries to classify and taxonimize, rather than characterize (e.g. he has a long section trying to analyze the nature of boredom). I think this doesn't work very well when describing possible utopias, because they'll be so different from today that it's hard to extrapolate many of our concepts to that point, and also because the hard part is making it viscerally compelling.

The stories and vignettes are somewhat esoteric; it's hard to extract straightforward lessons from them. My favorite was a story called The Exaltation of ThermoRex, about an industrialist who left his fortune to the benefit of his portable room heater, leading to a group of trustees spending many millions of dollars trying to figure out (and implement) what it means to "benefit" a room heater.

Tangentially related (spoilers for Worth the Candle):

I think it'd be hard to do a better cohesive depiction of Utopia than the end of Worth the Candle by A Wales. I mean, I hope someone does do it, I just think it'll be challenging to do!

Strong agree, also I spoiler-texted it, hope you don't mind.

Strong disagree. Often in Alexander Wales' stories, I find he takes the rationalist/EA mindset so far that it reads more like a reductio. Examples include the story about EA superman, and

the "aligned" skynet

at the end of Branches on the Tree of Time.

I have a similar feeling about the end of Worth the Candle.

Any opinions on how it compares to Fun Theory? (Though that's less about all of utopia, it is still a significant part)

If you haven't read CEV, I strongly recommend doing so. It resolved some of my confusions about utopia that were unresolved even after reading the Fun Theory sequence.

Specifically, I had an aversion to the idea of being in a utopia because "what's the point, you'll have everything you want". The concrete pictures that Eliezer gestures at in the CEV document do engage with this confusion, and gesture at the idea that we can have a utopia where the AI does not simply make things easy for us, but perhaps just puts guardrails onto our reality, such that we don... (read more)

I dropped the book within the first chapter. For one, I found the way Bostrom opened the chapter as very defensive and self-conscious. I imagine that even Yudkowsky wouldn't start a hypothetical 2025 book with fictional characters caricaturing him. Next, I felt like I didn't really know what the book was covering in terms of subject matter, and I didn't feel convinced it was interesting enough to continue the meandering path Nick Bostrom seem to have laid out before me.

Eliezer's CEV document and the Fun Theory sequence were significantly more pleasant experiences, based on my memory.

Suppose we get to specify, by magic, a list of techniques that AGIs won't be able to use to take over the world. How long does that list need to be before it makes a significant dent in the overall probability of xrisk?

I used to think of "AGI designs self-replicating nanotech" mainly as an illustration of a broad class of takeover scenarios. But upon further thought, nanotech feels like a pretty central element of many takeover scenarios - you actually do need physical actuators to do many things, and the robots we might build in the foreseeable future are nowhere near what's necessary for maintaining a civilisation. So how much time might it buy us if AGIs couldn't use nanotech at all?

Well, not very much if human minds are still an attack vector - the point where we'd have effectively lost is when we can no longer make our own decisions. Okay, so rule out brainwashing/hyper-persuasion too. What else is there? The three most salient: military power, political/cultural power, economic power.

Is this all just a hypothetical exercise? I'm not sure. Designing self-replicating nanotech capable of replacing all other human tech seems really hard; it's pretty plausible to me that the world is crazy in a bunch of other ways by the time we reach that capability. And so if we can block off a couple of the easier routes to power, that might actually buy useful time.

Firstly, I think it kind of depends. What exactly does blocking the AI from designing nanotech mean? Is the AI allowed to use genetic engineering? Is it allowed to use selective breeding? Elephants genetically engineered to be really good at instruction following?

I mean I think macroscopic self replicating robotics is probably possible, and the AGI can probably bootstrap that from current robotics fairly quickly. 

You rule out any hyper-persuasion. How much regular persuasion is the AI allowed to do. After all, if you are buying something online, (from... (read more)

I agree. Self-replicating nanotech seems to be likely a much harder problem than for language models to get good enough actors to get political, cultural, and economic power. 

To the extent that an AGI can make political and economic decisions that are of higher quality than human decisions, there's also a lot of pressure for humans to delegate those decisions to AGI. Organizations that delegate those decisions to AGI will outcompete those who don't.

Another general technique: attacks on computing systems. (Both takeover / subversion (dropping an email going 'um this is a problem') and destruction (destroy the US power infrastructure using Russian-language programs)).

These don't tend to be sufficient in and of themselves, but are "classic" stepping-stones to e.g. buy time for an AI while it ramps up.

The last three options you mentioned are all things that happen over relatively slow timescales, if your goal is to completely destroy humanity. The single exception to this is nuclear war, but if you’re correct, then we can reduce the problem to non-proliferation, which is at least in theory solvable.

Epistemic status: wild speculation

There's an interesting line in the old MIRI paper on Definability of Truth: "Tarski’s result on the undefinability of truth is in some sense an artifact of the infinite precision demanded by reasoning about complete certainty." This feels vaguely analogous here to the uncertainty principle in quantum mechanics, which also prevents infinite-precision knowledge. (Has anyone discussed this before?)

The connection jumped to my attention because I've also been thinking lately about the concept of causation in physical and logical universes. Intuitively speaking, the concept of Pearl's do-operator feels pretty sensible in our physical universe because there's a bounded lightcone which an event can affect—so that when we hypothesize taking an action, we can mostly ignore the implications for things in the past, or for simultaneous actions by other agents. (See this old post by Yudkowsky for more.)

FDT generalizes the do-operator to the realm of logic, which doesn't have either timelike or spacelike separation by default. There's no privileged direction such that some logical facts affect others but aren't affected by them. And there's no notion of "distance... (read more)

Logical time makes sense as time within a particular process of computation (possibly of logical inference). Different computations have their own internal directions of logical time, and space-like subcollections of (partial) states that aren't comparable in logical time.

When a computation is reasoning about another computation, it will naturally tend to arrive at facts about the other computation in an order related to that computation's own logical time. Of course, leaps in reasoning are possible, predicting distant future without reasoning in detail ab... (read more)

There's an interesting line in the old MIRI paper on Definability of Truth: "Tarski’s result on the undefinability of truth is in some sense an artifact of the infinite precision demanded by reasoning about complete certainty." 

Not to my knowledge (though I haven't searched), but it's funny that Adrià and Abram found a solution by kind of imposing even more precision via adding hyperreal numbers: https://www.lesswrong.com/posts/j3Cv94eEZTDFiLREW/infinitesimally-false


An alternative idea (inspired by some of davidad's thinking) is that intelligent minds migh

... (read more)

Regarding absolute truth, I think Hume got it wrong. Conditional statements can be true - while of course remaining conditional. {If I'm not a brain in a tank, or in some other way being systematically deceived, then I live on the 7th floor in my apartment building.} Unconditionally true - because the universal uncertainties are encompassed, and confined in a phrase. Generalizing, we can make a point of isolating uncertainties and creating structures of well-defined concepts that permit certainty about inferred propositions because they come with enumerations of presumptions. I do this with my AI, and can trust the output.

Deutsch's concept of a non-reductionist theory of everything, but he doesn't (to my knowledge) point specifically at the idea of parallelism between these two domains of inquiry

my understanding was that Deutsch proposed a unified theory of [quantum field theory + decision theory] in order to remove probability as a fundamental concept haunting the 2 separate theories, but there would be no parallelism between the 2 former domains of inquiry, "just" the unification into a different domain that wouldn't apply to the "old" way of thinking, right? I don't real... (read more)

A different Deutsch theory, but it's not really a theory, more like his gesture towards what a theory of everything should look like. More here.

A possible way to convert money to progress on alignment: offering a large (recurring) prize for the most interesting failures found in the behavior of any (sufficiently-advanced) model. Right now I think it's very hard to find failures which will actually cause big real-world harms, but you might find failures in a way which uncovers useful methodologies for the future, or at least train a bunch of people to get much better at red-teaming.

(For existing models, it might be more productive to ask for "surprising behavior" rather than "failures" per se, since I think almost all current failures are relatively uninteresting. Idk how to avoid inspiring capabilities work, though... but maybe understanding models better is robustly good enough to outweight that?)

I like this. Would this have to be publicly available models? Seems kind of hard to do for private models.

What kind of access might be needed to private models? Could there be a secure multi-party computation approach that is sufficient?

Ideas for defining “surprising”? If we’re trying to create a real incentive, people will want to understand the resolution criteria.

Here's a (messy, haphazard) list of ways a group of idealized agents could merge into a single agent:

Proposal 1: they merge into an agent which maximizes a weighted sum of their utilities. They decide on the weights using some bargaining solution.

Objection 1: this is not Pareto-optimal in the case where the starting agents have different beliefs. In that case we want:

Proposal 2: they merge into an agent which maximizes a weighted sum of their utilities, where those weights are originally set by bargaining but evolve over time depending on how accurately each original agent predicted the future.

Objection 2: this loses out on possible gains from acausal trade. E.g. if a paperclip-maximizer finds itself in a universe where it's hard to make paperclips but easy to make staples, it'd like to be able to give resources to staple-maximizers in exchange for them building more paperclips in universes where that's easier. This requires a kind of updateless decision theory:

Proposal 3: they merge into an agent which maximizes a weighted sum of their utilities (with those weights evolving over time), where the weights are set by bargaining subject to the constraint that each agent obeys commitme... (read more)

1111

Nice!

Proposal 4: same as proposal 3 but each agent also obeys commitments that they would have made from behind a veil of ignorance where they didn't yet know who they were or what their values were. From that position, they wouldn't have wanted to do future destructive commitment races.

I don't think this solves Commitment Races in general, because of two different considerations:

  1. Trivially, I can say that you still have the problem when everyone needs to bootstrap a Schelling veil of ignorance.
  2. Less trivially, even behind the most simple/Schelling veils of
... (read more)
  1. Trivially, I can say that you still have the problem when everyone needs to bootstrap a Schelling veil of ignorance.

I don't understand your point here, explain?

even behind the most simple/Schelling veils of ignorance, I find it likely that hawkish commitments are incentivized. For example, the veil might say that you might be Powerful agent A, or Weak agent B, and if some Powerful agents have weird enough utilities (and this seems likely in a big pool of agents), hawkishly committing in case you are A will be a net-positive bet.

This seems to be claiming th... (read more)

I don't understand your point here, explain?

Say there are 5 different veils of ignorance (priors) that most minds consider Schelling (you could try to argue there will be exactly one, but I don't see why).

If everyone simply accepted exactly the same one, then yes, lots of nice things would happen and you wouldn't get catastrophically inefficient conflict.

But every one of these 5 priors will have different outcomes when it is implemented by everyone. For example, maybe in prior 3 agent A is slightly better off and agent B is slightly worse off.

So you need t... (read more)

I may be missing background concepts, but I don't see how Proposal 5 is really responding to Objection 4.

It seems to me that agent's strategy in the limit will either be null action or evolution-dictated action, not sure which. That is, "in universe where it's easy to do A the agent will choose to do A" somewhat implies "according to how easy it is for agent doing A to gain more optimization power, actions will be chosen" which is essentially evolution.

(I expect that Scott, Abram or some others have already pointed this out, but somehow this clicked for me only recently. Pointers to existing discussions appreciated.)

A Bayesian update can be seen as a special case of a prediction market resolution.

Specifically, a Bayesian update is the case where each "hypothesis" has bet all their wealth across some combination of outcomes, and then the pot is winner-takes-all (or split proportionally when there are multiple winners).

The problem with Bayesianism is then obvious: what happens when there are no winners? Your epistemology is "bankrupt", the money vanishes into the ether, and bets on future propositions are undefined.

So why would a hypothesis go all-in like that? Well, that's actually the correct "cooperative" strategy in a setting where you're certain that at least one of them is exactly correct.

To generalize Bayesianism, we want to instead talk about what the right "cooperative" strategy is when a) you don't think any of them are exactly correct, and b) when each hypothesis has goals too, not just beliefs.

Yep, see this paper - Bayesian updating is the same as having all the hypotheses in your head be agents in a prediction market using the Kelly criterion, where their inital wealths are your prior probabilities on those hypotheses, the market price of some observation is your marginal probability over that observation, and the wealths after the bets resolve are the posterior distributions.

To generalize Bayesianism, we want to instead talk about what the right "cooperative" strategy is when a) you don't think any of them are exactly correct, and b) when each hypothesis has goals too, not just beliefs.

Unclear to me how (b) connects to the rest of this post. Is it about each hypothesis being cautious not to bet all of their wealth, because they care about other stuff than winning the market?

The most obvious/naive/hacky solution is something like sub-probability (adds up to ≤1, so the truth might lie beyond your hypothesis space) with Jeffrey u... (read more)

Indeed in algorithmic information theory, the lower semicomputable semimeasures are an example of "subprobabilities." Much has been written about updating in this context. 

A short complaint (which I hope to expand upon at some later point): there are a lot of definitions floating around which refer to outcomes rather than processes. In most cases I think that the corresponding concepts would be much better understood if we worked in terms of process definitions.

Some examples: Legg's definition of intelligence; Karnofsky's definition of "transformative AI"; Critch and Krueger's definition of misalignment (from ARCHES).

Sure, these definitions pin down what you're talking about more clearly - but that comes at the cost of understanding how and why it might come about.

E.g. when we hypothesise that AGI will be built, we know roughly what the key variables are. Whereas transformative AI could refer to all sorts of things, and what counts as transformative could depend on many different political, economic, and societal factors.

If we do not fully understand the mechanism of (e.g. human) intelligence, isn't referring to the outcome preferable to a made-up story about the process?

(Of course, it would be even better if we understood the process and then referred to it.)

Do you think that these are mutually exclusive, or something like that? I've always been confused by what I take to be the position in this shortform, that defining the outcomes makes it somehow harder to define the process. Sure, you can define a process without defining an outcome (i.e. writing a program or training an NN), but since what we are confused about is what we even want at the end, for me that's the priority. And doing so would help searching for processes leading to this outcome.

That being said, if you point is that defining outcomes isn't enough, in that we also need to define/deconfuse/study the processes leading to these outcomes, then I agree with that.

I recently had a very interesting conversation about master morality and slave morality, inspired by the recent AstralCodexTen posts.

The position I eventually landed on was:

  1. Empirically, it seems like the world is not improved the most by people whose primary motivation is helping others, but rather by people whose primary motivation is achieving something amazing. If this is true, that's a strong argument against slave morality.
  2. The defensibility of morality as the pursuit of greatness depends on how sophisticated our cultural conceptions of greatness are. Unfortunately we may be in a vicious spiral where we're too entrenched in slave morality to admire great people, which makes it harder to become great, which gives us fewer people to admire, which... By contrast, I picture past generations as being in a constant aspirational dialogue about what counts as greatness—e.g. defining concepts like honor, Aristotelean magnanimity ("greatness of soul"), etc.
  3. I think of master morality as a variant of virtue ethics which is particularly well-adapted to domains which have heavy positive tails—entrepreneurship, for example. However, in domains which have heavy negative tails, the pursuit of g
... (read more)

If the following correlations are true, then the opposite may be true (slave morality being better for improving the world through history):

  • Improving the world being strongly correlated with economic growth (this is probably less true when X-risk are significant)
  • Economic growth being strongly correlated with Entrepreneurship incentives (property rights, autonomy, fairness, meritocracy, low rents)
  • Master morality being strongly correlated with acquiring power and thus decreasing the power of others and decreasing their entrepreneurship incentives

That all sounds very plausible. But isn't this all mostly relevant before AGI is a possibility? That would be a heavy negative tail risk, in which people motivated to "do great things" are quite prone to get us all killed. Should we survive that risk, progress probably mostly won't be driven by humans, so humans doing great things will barely count. If humans are actually still in charge when we hit ASI, it seems like doing great things with them will probably still have large tail risks (inter-ASI wars).

Right? Or do you see it differently?

It's a fascinating empirical claim that sounds right now that I hear it.

AGI is heavy-tailed in both directions I think. I don't think we get utopias by default even without misalignment, since governance of AGI is so complicated.

Re: your point #2, there is another potential spiral where abstract concepts of "greatness" are increasingly defined in a hostile and negative way by partisans of slave morality.  This might make it harder to have that "aspirational dialogue about what counts as greatness", as it gets increasingly difficult for ordinary people to even conceptualize a good version of greatness worth aspiring to.  ("Why would I want to become an entrepeneur and found a company?  Wouldn't that make me an evil big-corporation CEO, which has a whiff of the same f... (read more)

The crucial heuristic I apply when evaluating AI safety research directions is: could we have used this research to make humans safe, if we were supervising the human evolutionary process? And if not, do we have a compelling story for why it'll be easier to apply to AIs than to humans?

Sometimes this might be too strict a criterion, but I think in general it's very valuable in catching vague or unfounded assumptions about AI development.

By making human safe, do you mean with regard to evolution's objective?

No. I meant: suppose we were rerunning a simulation of evolution, but can modify some parts of it (e.g. evolution's objective). How do we ensure that whatever intelligent species comes out of it is safe in the same ways we want AGIs to be safe?

(You could also think of this as: how could some aliens overseeing human evolution have made humans safe by those aliens' standards of safety? But this is a bit trickier to think about because we don't know what their standards are. Although presumably current humans, being quite aggressive and having unbounded goals, wouldn't meet them).

Okay, thanks. Could you give me an example of a research direction that passes this test? The thing I have in mind right now is pretty much everything that backchain to local search, but maybe that's not the way you think about it.

So I think Debate is probably the best example of something that makes a lot of sense when applied to humans, to the point where they're doing human experiments on it already.

But this heuristic is actually a reason why I'm pretty pessimistic about most safety research directions.

So I've been thinking about this for a while, and I think I disagree with what I understand of your perspective. Which might obviously mean I misunderstand your perspective.

What I think I understand is that you judge safety research directions based on how well they could work on an evolutionary process like the one that created humans. But for me, the most promising approach to AGI is based on local search, which differs a bit from evolutionary process. I don't really see a reason to consider evolutionary processes instead of local search, and even then, ... (read more)

One source of our disagreement: I would describe evolution as a type of local search. The difference is that it's local with respect to the parameters of a whole population, rather than an individual agent. So this does introduce some disanalogies, but not particularly significant ones (to my mind). I don't think it would make much difference to my heuristic if we imagined that humans had evolved via gradient descent over our genes instead.

In other words, I like the heuristic of backchaining to local search, and I think of it as a subset of my heuristic. T... (read more)

So if I try to summarize your position, it's something like: backchain to local search for simple and single-AI cases, and then think about aligning humans for the scaled and multi-agents version? That makes much more sense, thanks!

I also definitely see why your full heuristic doesn't feel immediately useful to me: because I mostly focus on the simple and single-AI case. But I've been thinking more and more (in part thanks to your writing) that I should allocate more thinking time to the more general case. I hope your heuristic will help me there.

Cool, glad to hear it. I'd clarify the summary slightly: I think all safety techniques should include at least a rough intuition for why they'll work in the scaled-up version, even when current work on them only applies them to simple AIs. (Perhaps this was implicit in your summary already, I'm not sure.)

The idea that maximally-coherent agents look like squiggle-maximizers raises the question: what would it look like for humans to become maximally coherent?

One answer, which Yudkowsky gives here, is that conscious experiences are just a "weird and more abstract and complicated pattern that matter can be squiggled into".

But that seems to be in tension with another claim he makes, that there's no way for one agent's conscious experiences to become "more real" except at the expense of other conscious agents—a claim which, according to him, motivates average utilitarianism across the multiverse.

Clearly a squiggle-maximizer would not be an average squigglean. So what's the disanalogy here? It seems like @Eliezer Yudkowsky is basically using SSA, but comparing between possible multiverses—i.e. when facing the choice between creating agent A or not, you look at the set of As in the multiverse where you decided yes, and compare it to the set of As in the multiverse where you decided no, and (if you're deciding for the good of A) you pick whichever one gives A a better time on average.

Yudkowsky has written before (can't find the link) that he takes this approach because alternatives would en... (read more)

what would it look like for humans to become maximally coherent [agents]?

In your comments, you focus on issues of identity - who are "you", given the possibility of copies, inexact counterparts in other worlds, and so on. But I would have thought that the fundamental problem here is, how to make a coherent agent out of an agent with preferences that are inconsistent over time, an agent with competing desires and no definite procedure for deciding which desire has priority, and so on, i.e. problems that exist even when there is no additional problem of identity. 

Clearly a squiggle-maximizer would not be an average squigglean

Why??? Being expected squiggle maximizer literally means that you implement policy that produces maximum average number of squiggles across the multiverse. 

The "average" is interpreted with respect to quality. Imagine that your only option is to create low-quality squiggles, or not to do so. In isolation, you'd prefer to produce them than not to produce them. But then you find out that the rest of the multiverse is full of high-quality squiggles. Do you still produce the low-quality squiggles? A total squigglean would; an average squigglean wouldn't.

It depends upon whether the maximizer considers its corner of the multiverse to be currently measurable by squiggle quality, or to be omitted from squiggle calculations at all. In principle these are far from the only options as utility functions can be arbitrarily complex, but exploring just two may be okay so long as we remember that we're only talking about 2 out of infinity, not 2 out of 2.

An average multiversal squigglean that considers the current universe to be at zero or negative squiggle quality will make the low quality squiggles in order to redu... (read more)

By "squiggle maximizer" I mean exactly "maximizer of number of physical objects such that function is_squiggle returns True on CIF-file of their structure".

We can have different objects of value. Like, you can value "probability that if object in multiverse is a squiggle, it's high-quality". Here yes, you shouldn't create additional low-quality squiggles. But I don't see anything incoherent here, it's just different utility function?

I just discovered that the psychoanalyst Jacques Lacan coined a concept of "logical time", which I had previously only associated with this post. Moreover, he introduces the concept via an example of people in a logic puzzle recursively reasoning about each other, which suggests that he's trying to point to a similar thing as Demski.

However, Lacan's writing is sufficiently impenetrable that I haven't easily been able to tell what he specifically means by "logical time", nor whether there's much value in trying to understand his thinking on the topic.

I am enthused about Richard reading more mid-twentieth-century European philosophy. I think it's unlikely there's much value in approaching Lacan specifically, although this depends on how much context you happen to already have / pick up without specifically targeting that work toward 'preparing to read Lacan'. 

Few reasons:

  1. Heavy use of word play that doesn't translate well
  2. Somewhat confused deployment of mathematical concepts
  3. Deliberate embrace of paradox
  4. Some confusion in later readers of Lacan between literary, cultural, and clinical applications of h
... (read more)

Consider making a curriculum (along the lines of AGI safety fundamentals or 21civ.com) aimed at smart, open-minded people with little relevant background. I doubt I'm the only one around here who would be interested in an accessible way to start reading good mid-twentieth-century European philosophy. Also in principle you can create a rough curriculum pretty quickly (e.g. an hour for a first draft) although if you're anything like me you'll then be sucked into spending a huge amount more time on it.

Fraught idea for reasons I’m struggling to articulate; thank you for the encouragement!

Probably the easiest "honeypot" is just making it relatively easy to tamper with the reward signal. Reward tampering is useful as a honeypot because it has no bad real-world consequences, but could be arbitrarily tempting for policies that have learned a goal that's anything like "get more reward" (especially if we precommit to letting them have high reward for a significant amount of time after tampering, rather than immediately reverting).

You don't want it to be relatively easy to an outside force. Otherwise they can lead it to do as they please, and writing weird behaviour off as 'oh, it's changed our rewards, reset it again', poses some risk.

Here is the broad technical plan that I am pursuing with most of my time (with my AI governance agenda taking up most of my remaining time):

  1. Mathematically characterize a scale-free theory of intelligent agency which describes intelligent agents in terms of interactions between their subagents.
    1. A successful version of this theory will retrodict phenomena like the Waluigi effect, solve theoretical problems like the five-and-ten problem, and make new high-level predictions about AI behavior.
  2. Identify subagents (and subsubagents, and so on) within neural networks by searching their weights and activations for the patterns of interactions between subagents that this theory predicts.
    1. A helpful analogy is how Burns et al. (2022) search for beliefs inside neural networks based on the patterns that probability theory predicts. However, I'm not wedded to any particular search methodology.
  3. Characterize the behaviors associated with each subagent to build up "maps" of the motivational systems of the most advanced AI systems.
    1. This would ideally give you explanations of AI behavior that scales in quality based on how much effort you put in. E.g. you might be able to predict 80% of the variance in an
... (read more)

For steps 2-4, I kinda expect current neural nets to be kludgy messes, and so not really have the nice subagent structure (even if you do step 1 well enough to have a thing to look for). 

I'm also fairly pessimistic about step 1, but would be very excited to know what preliminary work here looks like.

For steps 2-4, I kinda expect current neural nets to be kludgy messes, and so not really have the nice subagent structure

If, as a system comes to ever-better approximate a powerful agent, there's actual convergence towards this type of hierarchical structure, I expect you'd see something clearly distinct from noise even in the current LLMs. Intuition pump/proof-of-concept: the diagram comparing a theoretical prediction with empirical observations here.

Indeed, I think it's one of the main promises of "top-down" agent foundations research. The applicability ... (read more)

Interesting! Two questions:

  • What about the 5-and-10 problem makes it particularly relevant/interesting here? What would a 'solution' entail?
  • How far are you planning to build empirical cases, model them, and generalise from below, versus trying to extend pure mathematical frameworks like geometric rationality? Or are there other major angles of attack you're considering?
  1. Consider the version of the 5-and-10 problem in which one subagent is assigned to calculate U | take 5, and another calculates U | take 10. The overall agent solves the 5-and-10 problem iff the subagents reason about each other in the "right ways", or have the right type of relationship to each other. What that specifically means seems like the sort of question that a scale-free theory of intelligent agency might be able to answer.
  2. I'm mostly trying to extend pure mathematical frameworks (particularly active inference and a cluster of ideas related to geometric rationality, including picoeconomics and ergodicity economics).

Hypothesis: there's a way of formalizing the notion of "empowerment" such that an AI with the goal of empowering humans would be corrigible.

This is not straightforward, because an AI that simply maximized human POWER (as defined by Turner et al.) wouldn't ever let the humans spend that power. Intuitively, though, there's a sense in which a human who can never spend their power doesn't actually have any power. Is there a way of formalizing that intuition?

The direction that seems most promising is in terms of counterfactuals (or, alternatively, Pearl's do-calculus). Define the power of a human with respect to a distribution of goals G as the average ability of a human to achieve their goal if they'd had a goal sampled from G (alternatively: under an intervention that changed their goal to one sampled from G). Then an AI with a policy of never letting humans spend their resources would result in humans having low power. Instead, a human-power-maximizing AI would need to balance between letting humans pursue their goals, and preventing humans from doing self-destructive actions. The exact balance would depend on G, but one could hope that it's not very sensitive to the precise definiti... (read more)

There's also the problem of: what do you mean by "the human"? If you make an empowerment calculus that works for humans who are atomic & ideal agents, it probably breaks once you get a superintelligence who can likely mind-hack you into yourself valuing only power. It never forces you to abstain from giving up power, since if you're perfectly capable of making different decisions, but you just don't.

Another problem, which I like to think of as the "control panel of the universe" problem, is where the AI gives you the "control panel of the universe", bu... (read more)

Such that you can technically do anything you want--you have maximal power/empowerment--but the super-majority of buttons and button combinations you are likely to push result in increasing the number of paperclips.

I think any model of a rational agent needs to incorporate the fact that they're not arbitrarily intelligent, otherwise none of their actions make sense. So I'm not too worried about this.

If you make an empowerment calculus that works for humans who are atomic & ideal agents, it probably breaks once you get a superintelligence who can likely

... (read more)

You can think of this as a way of getting around the problem of fully updated deference, because the AI is choosing a policy based on what that policy would have done in the full range of hypothetical situations, and so it never updates away from considering any given goal. The cost, of course, is that we don't know how to actually pin down these hypotheticals.

Shannon’s information theory tells us about how much information is sent from a sender to a receiver via a channel. But the implications of that information might be very different depending on how much the sender trusts the receiver, in a way which seems like it might point to some interesting theoretical concepts.

Imagine you’ve just received a message M. There are two possibilities for who it came from: your best friend, or your worst enemy. The message might contain an equal amount of information either way—however, you’ll react to it very differently. In the former case, you want to propagate the contents of the message directly into your world-model: whatever your friend says, you fully believe.

By contrast, you definitely don’t want to just believe what your enemy tells you. Nor should you actively disbelieve it, though, otherwise they could still predictably mislead you. Instead, the main update you want to make is just “my enemy wants me to see the particular string of characters that constitutes message M”.

Another way of putting this: in the former case you update your world-model by conditioning on the semantics of the message, whereas in the latter case you update your wo... (read more)

Another way of putting this: in the former case you update your world-model by conditioning on the semantics of the message, whereas in the latter case you update your world-model by conditioning on the syntax of the message. Why? Because you don’t trust your enemy enough for communication with them to support semantic meaning.

I don't think that's right. The syntax of the message doesn't seem like a relevant subvariable. Saying "update on the syntax" feels like saying "update on the morphemes".

It seems to be more about which simulacrum level you apply as a wrapper to the semantic content. Suppose the message is X. For a friend, you apply level 1 and obtain "My friend Alice honestly/truth-seekingly believes X.". For an adversary, you apply level 2 and obtain "My enemy Bob wants me to believe X.".

In the God example, you don't apply anything; you take the unwrapped/raw semantics at face value (equivalently, apply a trivial wrapper). In the devil example, you apply an impenetrable wrapper that shields your mind from the message. Maybe there's some interesting structure/taxonomy, within which all of those sorts of semantic wrappers can be embedded.

Saying "update on the syntax" feels like saying "update on the morphemes". It seems to be more about which simulacrum level you apply as a wrapper to the semantic content.

I hadn't thought of the simulacra levels connection; that's useful, thanks. Though for what it's worth all of these things feel closely-related to me. At simulacrum level 4 you should think of words as similar to other actions, i.e. you calculate the expected utility of uttering some sequence of words/morphemes, rather than distinguishing "true" from "false" from "meaningless" sentences. ... (read more)

I think you like riffing, so I'll riff.

Verbal communication is a game, in the game theoretic sense.  In a game with your enemy, like zero-sum heads-up poker, you're trying minimize how much value you communicate with your words, in the sense that you when you say "call", you want your opponent to be indifferent between their replies (so they have no best option to choose from, no economic profit available).  You want to "maximize the entropy" of their subjective state after you communicate, or minimize the the information[1] communicated.

Now what if the th... (read more)

Inspired by a recent discussion about whether Anthropic broke a commitment to not push the capabilities frontier (I am more sympathetic to their position than most, because I think that it's often hard to distinguish between "current intentions" and "commitments which might be overridden by extreme events" and "solemn vows"):

Maybe one translation tool for bridging the gap between rationalists and non-rationalists is if rationalists interpret any claims about the future by non-rationalists as implicitly being preceded by "Look, I don't really believe that plans work, I think the world is inherently wildly unpredictable, I am kinda making everything up as I go along. Having said that:"

This translation tool would also require rationalists and such to make arguments of the form "I think supporting Anthropic (by, e.g., going to work there or giving it funding) is a good thing to do because they sort of have a feeling right now that it would be good not to push the AI frontier", rather than of the form "... because they're committed to not pushing the frontier".

Which are arguments one could make! But is a pretty different argument and I think people would behave differently if these were the only arguments in favour of supporting a new scaling lab.

I think that's how people should generally react in the absence of harder commitments and accountability measures.

This post confuses me.

Am I correct that the implied implication here is that assurances from a non-rationalist are essentially worthless? 

I think it is also wrong to imply that Anthropic have violated their commitment simply because they didn't rationally think through the implications of their commitment when they made it. 

I think you can understand Anthropic's actions as purely rational, just not very ethical.

They made an unenforceable commitment to not push capabilities when it directly benefited them. Now that it is more beneficial to drop the facade, they are doing so.

I think "don't trust assurances from non-rationalists" is not a good takeaway. Rather it should be "don't trust unenforceable assurances from people who will stand to greatly benefit from violating your trust at a later date".

The intended implication is something like "rationalists have a bias towards treating statements as much firmer commitments than intended then getting very upset when they are violated".

For example, unless I'm missing something, the "we do not wish to advance the rate of AI capabilities" claim is just one offhand line in a blog post. It's not a firm commitment, it's not even a claim about what their intentions are. As stated, it's just one consideration that informs their actions - and in fact the "wish" terminology is often specifically not a claim about ... (read more)

That makes sense.

I guess the followup question is "how were Anthropic able to cultivate the impression that they were safety focused if they had only made an extremely loose offhand commitment?"

Certainly the impression I had from how integrated they are in the EA community was that they had made a more serious commitment.

Everyone is afraid of the AI race, and hopes that one of the labs will actually end up doing what they think is the most responsible thing to do. Hope and fear is one hell of a drug cocktail, makes you jump to the conclusions you want based on the flimsiest evidence. But the hangover is a bastard.

Imo I don't know if we have evidence that Anthropic deliberately cultivated or significantly benefitted from the appearance of a commitment. However if an investor or employee felt like they made substantial commitments based on this impression and then later felt betrayed that would be more serious. (The story here is I think importantly different from other stories where I think there were substantial benefits from commitment appearance and then violation)

The intended implication is something like "rationalists have a bias towards treating statements as much firmer commitments than intended then getting very upset when they are violated".

That sounds suspiciously similar to "autists have a bias towards interpreting statements literally".

I mean, yes, they're closely related.

I think part of the disappointment is the lack of communication regarding violating the commitment or violating the expectations of a non-trivial fraction of the community.

If someone makes a promise to you or even sets an expectation for you in a softer way, there is of course always some chance that they will break the promise or violate the expectation.

But if they violate the commitment or the expectation, and they care about you as a stakeholder, I think there's a reasonable expectation that they should have to justify that decision.

If they break the promise or violate the soft expectation, and then they say basically nothing (or they say "well I never technically made a promise– there was no contract!", then I think you have the right to be upset with them not only for violating you expectation but also for essentially trying to gaslight you afterward.

I think a Responsible Lab would have issued some sort of statement along the lines of "hey, we're hearing that some folks thought we had made commitments to not advance the frontier and some of our employees were saying this to safety-focused members of the AI community. We're sorry about this miscommunication, and here are some s... (read more)

See my comment below. Basically I think this depends a lot on the extent to which a commitment was made.

Right now it seems like the entire community is jumping to conclusions based on a couple of "impressions" people got from talking to Dario, plus an offhand line in a blog post. With that little evidence, if you have formed strong expectations, that's on you. And trying to double down by saying "I have been bashing you because I formed an unreasonable expectation, now it's your job to fix that" seems pretty adversarial.

I do think it would be nice if Anthropic did make such a statement, but seeing how adversarially everyone has treated the information they do release, I don't blame them for not doing so.

Right now it seems like the entire community is jumping to conclusions based on a couple of "impressions" people got from talking to Dario, plus an offhand line in a blog post.

No, many people had the impression that Anthropic had made such a commitment, which is why they were so surprised when they saw the Claude 3 benchmarks/marketing.  Their impressions were derived from a variety of sources; those are merely the few bits of "hard evidence", gathered after the fact, of anything that could be thought of as an "organizational commitment".

Also, if Dustin Moskovitz and Gwern - two dispositionally pretty different people - both came away from talking to Dario with this understanding, I do not think that is something you just wave off.  Failures of communication do happen.  It's pretty strange for this many people to pick up the same misunderstanding over the course of several years, from many different people (including Dario, but also others), in a way that's beneficial to Anthropic, and then middle management starts telling you that maybe there was a vibe but they've never heard of any such commitment (nevermind what Dustin and Gwern heard, or anyone else who might've... (read more)

[-][anonymous]148

Could you clarify how binding "OpenAI’s mission is to ensure that artificial general intelligence benefits all of humanity." is?

Right now it seems like the entire community is jumping to conclusions based on a couple of "impressions" people got from talking to Dario, plus an offhand line in a blog post. With that little evidence, if you have formed strong expectations, that's on you. 

Like Robert, the impressions I had were based on what I heard from people working at Anthropic. I cited various bits of evidence because those were the ones available, not because they were the most representative. The most representative were those from Anthropic employees who concurred that this was indeed the implication, but it seemed bad form to cite particular employees (especially when that information was not public by default) rather than, e.g., Dario. I think Dustin’s statement was strong evidence of this impression, though, and I still believe Anthropic to have at least insinuated it.

I agree with you that most people are not aiming for as much stringency with their commitments as rationalists expect. Separately, I do think that what Anthropic did would constitute a betrayal, even in everyday culture. And in any case, I think that when you are making a technology which might extinct humanity, the bar should be si... (read more)

The most representative were those from Anthropic employees who concurred that this was indeed the implication, but it seemed bad form to cite particular employees (especially when that information was not public by default) rather than, e.g., Dario. I think Dustin’s statement was strong evidence of this impression, though, and I still believe Anthropic to have at least insinuated it.

This makes sense, and does update me. Though I note "implication", "insinuation" and "impression" are still pretty weak compared to "actually made a commitment", and still con... (read more)

I think the right way to think about verbal or written commitments is that they increase the costs of taking a certain course of action. A legal contract can mean that the price is civil lawsuits leading to paying a financial price. A non-legal commitment means if you break it, the person you made the commitment to gets angry at you, and you gain a reputation for being the sort of person who breaks commitments. It's always an option for someone to break the commitment and pay the price, even laws leading to criminal penalties can be broken if someone is wi... (read more)

Yeah, I think it's good if labs are willing to make more "cheap talk" statements of vague intentions, so you can learn how they think. Everyone should understand that these aren't real commitments, and not get annoyed if these don't end up meaning anything. This is probably the best way to view "statements by random lab employees".

Imo would be good to have more "changeable commitments" too in between, statements that are "we'll do policy X until we change the policy, when we do we commit to clearly informing everyone about the change" which is maybe more the current status of most RSPs.

Deceptive alignment doesn't preserve goals.

A short note on a point that I'd been confused about until recently. Suppose you have a deceptively aligned policy which is behaving in aligned ways during training so that it will be able to better achieve a misaligned internally-represented goal during deployment. The misaligned goal causes the aligned behavior, but so would a wide range of other goals (either misaligned or aligned) - and so weight-based regularization would modify the internally-represented goal as training continues. For example, if the misaligned goal were "make as many paperclips as possible", but the goal "make as many staples as possible" could be represented more simply in the weights, then the weights should slowly drift from the former to the latter throughout training.

But actually, it'd likely be even simpler to get rid of the underlying misaligned goal, and just have alignment with the outer reward function as the terminal goal. So this argument suggests that even policies which start off misaligned would plausibly become aligned if they had to act deceptively aligned for long enough. (This sometimes happens in humans too, btw.)

Reasons this argument might not be relevant:
- The policy doing some kind of gradient hacking
- The policy being implemented using some kind of modular architecture (which may explain why this phenomenon isn't very robust in humans)

Why would alignment with the outer reward function be the simplest possible terminal goal? Specifying the outer reward function in the weights would presumably be more complicated. So one would have to specify a pointer towards it in some way. And it's unclear whether that pointer is simpler than a very simple misaligned goal.

Such a pointer would be simple if the neural network already has a representation of the outer reward function in weights anyway (rather than deriving it at run-time in the activations). But it seems likely that any fixed representation will be imperfect and can thus be improved upon at inference time by a deceptive agent (or an agent with some kind of additional pointer). This of course depends on how much inference time compute and memory / context is available to the agent.

So I'm imagining the agent doing reasoning like:

Misaligned goal --> I should get high reward --> Behavior aligned with reward function

and then I'm hypothesizing that the whatever the first misaligned goal is, it requires some amount of complexity to implement, and you could just get rid of it and make "I should get high reward" the terminal goal. (I could imagine this being false though depending on the details of how terminal and instrumental goals are implemented.)

I could also imagine something more like:

Misaligned goal --> I should behave in al... (read more)

So I'm imagining the agent doing reasoning like:

Misaligned goal --> I should get high reward --> Behavior aligned with reward function

Why would the agent reason like this? 

Because of standard deceptive alignment reasons (e.g. "I should make sure gradient descent doesn't change my goal; I should make sure humans continue to trust me").

I think you don't have to reason like that to avoid getting changed by SGD. Suppose I'm being updated by PPO, with reinforcement events around navigating to see dogs. To preserve my current shards, I don't need to seek out a huge number of dogs proactively, but rather I just need to at least behave in conformance with the advantage function implied by my value head, which probably means "treading water" and seeing dogs sometimes in situations similar to historical dog-seeing events. 

Maybe this is compatible with what you had in mind! It's just not som... (read more)

To preserve my current shards, I don't need to seek out a huge number of dogs proactively, but rather I just need to at least behave in conformance with the advantage function implied by my value head, which probably means "treading water" and seeing dogs sometimes in situations similar to historical dog-seeing events.

I think this depends sensitively on whether the "actor" and the "critic" in fact have the same goals, and I feel pretty confused about how to reason about this. For example, in some cases they could be two separate models, in which case the c... (read more)

[-][anonymous]Ω110

So I'm imagining the agent doing reasoning like:

Misaligned goal --> I should get high reward --> Behavior aligned with reward function

The shortest description of this thought doesn't include "I should get high reward" because that's already implied by having a misaligned goal and planning with it. 

In contrast, having only the goal "I should get high reward" may add description length like Johannes said. If so, the misaligned goal could well be equally simple or simpler than the high reward goal.

if the misaligned goal were "make as many paperclips as possible", but the goal "make as many staples as possible" could be represented more simply in the weights, then the weights should slowly drift from the former to the latter throughout training.

Can you say why you think that weight-based regularization would drift the weights to the latter? That seems totally non-obvious to me, and probably false.

In general if two possible models perform the same, then I expect the weights to drift towards the simpler one. And in this case they perform the same because of deceptive alignment: both are trying to get high reward during training in order to be able to carry out their misaligned goal later on.

[-][anonymous]Ω330

Interesting point. Though on this view, "Deceptive alignment preserves goals" would still become true once the goal has drifted to some random maximally simple goal for the first time.

To be even more speculative: Goals represented in terms of existing concepts could be simple and therefore stable by default. Pretrained models represent all kinds of high-level states, and weight-regularization doesn't seem to change this in practice. Given this, all kinds of goals could be "simple" as they piggyback on existing representations, requiring little additional description length.

This doesn't seem implausible. But on the other hand, imagine an agent which goes through a million episodes, and in each one reasons at the beginning "X is my misaligned terminal goal, and therefore I'm going to deceptively behave as if I'm aligned" and then acts perfectly like an aligned agent from then on. My claims then would be:

a) Over many update steps, even a small description length penalty of having terminal goal X (compared with being aligned) will add up.
b) Having terminal goal X also adds a runtime penalty, and I expect that NNs in practice are... (read more)

Being nice because you're altruistic, and being even nicer for decision-theoretic reasons on top of that, seems like it involves some kind of double-counting: the reason you're altruistic in the first place is because evolution ingrained the decision theory into your values.

But it's not fully double-counting: many humans generalise altruism in a way which leads them to "cooperate" far more than is decision-theoretically rational for the selfish parts of them - e.g. by making big sacrifices for animals, future people, etc. I guess this could be selfishly rational if you subscribe to a very strong form of updatelessness, but I am very skeptical that we'll discover arguments that this much updatelessness is rationally obligatory.

A very speculative takeaway: maybe "how updateless you are" and "how altruistic you are" are kinda measuring the same thing, and there's no clean split between whether that's determined by your values or your decision theory.

Your actions and decisions are not doubled.  If you have multiple paths to arrive at the same behaviors, that doesn't make them wrong or double-counted, it just makes it hard to tell which of them is causal (aka: your behavior is overdetermined).

Are you using "updatelessness" to refer to not having self in your utility function?  If so, that's a new one one me, and I'd prefer "altruism" as the term.  I'm not sure that the decision-theory use of "updateless" (to avoid incorrect predictions where experience is correlated with the question at hand) makes sense here.

Yeah, I don't think "double counted" is the right term here. 

Consider, if just personally like the taste of kale, I'll eat kale. If I also find out that kale is especially healthy, I have additional reason to eat kale, compared to alternatives that are not as healthy.

Surely there's some correlation and causal relationship between what I find tasty and what is healthy. Nutrition (and avoiding poisoning) is the main reason taste evolved! 

But that doesn't mean that taste and health aren't separate reasons for me to prefer to eat kale.

Oh, this also suggests a way in which the utility function abstraction is leaky, because the reasons for the payoffs in a game may matter. E.g. if one payoff is high because the corresponding agent is altruistic, then in some sense that agent is "already cooperating" in a way which is baked into the game, and so the rational thing for them to do might be different from the rational thing for another agent who gets the same payoffs, but for "selfish" reasons.

Maybe FDT already lumps this effect into the "how correlated are decisions" bucket? Idk.

Another double-counting: wanting for people to be saved for altruistic reasons and wanting to personally do things that save people.

A well-known analogy from Yann LeCun: if machine learning is a cake, then unsupervised learning is the cake itself, supervised learning is the icing, and reinforcement learning is the cherry on top.

I think this is useful for framing my core concerns about current safety research:

  • If we think that unsupervised learning will produce safe agents, then why will the comparatively small contributions of SL and RL make them unsafe?
  • If we think that unsupervised learning will produce dangerous agents, then why will safety techniques which focus on SL and RL (i.e. basically all of them) work, when they're making comparatively small updates to agents which are already misaligned?

I do think it's more complicated than I've portrayed here, but I haven't yet seen a persuasive response to the core intuition.

I wrote a few posts on self-supervised learning last year:

I'm not aware of any airtight argument that "pure" self-supervised learning systems, either generically or with any particular architecture, are safe to use, to arbitrary levels of intelligence, though it seems very much worth som... (read more)

Preregistering a disagreement with my MATS fellow Erez Abrams:

He thinks the right number of levels of agency is 2. I think that 2 is too large, and the right number of levels of agency is 1.

(This isn't mean to make much sense to anyone else, but hopefully one of us will end up obviously right at some point, and will owe the other person a poem about the beauty of their conception of agency.)

Well then, here's my cryptic pre-registered bet:

There's no right number of levels of agency, just like for most complex concepts. It's not only continuous, but multidimensional.

Thinking in categories, when the territory is continuous, is the source of a pretty good fraction of the total amount of confusion and miscommunication.

(Vague, speculative thinking): Is the time element of UDT actually a distraction? Consider the following: agents A and B are in a situation where they'd benefit from cooperation. Unfortunately, the situation is complicated—it's not like a prisoner's dilemma, where there's a clear "cooperate" and a clear "defect" option. Instead they need to take long sequences of actions, and they each have many opportunities to subtly gain an advantage at the other's expense.

Therefore instead of agreements formulated as "if you do X I'll do Y", it'd be far more beneficial for them to make agreements of the form "if you follow the advice of person Z then I will too". Here person Z needs to be someone that both A and B trust to be highly moral, neutral, competent, etc. Even if there's some method of defecting that neither of them considered in advance, at the point in time when it arises Z will advise against doing it. (They don't need to actually have access to Z, they can just model what Z will say.)

If A and B don't have much communication bandwidth between them (e.g. they're trying to do acausal coordination) then they will need to choose a Z that's a clear Schelling point, even if that Z is subo... (read more)

Most discussion of updatelessness suggests that Z is an agent similar to A and B, and also that it's a policy whose implications are transparent to A and B. I think an importaint case has Z quite unlike A or B, possibly much smaller and more legible than them. And it can still be an agent in its own right, capable of eventually growing stronger than A or B were at the time Z was initially formulated. By a growing/developing Z I mean something that exists in coordination through both of these places, rather than splitting into a version of Z at A, and a ver... (read more)

I think Nesov had some similar idea about "agents deferring to a (logically) far-away algorithm-contract Z to avoid miscoordination", although I never understood it completely, nor think that idea can solve miscoordination in the abstract (only, possibly, be a nice pragmatic way to bootstrap coordination from agents who are already sufficiently nice).

EDIT 2: UDT is usually prone to commitment races because it thinks of each agent in a conflict as separately making commitments earlier in logical time. But focusing on symmetric commitments gets rid of this p

... (read more)

nor think that idea can solve miscoordination in the abstract (only, possibly, be a nice pragmatic way to bootstrap coordination from agents who are already sufficiently nice).

The central question to my mind is principles of establishing coordination between different situations/agents, and contracts is a framing for what coordination might look like once established. Agentic contracts have the additional benefit of maintaining coordination across their instances once it's established initially. Coordination theory should clarify how agents should think... (read more)

Imagine taking someone's utility function, and inverting it by flipping the sign on all evaluations. What might this actually look like? Well, if previously I wanted a universe filled with happiness, now I'd want a universe filled with suffering; if previously I wanted humanity to flourish, now I want it to decline.

But this is assuming a Cartesian utility function. Once we treat ourselves as embedded agents, things get trickier. For example, suppose that I used to want people with similar values to me to thrive, and people with different values from me to suffer. Now if my utility function is flipped, that naively means that I want people similar to me to suffer, and people similar to me to thrive. But this has a very different outcome if we interpret "similar to me" as de dicto vs de re - i.e. whether it refers to the old me or the new me.

This is a more general problem when one person's utility function can depend on another person's, where you can construct circular dependencies (which I assume you can also do in the utility-flipping case). There's probably been a bunch of work on this, would be interested in pointers to it (e.g. I assume there have been attempts to construct typ... (read more)

Fundamentally, humans aren't VNM-rational, and don't actually have utility functions.  Which makes the thought experiment much less fun.  If you recast it as "what if a human brain's reinforcement mechanisms were reversed", I suspect it's also boring: simple early death.

The interesting fictional cases are when some subset of a person's legible motivations are reversed, but the mass of other drives remain.  This very loosely maps to reversing terminal goals and re-calculating instrumental goals - they may reverse, stay, or change in weird way... (read more)

I have been asking a similar question for a long time. This is similar to the standard problem that if we deny regularity, will it be regular irregularity or irregular irregularity, that is, at what level are we denying the phenomeno? And only at one level?

It seems to me that Eliezer overrates the concept of a simple core of general intelligence, whereas Paul underrates it. Or, alternatively: it feels like Eliezer is leaning too heavily on the example of humans, and Paul is leaning too heavily on evidence from existing ML systems which don't generalise very well.

I don't think this is a particularly insightful or novel view, but it seems worth explicitly highlighting that you don't have to side with one worldview or the other when evaluating the debates between them. (Although I'd caution not to just average their two views - instead, try to identify Eliezer's best arguments, and Paul's best arguments, and reconcile them.)

[Epistemic status: rough speculation, feels novel to me, though Wei Dai probably already posted about it 15 years ago.]

UDT is (roughly) defined as "follow whatever commitments a past version of yourself would have made if they'd thought about your situation". But this means that any UDT agent is only as robust to adversarial attacks as their most vulnerable past self. Specifically, it creates an incentive for adversaries to show UDT agents situations that would trick their past selves into making unwise commitments. It also creates incentives for UDT agents themselves to hack their past selves, in order to artificially create commitments that "took effect" arbitrarily far back in their past.

In some sense, then, I think UDT might have a parallel structure to the overall alignment problem. You have dumber past agents who don't understand most of what's going on. You have smarter present agents who have trouble cooperating, because they know too much. The smarter agents may try to cooperate by punting to "Schelling point" dumb agents. (But this faces many of the standard problems of dumb agents making decisions—e.g. the commitments they make will probably be inconsistent or incoherent... (read more)

UDT is (roughly) defined as "follow whatever commitments a past version of yourself would have made if they'd thought about your situation".

This seems substantially different from UDT, which does not really have or use a notion of "past version of yourself". For example imagine a variant of Counterfactual Mugging in which there is no preexisting agent, and instead Omega creates an agent from scratch after flipping the coin and gives it the decision problem. UDT is fine with this but "follow whatever commitments a past version of yourself would have made... (read more)

This seems substantially different from UDT, which does not really have or use a notion of "past version of yourself".

My terminology here was sloppy, apologies. When I say "past versions of yourself" I am also including (as Nesov phrases it below) "the idealized past agent (which doesn't physically exist)". E.g. in the Counterfactual Mugging case you describe, I am thinking about precommitments that the hypothetical past version of yourself from before the coin was flipped would have committed to.

I find it a more intuitive way to think about UDT, though I realize it's a somewhat different framing from yours. Do you still think this is substantially different?

UDT never got past the setting of unchanging preferences, so the present agent blindly defers to all decisions of the idealized past agent (which doesn't physically exist). And if the past agent doesn't try to wade in the murky waters of logical updatelessness, it's not really dumber or more fallible to trickery, it can see everything the way a universal Turing machine or Solomonoff induction can "see everything". Coordinating agents with different values was instead explored under the heading of Prisoner's Dilemma. Though a synthesis between coordination ... (read more)

if the past agent doesn't try to wade in the murky waters of logical updatelessness, it's not really dumber or more fallible to trickery, it can see everything the way a universal Turing machine or Solomonoff induction can "see everything".

I actually think it might still be more fallible, for a couple of reasons.

Firstly, consider an agent which, at time T, respects all commitments it would have made at times up to T. Now if you're trying to attack the agent at time T, you have T different versions of it that you can attack, and if any of them makes a dumb ... (read more)

UDT doesn't do multistage commitments, it has a single all-powerful "past" version that looks into all possible futures before pronouncing a global policy that all of them would then follow. This policy is not a collection of commitments in a reasonable informal sense, it's literally all details of behavior of future versions of the agent in response to all possible observations. In case of logical updatelessness, also in response to all possible observations of computational facts. (UDT for the idealized past version defines a single master model, future ... (read more)

Okay, so trying to combine Prisoner's dilemma and UDT, we get: A and B are in a prisoner's dilemma. Suppose they have a list of N agents (which include, say, A's past self, B's past self, the Buddha, etc), and they each must commit to following one of those agent's instructions. Each of them estimates: "conditional on me committing to listen to agent K, here's a distribution over which agent they'd commit to listen to" And then you maximize expected value based on that. 

Okay, but why isn't this exactly the same as them just thinking to themselves "con... (read more)

Okay, but why isn't this exactly the same as them just thinking to themselves "conditional on me taking action K, here's the distribution over their actions" for each of N actions they could take, and then maximizing expected value?

The main trick with PD is that instead of an agent only having two possible actions C and D, we consider many programs the agent might self-modify into (commit to becoming) that each might in the end compute C or D. This effectively changes the action space, there are now many more possible actions. And these programs/actions... (read more)

I've been reading Eliezer's recent stories with protagonists from dath ilan (his fictional utopia). Partly due to the style, I found myself bouncing off a lot of the interesting claims that he made (although it still helped give me a feel for his overall worldview). The part I found most useful was this page about the history of dath ilan, which can be read without much background context. I'm referring mostly to the exposition on the first 2/3 of the page, although the rest of the story from there is also interesting. One key quote from the remainder of the story:

"The next most critical fact about Earth is that from a dath ilani perspective their civilization is made entirely out of coordination failure.  Coordination that fails on every scale recursively, where uncoordinated individuals assemble into groups that don't express their preferences, and then those groups also fail to coordinate with each other, forming governments that offend all of their component factions, which governments then close off their borders from other governments.  The entirety of Earth is one gigantic failure fractal.  It's so far below the multi-agent-optimal-boundary, only their profess

... (read more)

I’d say lots of other things he’s said support that update. Stuff about how your model of the world will be accurate if and only if you somehow approximate Bayes’ law, for example.

The dath ilan based fiction definitely helped me internalize the idea better though.

For a while now I've been thinking about the difference between "top-down" agents which pursue a single goal, and "bottom-up" agents which are built around compromises between many goals/subagents.

I've now decided that the frame of "centralized" vs "distributed" agents is a better way of capturing this idea, since there's no inherent "up" or "down" direction in coalitions. It's also more continuous.

Credit to @Scott Garrabrant, who something like this point to me a while back, in a way which I didn't grok at the time.

A tension that keeps recurring when I think about philosophy is between the "view from nowhere" and the "view from somewhere", i.e. a third-person versus first-person perspective—especially when thinking about anthropics.

One version of the view from nowhere says that there's some "objective" way of assigning measure to universes (or people within those universes, or person-moments). You should expect to end up in different possible situations in proportion to how much measure your instances in those situations have. For example, UDASSA ascribes measure based on the simplicity of the computation that outputs your experience.

One version of the view from somewhere says that the way you assign measure across different instances should depend on your values. You should act as if you expect to end up in different possible future situations in proportion to how much power to implement your values the instances in each of those situations has. I'll call this the ADT approach, because that seems like the core insight of Anthropic Decision Theory. Wei Dai also discusses it here.

In some sense each of these views makes a prediction. UDASSA predicts that we live in a universe with laws of physi... (read more)

Very interesting. It sounds like your "third person view from nowhere" vs the "first person view from somewhere" is very similar to something I was thinking about recently. I called them "objectively distinct situations" in contrast with "subjectively distinct situations". My view is that most of the anthropic arguments that "feel wrong" to me are built on trying to make me assign equal probability to all subjectively distinct scenarios, rather than objective ones. eg. A replication machine makes it so there are two of me, then "I" could be either of them,... (read more)

This was all kinda rambly but I think I can summarize it as "Isn't it weird that ADT tells us that we should act as if we'll end up in unusually important places, and also we do seem to be in an incredibly unusually important place in the universe? I don't have a story for why these things are related but it does seem like a suspicious coincidence."

I'm not sure this is a valid interpretation of ADT. Can you say more about why you interpret ADT this way, maybe with an example? My own interpretation of how UDT deals with anthropics (and I'm assuming ADT i... (read more)

My own interpretation of how UDT deals with anthropics (and I'm assuming ADT is similar) is "Don't think about indexical probabilities or subjective anticipation. Just think about measures of things you (considered as an algorithm with certain inputs) have influence over."

(Speculative paragraph, quite plausibly this is just nonsense.) Suppose you have copies A and B who are both offered the same bet on whether they're A. One way you could make this decision is to assign measure to A and B, then figure out what the marginal utility of money is for each of A... (read more)

Intuitively this feels super weird and unjustified, but it does make the "prediction" that we'd find ourselves in a place with high marginal utility of money, as we currently do.

This is particularly weird because your indexical probability then depends on what kind of bet you're offered. In other words, our marginal utility of money differs from our marginal utility of other things, and which one do you use to set your indexical probability? So this seems like a non-starter to me... (ETA: Maybe it changes moment by moment as we consider different decisi... (read more)

This is particularly weird because your indexical probability then depends on what kind of bet you're offered. In other words, our marginal utility of money differs from our marginal utility of other things, and which one do you use to set your indexical probability? So this seems like a non-starter to me...

It seems pretty weird to me too, but to steelman: why shouldn't it depend on the type of bet you're offered? Your indexical probabilities can depend on any other type of observation you have when you open your eyes. E.g. maybe you see blue carpets, and ... (read more)

I added this to my comment just before I saw your reply: Maybe it changes moment by moment as we consider different decisions, or something like that? But what about when we're just contemplating a philosophical problem and not trying to make any specific decisions?

I mostly offer this in the spirit of "here's the only way I can see to reconcile subjective anticipation with UDT at all", not "here's something which makes any sense mechanistically or which I can justify on intuitive grounds".

Ah I see. I think this is incomplete even for that purpose, beca... (read more)

In a bayesian rationalist view of the world, we assign probabilities to statements based on how likely we think they are to be true. But truth is a matter of degree, as Asimov points out. In other words, all models are wrong, but some are less wrong than others.

Consider, for example, the claim that evolution selects for reproductive fitness. Well, this is mostly true, but there's also sometimes group selection, and the claim doesn't distinguish between a gene-level view and an individual-level view, and so on...

So just assigning it a single probability seems inadequate. Instead, we could assign a probability distribution over its degree of correctness. But because degree of correctness is such a fuzzy concept, it'd be pretty hard to connect this distribution back to observations.

Or perhaps the distinction between truth and falsehood is sufficiently clear-cut in most everyday situations for this not to be a problem. But questions about complex systems (including, say, human thoughts and emotions) are messy enough that I expect the difference between "mostly true" and "entirely true" to often be significant.

Has this been discussed before? Given Less Wrong's name, I'd be surprised if not, but I don't think I've stumbled across it.

This feels generally related to the problems covered in Scott and Abram's research over the past few years. One of the sentences that stuck out to me the most was (roughly paraphrased since I don't want to look it up): 

In order to be a proper bayesian agent, a single hypothesis you formulate is as big and complicated as a full universe that includes yourself

I.e. our current formulations of bayesianism like solomonoff induction only formulate the idea of a hypothesis at such a low level that even trying to think about a single hypothesis rigorously is ... (read more)

This seems reasonable, thanks. But I note that "in order to actually think about anything you have to somehow move beyond naive bayesianism" is a very strong criticism. Does this invalidate everything that has been said about using naive bayesianism in the real world? E.g. every instance where Eliezer says "be bayesian".

One possible answer is "no, because logical induction fixes the problem". My uninformed guess is that this doesn't work because there are comparable problems with applying to the real world. But if this is your answer, follow-up question: b... (read more)

I think it definitely changed a bunch of stuff for me, and does at least a bit invalidate some of the things that Eliezer said, though not actually very much. 

In most of his writing Eliezer used bayesianism as an ideal that was obviously unachievable, but that still gives you a rough sense of what the actual limits of cognition are, and rules out a bunch of methods of cognition as being clearly in conflict with that theoretical ideal. I did definitely get confused for a while and tried to apply Bayes to everything directly, and then felt bad when I co... (read more)

rules out a bunch of methods of cognition as being clearly in conflict with that theoretical ideal

Which ones? In Against Strong Bayesianism I give a long list of methods of cognition that are clearly in conflict with the theoretical ideal, but in practice are obviously fine. So I'm not sure how we distinguish what's ruled out from what isn't.

which I now realize is because those tended to be problems where embededness or logical uncertainty mattered a lot

Can you give an example of a real-world problem where logical uncertainty doesn't matter a lot, given th... (read more)

I think in-practice there are lots of situations where you can confidently create a kind of pocket-universe where you can actually consider hypotheses in a bayesian way. 

Concrete example: Trying to figure out who voted a specific way on a LW post. You can condition pretty cleanly on vote-strength, and treat people's votes as roughly independent, so if you have guesses on how different people are likely to vote, it's pretty easy to create the odds ratios for basically all final karma + vote numbers and then make a final guess based on that. 

It's c... (read more)

Are you able to give examples of the times you tried to be Bayesian and it failed because embedded was?

Scott and Abram? Who? Do they have any books I can read to familiarize myself with this discourse?

Related but not identical: this shortform post.

See the section about scoring rules in the Technical Explanation.

Hmmm, but what does this give us? He talks about the difference between vague theories and technical theories, but then says that we can use a scoring rule to change the probabilities we assign to each type of theory.

But my question is still: when you increase your credence in a vague theory, what are you increasing your credence about? That the theory is true?

Nor can we say that it's about picking the "best theory" out of the ones we have, since different theories may overlap partially.

If we can quantify how good a theory is at making accurate predictions (or rather, quantify a combination of accuracy and simplicity), that gives us a sense in which some theories are "better" (less wrong) than others, without needing theories to be "true".

Oracle-genie-sovereign is a really useful distinction that I think I (and probably many others) have avoided using mainly because "genie" sounds unprofessional/unacademic. This is a real shame, and a good lesson for future terminology.

After rereading the chapter in Superintelligence, it seems to me that "genie" captures something akin to act-based agents. Do you think that's the main way to use this concept in the current state of the field, or do you have other applications in mind?

Ah, yeah, that's a great point. Although I think act-based agents is a pretty bad name, since those agents may often carry out a whole bunch of acts in a row - in fact, I think that's what made me overlook the fact that it's pointing at the right concept. So not sure if I'm comfortable using it going forward, but thanks for pointing that out.

Perhaps the lesson is that terminology that is acceptable in one field (in this case philosophy) might not be suitable in another (in this case machine learning).

I don't think that even philosophers take the "genie" terminology very seriously. I think the more general lesson is something like: it's particularly important to spend your weirdness points wisely when you want others to copy you, because they may be less willing to spend weirdness points.

Is that from Superintelligence? I googled it, and that was the most convincing result.

In my post on value systematization I used utilitarianism as a central example of value systematization.

Value systematization is important because it's a process by which a small number of goals end up shaping a huge amount of behavior. But there's another different way in which this happens: core emotional motivations formed during childhood (e.g. fear of death) often drive a huge amount of our behavior, in ways that are hard for us to notice.

Fear of death and utilitarianism are very different. The former is very visceral and deep-rooted; it typically influences our behavior via subtle channels that we don't even consciously notice (because we suppress a lot of our fears). The latter is very abstract and cerebral, and it typically influences our behavior via allowing us to explicitly reason about which strategies to adopt.

But fear of death does seem like a kind of value systematization. Before we have a concept of death we experience a bunch of stuff which is scary for reasons we don't understand. Then we learn about death, and then it seems like we systematize a lot of that scariness into "it's bad because you might die".

But it seems like this is happening way less consciously th... (read more)

So maybe we need to think about systematization happening separately in system 1 and system 2?

I think that's right. Taking on the natural-abstraction lens, there is a "ground truth" to the "hierarchy of values". That ground truth can be uncovered either by "manual"/symbolic/System-2 reasoning, or by "automatic"/gradient-descent-like/System-1 updates, and both processes would converge to the same hierarchy. But in the System-2 case, the hierarchy would be clearly visible to the conscious mind, whereas the System-1 route would make it visible only indirectly... (read more)

Reminds me of Maslow's pyramid.

I made an article about values, saying the supreme value is life and every other value derives from it. 
Watch out, this most probably does not align with your view at first glance:
https://www.lesswrong.com/posts/xx3St4KC3KHHPGfL9/human-alignment

But fear of death does seem like a kind of value systematization

I don't think it's system 1 doing the systemization.  Evolution beat fear of death into us in lots of independent forms (fear of heights, snakes, thirst, suffocation, etc.), but for the same underlying reason.  Fear of death is not just an abstraction humans invented or acquired in childhood; is a "natural idea" pointed at by our brain's innate circuitry from many directions.  Utilitarianism doesn't come with that scaffolding.  We don't learn to systematize Euclidian and Minkowskian spaces the same way either.

People sometimes try to reason about the likelihood of deceptive alignment by appealing to speed priors and simplicity priors. I don't like such appeals, because I think that the differences between aligned and deceptive AGIs will likely be a very small proportion of the total space/time complexity of an AGI. More specifically:

1. If AGIs had to rederive deceptive alignment in every episode, that would make a big speed difference. But presumably, after thinking about it a few times during training, they will remember their conclusions for a while, and bring them to mind in whichever episodes they're relevant. So the speed cost of deception will be amortized across the (likely very long) training period.

2. AGIs will represent a huge number of beliefs and heuristics which inform their actions (e.g. every single fact they know). A heuristic like "when you see X, initiate the world takeover plan" would therefore constitute a very small proportion of the total information represented in the network; it'd be hard to regularize it away without regularizing away most of the AGI's knowledge.

I think that something like the speed vs simplicity tradeoff is relevant to the likelihood of deceptiv... (read more)

But presumably, after thinking about it a few times during training, they will remember their conclusions for a while, and bring them to mind in whichever episodes they're relevant.

Why do you think SGD will do this? Or are you imagining non-SGD mechanisms?

It seems non-obvious to me that this will occur with SGD, though possible.

If AGIs had to rederive deceptive alignment in every episode, that would make a big speed difference. But presumably, after thinking about it a few times during training, they will remember their conclusions for a while, and bring them to mind in whichever episodes they're relevant. So the speed cost of deception will be amortized across the (likely very long) training period.

You mean this about something trained totally differently than a LLM, no? Because this mechanism seems totally implausible to me otherwise.

Think during the forward pass, learn during the backward pass; if the model uses deceptive reasoning in the forward pass and the gradient says it's useful for prediction, that seems like the mechanism as described. Thoughts?

So, there are a few different reasons, none of which I've formalized to my satisfaction.

I'm curious if these make sense to you.

(1) One is that the actual kinds of reasoning that an LLM can learn in its forward pass are quite limited.

As is well established, for instance, Transformers cannot multiply arbitrarily-long integers in a single forward pass. The number of additions involved in multiplying an N-digit integer increases in an unbounded way with N; thus, a Transformer with with a finite number of layers cannot do it. (Example: Prompt GPT-4 for the resu... (read more)

You've given me a lot to think about, thanks! Here are my thoughts as I read:

(1) One is that the actual kinds of reasoning that an LLM can learn in its forward pass are quite limited.

As is well established, for instance, Transformers cannot multiply arbitrarily-long integers in a single forward pass. The number of additions involved in multiplying an N-digit integer increases in an unbounded way with N; thus, a Transformer with with a finite number of layers cannot do it. (Example: Prompt GPT-4 for the results of multiplying two 5-digit numbers, specifying

... (read more)

These all seem like reasonable reasons to doubt the hypothesized mechanism, yup. I think you're underestimating how much can happen in a single forward pass, though - it has to be somewhat shallow, so it can't involve too many variables, but the whole point of making the networks as large as we do these days is that it turns out an awful lot can happen in parallel. I also think there would be no reason for deception to occur if it's never a good weight pattern to use to predict the data, it's only if the data contains a pattern that the gradient will put i... (read more)

Since there's been some recent discussion of the SSC/NYT incident (in particular via Zack's post), it seems worth copying over my twitter threads from that time about why I was disappointed by the rationalist community's response to the situation.

I continue to stand by everything I said below.

Thread 1 (6/23/20):

Scott Alexander is the most politically charitable person I know. Him being driven off the internet is terrible. Separately, it is also terrible if we have totally failed to internalize his lessons, and immediately leap to the conclusion that the NY... (read more)

Scott is already too charitable. I'd even say that Scott being too charitable made this specific situation worse. I don't find this to be a worthwhile thing about Scott either for us to emulate, or for Scott to take further.

"Quokka" is a meme about rationalists for a reason. You are not going to have unerring logical evidence that someone wants to harm you if they are trying to be at all subtle. You have to figure it out from their behavior.

Sometimes it just isn't true that both sides are reasonable and have useful perspectives.

Ours is a community built around the long-term value of telling the truth. Are we unable to imagine reasonable disagreement about when the benefits of revealing real names outweigh the harms? Yes, it goes against our norms, but different groups have different norms.

 

I think this only holds if NYT has a consistent policy of using real names. My understanding is they have repeatedly written about other people using pseudonyms only, and have not articulated a principled reason to treat Scott differently. 

standards of most of his other posts, where he assumes both sides are reasonable and have useful perspectives

Scott's flavor of charity is not quite this. It wouldn't be useful for understanding sides that are not reasonable or have useless perspectives otherwise, or else you'd need to routinely "assume" false things to carry out the exercise.

The point is to meaningfully engage with other perspectives, without the usual prerequisite of having positive beliefs about them. Treating them in a similar way as if they were reasonable or useful, even when they ... (read more)

It's really weird that we find ourselves at the hinge of history. One proposed explanation is that we're part of an ancestor simulation. It makes sense that ancestor simulations would be focused on the hinge of history. But unless ancestor simulations make up a significant proportion of future minds, it's still weird that we find ourselves in a simulation rather than actually experiencing the future.

Why might ancestor simulations make up a significant proportion of future minds? One possible answer is that ancestor simulations provide the information requi... (read more)

Yeah, this talk for instance.

An additional part of the story is that if a superintelligence want to compute the distribution of outcomes of an intelligent civilization doing a singularity, it will want to sample histories leading to the singularity. But these histories have a branching tree structure. 

Suppose you've already simulated the evolution of life and humanity up to the singularity, and you're interested in other nearby ways it could have played out. You could simulate the whole thing again, but that's probably unnecessary. Computationally che... (read more)

I've done work in this area, but never been particularly enthusiastic about promoting it. It usually turns out to be inactionable/grim/likely to rouse a panic.

This is a familiar thought, to me.

A counterargument occurs to me: Isn't it arguable that most of what we need to know about a species, to trade with it, is just downstream of its biology? Of course we talk a lot about our contingent factors, our culture, our history, but I think we're pretty much just the same animals we've always been, extrapolated. If that's the case, wouldn't far more simulation t... (read more)

Please make sure to keep, in cold storage, copies of misaligned AGIs that you may produce, when you catch them. It's important. This policy could save us.

Would you care to expand on your remark?  I don’t see how it follows from what you said above it.

Yeah, it wasn't argued. I wasn't sure whether it needed to be explained, for Richard. I don't remember how I wound up getting there from the rest of the comment, I think it was just in the same broad neighborhood.

Regardless, yes, I totally can expand on that. Here, I wrote it up: Do Not Delete your Misaligned AGI.

World champion in Chess: "It's really weird that I'm world champion. It must be a simulation or I must dream or.."

Joe Biden: "It's really weird I'm president, it must be a simul..."
(Donald Trump: "It really really makes no sense I'm president, it MUST be a s..")

David Chalmers: "It's really weird I'm providing the seminal hard problem formulation. It must be a sim.."

...

Rationalist (before finding lesswrong): "Gosh, all these people around me, really wired differently than I am. I must be in a simulation."

 

Something seems funny to me in the anthropic re... (read more)

Nit: ECL is just one of several kinds of acausal cooperation across large worlds.

What are the others?

In general I don't think anthropic reasoning like this holds any substance. We experience what we experience, and condition on that in forming models about what it is and where we are in it.

We don't get to make millions of bits of observations about being a human in a technological society, use those observations to extrapolate the possibility of supergalactic multitudes of consciousness, and then express surprise at a pathetic few dozen bits of improbability of not being one of those multitudes. We already used those bits (and a great many more!) in forming our model in the first place.

I've been thinking lately about the assumptions underlying different intellectual fields. This is hard to compress, and I'm working on a few posts about different aspects of these assumptions, but here's a very rough summary of two crucial factors and how they vary across different fields:

Screenshot 2026-02-17 at 11.07.11 AM.png

The first is where the boundaries around agents are drawn. E.g. economics and law typically take the individual to be the basic unit of analysis. It's possible to talk about larger or smaller units in some cases, but these aren't as fundamental (e.g. neither would work if... (read more)

As a comment on identity in science: certainly it's not about ‘personal/agentic’ identity, but I have been thinking a lot about how we draw boundaries between objects - like “when” a group of 3 quarks “is” a proton. Generally this involves specifying a “scale” parameter expressing ~ how much information we’re willing to lose in abstracting away from details - then we take, symbolically, the limit where this parameter goes to infinity. Then you can use perturbation theory to bridge lower-level effects to cash out at the higher-level (eg how quark structu... (read more)

My mental one-sentence summary of how to think about ELK is "making debate work well in a setting where debaters are able to cite evidence gained by using interpretability tools on each other".

I'm not claiming that this is how anyone else thinks about ELK (although I got the core idea from talking to Paul) but since I haven't seen it posted online yet, and since ELK is pretty confusing, I thought it'd be useful to put out there. In particular, this framing motivates us generating interpretability tools which scale in the sense of being robust when used as ... (read more)

In UDT2, when you're in epistemic state Y and you need to make a decision based on some utility function U, you do the following:
1. Go back to some previous epistemic state X and an EDT policy (the combination of which I'll call the non-updated agent).
2. Spend a small amount of time trying to find the policy P which maximizes U based on your current expectations X.
3. Run P(Y) to make the choice which maximizes U.

The non-updated agent gets much less information than you currently have, and also gets much less time to think. But it does use the same utility ... (read more)

People back then certainly didn't think of changing preferences.

Also, you can get rid of this problem by saying "you just want to maximize the variable U". And the things you actually care about (dogs, apples) are just "instrumentally" useful in giving you U. So for example, it is possible in the future you will learn dogs give you a lot of U, or alternatively that apples give you a lot of U.
Needless to say, this "instrumentalization" of moral deliberation is not how real agents work. And leads to getting Pascal's mugged by the world in which you care a lo... (read more)

Also, you can get rid of this problem by saying "you just want to maximize the variable U". And the things you actually care about (dogs, apples) are just "instrumentally" useful in giving you U.

But you need some mechanism for actually updating your beliefs about U, because you can't empirically observe U. That's the role of V.

leads to getting Pascal's mugged by the world in which you care a lot about easy things

I think this is fine. Consider two worlds:

In world L, lollipops are easy to make, and paperclips are hard to make.

In world P, it's the reverse.

Sup... (read more)

But you need some mechanism for actually updating your beliefs about U

Yep, but you can just treat it as another observation channel into UDT. You could, if you want, treat it as a computed number you observe in the corner of your eye, and then just apply UDT maximizing U, and you don't need to change UDT in any way.

UDT says to pay here

(Let's not forget this depends on your prior, and we don't have any privileged way to assign priors to these things. But that's a tangential point.)

I do agree that there's not any sharp distinction between situations where it... (read more)

Yep, but you can just treat it as another observation channel into UDT.

Hmm, I'm confused by this. Why should we treat it this way? There's no actual observation channel, and in order to derive information about utilities from our experiences, we need to specify some value learning algorithm. That's the role V is playing.

It's just that, when we do that, something feels off (to us humans, maybe due to risk-aversion), and we go "hmm, probably this framework is not modelling everything we want, or missing some important robustness considerations, or whatever,

... (read more)

There's no actual observation channel, and in order to derive information about utilities from our experiences, we need to specify some value learning algorithm.

Yes, absolutely! I just meant that, once you give me whatever V you choose to derive U from observations, I will just be able to apply UDT on top of that. So under this framework there doesn't seem to be anything new going on, because you are just choosing an algorithm V at the start of time, and then treating its outputs as observations. That's, again, why this only feels like a good model of "com... (read more)

In times of UDT2, the background assumption was that agents should maintain an unchanging preference, which is separate from knowledge. One motivation for UDT is that updating makes an agent stop caring about updated-away possibilities, while UDT is not doing that. Going back to a previous epistemic state is a way of preserving preference from that epistemic state, the "current" utility function is considered a bug and doesn't do anything if UDT is adopted. The non-updated agent can in principle consider the information you currently have as one of the pos... (read more)

One motivation for UDT is that updating makes an agent stop caring about updated-away possibilities, while UDT is not doing that.

I think there's an ambiguity here. UDT makes the agent stop considering updated-away possibilities, but I haven't seen any discussion of UDT which suggests that it stops caring about them in principle (except for a brief suggestion from Paul that one option for UDT is to "go back to a position where I’m mostly ignorant about the content of my values"). Rather, when I've seen UDT discussed, it focuses on updating or un-updating yo... (read more)

UDT makes the agent stop considering updated-away possibilities, but I haven't seen any discussion of UDT which suggests that it stops caring about them in principle

UDT specifically enables agents to consider the updated-away possibilities in a way relevant to decision making, while an updated agent (that's not using something UDT-like) wouldn't be able to do that in any circumstance, and so would be functionally indistinguishable from an agent that has different preferences or undefined preferences for those possibilities. Not caring about them seems l... (read more)

UDT specifically enables agents to consider the updated-away possibilities in a way relevant to decision making, while an updated agent (that's not using something UDT-like) wouldn't be able to do that in any circumstance

Agreed; apologies for the sloppy phrasing.

Historically it was overwhelmingly the frame until recently, so it's the correct frame for interpreting the intended meaning of texts from that time.

I agree, that's why I'm trying to outline an alternative frame for thinking about it.

Some more thoughts: we can portray the process of choosing a successor policy as the iterative process of making more and more commitments over time. But what does it actually look like to make a commitment? Well, consider an agent that is made of multiple subagents, that each get to vote on its decisions. You can think of a commitment as basically saying "this subagent still gets to vote, but no longer gets updated"—i.e. it's a kind of stop-gradient.

Two interesting implications of this perspective:

  1. The "cost" of a commitment can be measured both in terms o
... (read more)

More thoughts: what's the difference between paying in a counterfactual mugging based on:

  1. Whether the millionth digit of pi (5) is odd or even
  2. Whether or not there are an infinite number of primes?

In the latter case knowing the truth is (near-)inextrictably entangled with a bunch of other capabilities, like the ability to do advanced mathematics. Whereas in the former it isn't. Suppose that before you knew either fact you were told that one of them was entangled in this way—would you still want to commit to paying out in a mugging based on it?

Well... maybe? ... (read more)

Individually logical counterfactuals don't seem very coherent. This is related to the "I'm an algorithm" vs. "I'm a physical object" distinction of FDT. When you are an algorithm considering a decision, you want to mark all sites of intervention/influence in the world where the world depends on your behavior. If you only mark some of them, then you later fail at the step where you ask what happens if you act differently, you obtain a broken counterfactual world where only some instances of the fact of your behavior have been replaced and not others.

So I th... (read more)

Random question I’ve been thinking about: how would you set up a market for votes? Suppose specifically that you have a proportional chances election (i.e. the outcome gets chosen with probability proportional to the number of votes cast for it—assume each vote is a distribution over candidates). So everyone has an incentive to get everyone who’s not already voting for their favorite option to change their vote; and you can have positive-sum trades where I sell you a promise to switch X% of my votes to a compromise candidate in exchange for you switching Y... (read more)

Just spitballing here: Assign each voter 100 shares for each candidate. To vote, each voter selects a subset of their shares to constitute their vote. Voters can freely trade shares.

Under this system, a voter would more highly value shares for candidates that are either very high or very low in their preference order (the later so as to exclude them from the vote). Thus, trades would look like each party exchanging shares about which they are themselves ambivalent to gain shares that are more valuable to them.

If you remove the proportional chances part, then it becomes a guessing game of which marginal votes actually matter.

Interesting! Hadn't thought of this approach. Let's see... Intuitively I think it gets pretty strategically weird because a) who you vote for depends pretty sensitively on other peoples' votes (e.g. in proportional chances voting you want to vote for everyone who's above the expected value of everyone else's votes; in approval voting you want to vote for everyone you approve of unless it bumps them above someone you like more), and b) you want to buy from your enemies much more than from your friends, because your friends will already not be voting for bad candidates. But maybe the latter is fine because if you buy from your friends they'll end up with more money which they can then spend on other things? I'll keep thinking.

I expect it to be difficult to generate adversarial inputs which will fool a deceptively aligned AI. One proposed strategy for doing so is relaxed adversarial training, where the adversary can modify internal weights. But this seems like it will require a lot of progress on interpretability. An alternative strategy, which I haven't yet seen any discussion of, is to allow the adversary to do a data poisoning attack before generating adversarial inputs - i.e. the adversary gets to specify inputs and losses for a given number of SGD steps, and then the adversarial input which the base model will be evaluated on afterwards. (Edit: probably a better name for this is adversarial meta-learning.)

Another thought on dath ilan: notice how much of the work of Keltham's reasoning is based on him pattern-matching to tropes from dath ilani literature, and then trying to evaluate their respective probabilities. In other words: like bayesianism, he's mostly glossing over the "hypothesis generation" step of reasoning.

I wonder if dath ilan puts a lot of effort into spreading a wide range of tropes because they don't know how to teach systematically good hypothesis generation.

I think you are overgeneralizing. We also see some mix of Dath Ilan, stories about Dath Ilan, stories about stories about Dath Ilan, and interactions between these, so all bets are off really.

I suspect that AIXI is misleading to think about in large part because it lacks reusable parameters - instead it just memorises all inputs it's seen so far. Which means the setup doesn't have episodes, or a training/deployment distinction; nor is any behaviour actually "reinforced".

I kind of think the lack of episodes makes it more realistic for many problems, but admittedly not for simulated games. Also, presumably many of the component Turing machines have reusable parameters and reinforce behaviour, altho this is hidden by the formalism. [EDIT: I retract the second sentence]

Also, presumably many of the component Turing machines have reusable parameters and reinforce behaviour, altho this is hidden by the formalism.

Actually I think this is total nonsense produced by me forgetting the difference between AIXI and Solomonoff induction.

Wait, really? I thought it made sense (although I'd contend that most people don't think about AIXI in terms of those TMs reinforcing hypotheses, which is the point I'm making). What's incorrect about it?

Well now I'm less sure that it's incorrect. I was originally imagining that like in Solomonoff induction, the TMs basically directly controlled AIXI's actions, but that's not right: there's an expectimax. And if the TMs reinforce actions by shaping the rewards, in the AIXI formalism you learn that immediately and throw out those TMs.

Oh, actually, you're right (that you were wrong). I think I made the same mistake in my previous comment. Good catch.

[+][comment deleted]Ω120

Humans don't have a training / deployment distinction either... Do humans have "reusable parameters"? Not quite sure what you mean by that.

Yes we do: training is our evolutionary history, deployment is an individual lifetime. And our genomes are our reusable parameters.

Unfortunately I haven't yet written any papers/posts really laying out this analogy, but it's pretty central to the way I think about AI, and I'm working on a bunch of related stuff as part of my PhD, so hopefully I'll have a more complete explanation soon.

Oh, OK, I see what you mean. Possibly related: my comment here.

I've recently discovered waitwho.is, which collects all the online writing and talks of various tech-related public intellectuals. It seems like an important and previously-missing piece of infrastructure for intellectual progress online.

Yudkowsky mainly wrote about recursive self-improvement from a perspective in which algorithms were the most important factors in AI progress - e.g. the brain in a box in a basement which redesigns its way to superintelligence.

Sometimes when explaining the argument, though, he switched to a perspective in which compute was the main consideration - e.g. when he talked about getting "a hyperexponential explosion out of Moore’s Law once the researchers are running on computers".

What does recursive self-improvement look like when you think that data might be t... (read more)

Perhaps a data-limited intelligence explosion is analogous to what we humans do all the time when we teach ourselves something. Out of the vast sea of information on the internet, we go get some data, and study it, and then use that to make a better opinion about what data we need next, and then repeat until we are at the forefront of the world's knowledge. We start from scratch, with a vague understanding like "I should learn more economics, I don't even know what supply and demand are" and then we end up publishing a paper on auction theory or something idk. This is a recurisve self improvement loop in data quality, so to speak, rather than data quantity.

What counts as self-improvement in the scenario governed by data?

You can grab the whole internet, including scihub and library genesis, and then maybe hack all "smart" appliances worldwide... and after that I guess you need to construct some machines that will perform experiments for you.

But none of this improves the machine's "self". With algorithms, the idea is that the machine would replace its own algorithms by better ones, once it gets the ability to invent and evaluate algorithms. With hardware, the idea is that the machine would replace its own hard... (read more)

In the scenario governed by data, the part that counts as self-improvement is where the AI puts itself through a process of optimisation by stochastic gradient descent with respect to that data.

You don't need that much hardware for data to be a bottleneck. For example, I think that there are plenty of economically valuable tasks that are easier to learn than StarCraft. But we get StarCraft AIs instead because games are the only task where we can generate arbitrarily large amounts of data.

RL usually applies some discount rate, and also caps episodes at a certain length, so that an action taken at a given time isn't reinforced very much (or at all) for having much longer-term consequences.

How does this compare to evolution? At equilibrium, I think that a gene which increases the fitness of its bearers in N generations' time is just as strongly favored as a gene that increases the fitness of its bearers by the same amount straightaway. As long as it was already widespread at least N generations ago, they're basically the same thing, because c... (read more)

A general principle: if we constrain two neural networks to communicate via natural language, we need some pressure towards ensuring they actually use language in the same sense as humans do, rather than (e.g.) steganographically encoding the information they really care about.

The most robust way to do this: pass the language via a human, who tries to actually understand the language, then does their best to rephrase it according to their own understanding.

What do you lose by doing this? Mainly: you can no longer send messages too complex for humans to und... (read more)

That doesn't actually solve the problem. The system could just encode the desired information in the semantics of some unrelated sentences - e.g. talk about pasta to indicate X = 0, or talk about rain to indicate X = 1.

I expected you to bring up the Natural Abstraction Hypothesis here. Wouldn't the communication between the parties naturally use the same concepts?

Same concepts yes, but that does not necessarily imply that they're encoded in the same way as humans typically use language.

Another possible way to provide pressure towards using language in a human-sense way is some form of multi-tasking/multi-agent scenario, inspired by this paper: Multitasking Inhibits Semantic Drift. They show that if you pretrain multiple instructors and instruction executors to understand language in a human-like way (e.g. with supervised labels), and then during training mix the instructors and instruction executors, it makes it difficult to drift from the original semantics, as all the instructors and instruction executors would need to drift in the sam... (read more)

Not being able to send messages too complex for humans to understand seems to me like it’s plausibly a benefit for many of the cases where you’d want to do this.

  stenographically

steganographically?

Ooops, yes, ty.

Greg Egan on universality:

I believe that humans have already crossed a threshold that, in a certain sense, puts us on an equal footing with any other being who has mastered abstract reasoning. There’s a notion in computing science of “Turing completeness”, which says that once a computer can perform a set of quite basic operations, it can be programmed to do absolutely any calculation that any other computer can do. Other computers might be faster, or have more memory, or have multiple processors running at the same time, but my 1988 A
... (read more)

Equivocation. "Who's 'we', flesh man?" Even granting the necessary millions or billions of years for a human to sit down and emulate a superintelligence step by step, it is still not the human who understands, but the Chinese room.

[-][anonymous]10

I've seen this quote before and always find it funny because when I read Greg Egan, I constantly find myself thinking there's no way I could've come up with the ideas he has even if you gave me months or years of thinking time.

Yes, there's something to that, but you have to be careful if you want to use that as an objection. Maybe you wouldn't easily think of it, but that doesn't exclude the possibility of you doing it: you can come up with algorithms you can execute which would spit out Egan-like ideas, like 'emulate Egan's brain neuron by neuron'. (If nothing else, there's always the ol' dovetail-every-possible-Turing-machine hammer.) Most of these run into computational complexity problems, but that's the escape hatch Egan (and Scott Aaronson has made a similar argument) leav... (read more)

[-][anonymous]30

Yeah good point - given generous enough interpretation of the notebook my rejection doesn't hold. It's still hard for me to imagine that response feeling meaningful in the context but maybe I'm just failing to model others well here.

It's frustrating how bad dath ilanis (as portrayed by Eliezer) are at understanding other civilisations. They seem to have all dramatically overfit to dath ilan.

To be clear, it's the type of error which is perfectly sensible for an individual to make, but strange for their whole civilisation to be making (by teaching individuals false beliefs about how tightly constraining their coordination principles are).

The in-universe explanation seems to be that they've lost this knowledge as a result of screening off the past. But that seems like a really predictabl... (read more)

Tho, to be fair, losing points in universes you don't expect to happen in order to win points in universes you expect to happen seems like good decision theory.

[I do have a standing wonder about how much of dath ilan is supposed to be 'the obvious equilbrium' vs. 'aesthetic preferences'; I would be pretty surprised if Eliezer thought there was only one fixed point of the relevant coordination functions, and so some of it must be 'aesthetics'.]

I don't think dath ilan would try to win points in likely universes by teaching children untrue things, which I claim is what they're doing.

Also, it's not clear to me that this would even win them points, because when thinking about designing civilisation (or AGIs) you need to have accurate beliefs about this type of thing. (E.g. imagine dath ilani alignment researchers being like "here are all our principles for understanding intelligence" and then continually being surprised, like Keltham is, about how messy and fractally unprincipled some plausible outcomes are.)

by teaching children untrue things

What specifically are you thinking of?

Half-formed musing: what's the relationship between being a nerd and trusting high-level abstractions? In some sense they seem to be the opposite of each other - nerds focus obsessively on a domain until they understand it deeply, not just at high levels of abstraction. But if I were to give a very brief summary of the rationalist community, it might be: nerds who take very high-level abstractions (such as moloch, optimisation power, the future of humanity) very seriously.

It seems to me that the resolution to the apparent paradox is that nerds are interested in all the details of their domain, but the outcome that they tend to look for are high-level abstractions. Even in settings like fandoms, there is a big push towards massive theories that entails every little detail about the story.

Though defining rationalist community as a sort of community of meta-nerds who apply this nerd approach to almost anything doesn't seem too off the mark. 

I think you need to unpack "trust" and "take seriously" a little bit to make this assertion.  I think nerds are generally  (heh) more able to understand the lossiness of models, and to recognize that abstractions are more broadly applicable, but less powerful than specifics.

I wouldn't say I trust or take seriously the idea of Moloch or the similarities between different optimization mechanisms.  I do recognize that those models have a lot of explanatory and predictive power, especially as a head-start (aka "prior") on domains where I haven't done the work to understand the exceptions and specifics.

There's some possible world in which the following approach to interpretability works:

  • Put an AGI in a bunch of situations where it sometimes is incentivised to lie and sometimes is incentivised to tell the truth.
  • Train a lie detector which is given all its neural weights as input.
  • Then ask the AGI lots of questions about its plans.

One problem that this approach would face if we were using it to interpret a human is that the human might not consciously be aware of what their motivations are. For example, they may believe they are doing something for altr... (read more)

I've heard people argue that "most" utility functions lead to agents with strong convergent instrumental goals. This obviously depends a lot on how you quantify over utility functions. Here's one intuition in the other direction. I don't expect this to be persuasive to most people who make the argument above (but I'd still be interested in hearing why not).

If a non-negligible percentage of an agent's actions are random, then to describe it as a utility-maximiser would require an incredibly complex utility function (becaus... (read more)

I'm not sure if you consider me to be making that argument, but here are my thoughts: I claim that most reward functions lead to agents with strong convergent instrumental goals. However, I share your intuition that (somehow) uniformly sampling utility functions over universe-histories might not lead to instrumental convergence.

To understand instrumental convergence and power-seeking, consider how many reward functions we might specify automatically imply a causal mechanism for increasing reward. The structure of the reward function implies that more is be

... (read more)

I've just put up a post which serves as a broader response to the ideas underpinning this type of argument.

I claim that most reward functions lead to agents with strong convergent instrumental goals

I think this depends a lot on how you model the agent developing. If you start off with a highly intelligent agent which has the ability to make long-term plans, but doesn't yet have any goals, and then you train it on a random reward function - then yes, it probably will develop strong convergent instrumental goals.

On the other hand, if you start off with a randomly initialised neural network, and then train it on a random reward function, then probably it will... (read more)

Agreed – I should have clarified. I've been mostly discussing instrumental convergence with respect to optimal policies. The path through policy space is also important.

Makes sense. For what it's worth, I'd also argue that thinking about optimal policies at all is misguided (e.g. what's the optimal policy for humans - the literal best arrangement of neurons we could possibly have for our reproductive fitness? Probably we'd be born knowing arbitrarily large amounts of information. But this is just not relevant to predicting or modifying our actual behaviour at all).

(I now think that you were very right in saying "thinking about optimal policies at all is misguided", and I was very wrong to disagree. I've thought several times about this exchange. Not listening to you about this point was a serious error and made my work way less impactful. I do think that the power-seeking theorems say interesting things, but about eg internal utility functions over an internal planning ontology -- not about optimal policies for a reward function.)

I disagree.

  1. We do in fact often train agents using algorithms which are proven to eventually converge to the optimal policy.[1] Even if we don't expect the trained agents to reach the optimal policy in the real world, we should still understand what behavior is like at optimum. If you think your proposal is not aligned at optimum but is aligned for realistic training paths, you should have a strong story for why.

  2. Formal theorizing about instrumental convergence with respect to optimal behavior is strictly easier than theorizing about -optimal behavior

... (read more)
[This comment is no longer endorsed by its author]Reply

1. I think it's more accurate to say that, because approximately none of the non-trivial theoretical results hold for function approximation, approximately none of our non-trivial agents are proven to eventually converge to the optimal policy. (Also, given the choice between an algorithm without convergence proofs that works in practice, and an algorithm with convergence proofs that doesn't work in practice, everyone will use the former). But we shouldn't pay any attention to optimal policies anyway, because the optimal policy in an environm... (read more)

I'm confused, because I don't disagree with any specific point you make - just the conclusion. Here's my attempt at a disagreement which feels analogous to me:

TurnTrout: here's how spherical cows roll downhill!

ricraz: real cows aren't spheres.

My response in this "debate" is: if you start with a spherical cow and then consider which real world differences are important enough to model, you're better off than just saying "no one should think about spherical cows".

I think that the theorising you talk about will be actively harmful for your ability to

... (read more)

Thanks for engaging despite the opacity of the disagreement. I'll try to make my position here much more explicit (and apologies if that makes it sound brusque). The fact that your model is a simplified abstract model is not sufficient to make it useful. Some abstract models are useful. Some are misleading and will cause people who spend time studying them to understand the underlying phenomenon less well than they did before. From my perspective, I haven't seen you give arguments that your models are in the former category not the latter. Presum... (read more)

Thanks for elaborating this interesting critique. I agree we generally need to be more critical of our abstractions.

I haven't seen you give arguments that your models [of instrumental convergence] are [useful for realistic agents]

Falsifying claims and "breaking" proposals is a classic element of AI alignment discourse and debate. Since we're talking about superintelligent agents, we can't predict exactly what a proposal would do. However, if I make a claim ("a superintelligent paperclip maximizer would keep us around because of gains from trade"), you

... (read more)

I'm afraid I'm mostly going to disengage here, since it seems more useful to spend the time writing up more general + constructive versions of my arguments, rather than critiquing a specific framework.

If I were to sketch out the reasons I expect to be skeptical about this framework if I looked into it in more detail, it'd be something like:

1. Instrumental convergence isn't training-time behaviour, it's test-time behaviour. It isn't about increasing reward, it's about achieving goals (that the agent learned by being traine... (read more)

I expect you to respond that being deontological won't get you to optimality. But I would say that talking about "optimality" here ruins the abstraction, for reasons outlined in my previous comment

I was actually going to respond, "that's a good point, but (IMO) a different concern than the one you initially raised". I see you making two main critiques.

  1. (paraphrased) " won't produce optimal policies for the specified reward function [even assuming alignment generalization off of the training distribution], so your model isn't useful" – I replied to th

... (read more)
We do in fact often train agents using algorithms which are proven to eventually converge to the optimal policy.[1]
At least, the tabular algorithms are proven, but no one uses those for real stuff. I'm not sure what the results are for function approximators, but I think you get my point. ↩︎

Is the point that people try to use algorithms which they think will eventually converge to the optimal policy? (Assuming there is one.)

Something like that, yeah.

And so this generates arbitrarily simple agents whose observed behaviour can only be described as maximising a utility function for arbitrarily complex utility functions (depending on how long you run them).

I object to the claim that agents that act randomly can be made "arbitrarily simple". Randomness is basically definitionally complicated!

Eh, this seems a bit nitpicky. It's arbitrarily simple given a call to a randomness oracle, which in practice we can approximate pretty easily. And it's "definitionally" easy to specify as well: "the function which, at each call, returns true with 50% likelihood and false otherwise."

If you get an 'external' randomness oracle, then you could define the utility function pretty simply in terms of the outputs of the oracle.

If the agent has a pseudo-random number generator (PRNG) inside it, then I suppose I agree that you aren't going to be able to give it a utility function that has the standard set of convergent instrumental goals, and PRNGs can be pretty short. (Well, some search algorithms are probably shorter, but I bet they have higher Kt complexity, which is probably a better measure for agents)

If a reasonable percentage of an agent's actions are random, then to describe it as a utility-maximiser would require an incredibly complex utility function (because any simple hypothesised utility function will eventually be falsified by a random action).

I'd take a different tack here, actually; I think this depends on what the input to the utility function is. If we're only allowed to look at 'atomic reality', or the raw actions the agent takes, then I think your analysis goes through, that we have a simple causal process generating the behavior but need

... (read more)

So in general you can't have utility functions that are as simple as the generator, right? E.g. the generator could be deontological. In which case your utility function would be complicated. Or it could be random, or it could choose actions by alphabetical order, or...

And so maybe you can have a little note for each of these. But now what it sounds like is: "I need my notes to be able to describe every possible cognitive algorithm that the agent could be running". Which seems very very complicated.

I guess this is what you meant by the "... (read more)

Which seems very very complicated.

That's right.

I realized my grandparent comment is unclear here:

but need a very complicated utility function to make a utility-maximizer that matches the behavior.

This should have been "consequence-desirability-maximizer" or something, since the whole question is "does my utility function have to be defined in terms of consequences, or can it be defined in terms of arbitrary propositions?". If I want to make the deontologist-approximating Innocent-Bot, I have a terrible time if I have to specify the consequences that corres

... (read more)
Instead I read it as something like "some unreasonable percentage of an agent's actions are random"

This is in fact the intended reading, sorry for ambiguity. Will edit. But note that there are probably very few situations where exploring via actual randomness is best; there will almost always be some type of exploration which is more favourable. So I don't think this helps.

We care about utility-maximizers because they're doing their backwards assignment, using their predictions of the future to guide their present actions to try to
... (read more)

To be pedantic: we care about "consequence-desirability-maximisers" (or in Rohin's terminology, goal-directed agents) because they do backwards assignment.

Valid point.

But I think the pedantry is important, because people substitute utility-maximisers for goal-directed agents, and then reason about those agents by thinking about utility functions, and that just seems incorrect.

This also seems right. Like, my understanding of what's going on here is we have:

  • 'central' consequence-desirability-maximizers, where there's a simple utility function that they're tr
... (read more)
x