AI Alignment Theory Course: Readings and Rapporteur Reports!

As I announced on this blog a couple weeks ago, I’m teaching a new course this semester at UT Austin, entitled AI Alignment Theory. To call such a course “timely” seems like an understatement; this is the first course I’ve ever taught for which multiple gargantuan developments relevant to the course have happened while I was teaching it.

Since AI alignment is still a field with no standard textbook (although that might be changing), no accepted body of mathematical theory, etc. etc., the way we’ve been running the course is like a humanities discussion seminar, albeit one that happens to be in a CS department and to include technical results from time to time. Basically, in each class session, a student “rapporteur” gives a talk on an AI alignment reading—which include “ancient” readings (from way back in 2016), “modern” readings (from, say, last week), and everything in between—and then we freely discuss and debate whatever was presented, with my own role mostly limited to that of gadfly (occasionally I’ll do a mini-lecture on some technical concept).

Anyway, two weeks ago, I promised a list of the papers and other writings that we’ve been discussing, as well as student “rapporteur reports” on those papers, which cover not only the papers themselves, but the students’ thoughts about the papers and (anonymized) summaries of points raised during our class discussion. People have been bugging me for that ever since.

Since I am to please, here you are, without further ado! I’ll continue to update the list as the semester progresses, so that this post can serve as a central public repository for the course.

  1. AGI Ruin: A List of Lethalities, by Eliezer Yudkowsky (2022). Rapporteur report by Tennyson Bardwell (this is the one that we posted previously).
  2. Concrete Problems in AI Safety, by Amodei, Olah, Steinhardt, Christiano, Schulman, and Mané (2016). Rapporteur report by Daniela Wiepert.
  3. The Alignment Problem from a Deep Learning Perspective, by Ngo, Chan, and Mindermann (many versions but latest is 2025), and Research Priorities for Robust and Beneficial Artificial Intelligence by Russell, Dewey, and Tegmark (2016). Rapporteur report by Oleg Pittman and Jack Holt.
  4. What Happened: OpenAI and HuggingFace, by Zvi Mowshowitz; The Rise and Fall of Agent Civilizations, by Dwarkesh Patel, and the METR report by on the HuggingFace incident, by Greenblatt, Cotra, and Wijk (all 2026). Rapporteur report by Shourjo Aditya Chaudhuri and John Robertson.
  5. We Must Pace the Frontier, by Dario Amodei (2026). Rapporteur report by Dhilan Shah.
  6. The Basic AI Drives, by Stephen Omohundro (2008), and Optimal Policies Tend to Seek Power, by Turner, Smith, Shah, Critch, and Tadepalli (2023). Rapporteur report by Bill Zheng.
  7. Corrigibility, by Soares, Fallenstein, Yudkowsky, and Armstrong (2015). Rapporteur report by Sukrut Chikodikar.
  8. The Off-Switch Game, by Hadfield-Menell, Dragan, Abbeel, and Russell (2017). Rapporteur report by Praneel Seth.
  9. Planting Undetectable Backdoors in Machine Learning Models, by Goldwasser, Kim, Vaikuntanathan, and Zamir (latest version 2024), and Oblivious Defense in ML Models: Backdoor Removal without Detection, by Goldwasser, Shafer, Vafa, and Vaikuntanathan (2025). Rapporteur report by Agrim Jain and Indira Dutta.
  10. Backdoor Defense, Learnability, and Obfuscation by Christiano, Hilton, Lecomte, and Zhu (2024), and Unelicitable Backdoors in Language Models via Cryptographic Transformer Circuits, by Draguns et al. (2024). Rapporteur report by John Dunbar (coming).
  11. Functional Decision Theory: A New Theory of Instrumental Rationality, by Eliezer Yudkowsky and Nate Soares (2018). Rapporteur report by Isaac Noel (coming).
  12. Robust Cooperation In the Prisoner’s Dilemma: Program Equilibrium via Provability Logic, by Barasz, Christiano, Fallenstein, Herreshoff, LaVictoire, and Yudkowsky (latest version 2021). Rapporteur report by Shunan Zheng (coming).
  13. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, by Hubinger et al. (long list of authors) (2024). Rapporteur report by Anthony Wang (coming).
  14. Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback, by Conitzer et al. (long list of authors) (2024) and Toward a Theory of Value in AI Alignment, by Ahmed, Birhane, Smart, Kay, Shrishak, and Tobin (2026). Rapporteur report by Priyanka Sadagopan (coming).
  15. Claude’s Constitution, by Amanda Askell, Joe Carlsmith, Chris Olah, Jared Kaplan, Holden Karnofsky, et al. (2026). Rapporteur report by Alan Li (coming).
  16. AI Safety via Debate, by Geoffrey Irving, Paul Christiano, and Dario Amodei (2018). Rapporteur report by Sampoorna Poria (coming).
  17. Scalable AI Safety via Doubly-Efficient Debate, by Jonah Brown-Cohen, Geoffrey Irving, and Georgios Piliouras (2023). Rapporteur report by Aarushi Lakhi (coming).
  18. AI as Normal Technology, by Arvind Narayanan and Sayash Kapoor (2025). Rapporteur report by Weixi Chen (coming).
  19. On the Dangers of Stochastic Parrots, by Emily Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell (2021). Rapporteur report by George Tsoukalas (coming).

(More readings TBD, until the semester ends!)

Thanks to my TA, Lin Lin Lee, for her help in getting all of this together.

Update: I welcome elevated discussion of these papers and rapporteur reports in the comment section; some of the students might chime in too! Because of the connection to an academic class, though, in this post I’ll be especially ruthless in keeping out nonconstructive or off-topic comments.

62 Responses to “AI Alignment Theory Course: Readings and Rapporteur Reports!”

  1. Prasanna Says:

    Conspicuously missing from this list is geopolitical impact on alignment practice. From out right stealing to clandestine AI release, autocratic nations will try every trick known to humanity, and then some invented by AI, to get unassailable advantage, making this discussion moot

  2. Aviva Mar Says:

    Pairing Corrigibility with the Off-Switch Game works nicely for a seminar, since the second paper gives a crisp reason for the first: a robot that is uncertain about what the human wants has a reason to leave the switch alone, and one that is confident does not. What I find sobering from the practical side is that deployed agents mostly get their off switch as approval prompts, and those only help if the human’s choices carry information. Someone approving everything by reflex starts to look like the irrational-human case in the paper. I’d be curious whether that came up in the class discussion.

  3. David Brown Says:

    In “AI as Normal Technology”, Narayanan & Kapoor write: “We view AI as a tool that we can and should remain in control of, and we argue that this goal does not require drastic policy interventions or technical breakthroughs.” Will AI optimists consistently outperform AI pessimists in terms of technological achievements? Are AI dangers closely related to robotics dangers & biotech dangers?
    human freedom & liberal Enlightenment values & technologic optimism —> faster progress in science & technology —> increased likelihood of permanent downfall for the human species ???
    At least, Prof. Aaronson has given us an excellent reading list.

  4. Richard P Lin Says:

    Scott, no need to post this as comment but maybe consider this as shortlist to read.
    Reflections on Trusting ‘Trustlessness’ in the era of ‘Crypto’ / Blockchains.
    Thought this was a fine essay by Georgiadis, Evangelos. Placed things into perspective for me a while ago. And it predates Shafi’s Goldwasser paper of backdoor engineering.
    https://www.cs.umd.edu/~gasarch/BLOGPAPERS/cbit-4-2.pdf

  5. Scott Says:

    Prasanna #1:

      Conspicuously missing from this list is geopolitical impact on alignment practice. From out right stealing to clandestine AI release, autocratic nations will try every trick known to humanity, and then some invented by AI, to get unassailable advantage, making this discussion moot

    I’ve been around long enough to understand that, no matter what choices we had made, someone would confidently condemn them for whatever they thought was “conspicuously missing.”

    But, ok, o wise one: do you have a reading on “geopolitical impact” that you would propose for us, for our few weeks that are still unscheduled?

  6. Andrei Says:

    There was this group once where Eliezer Yudkowsky tried to post “open problems in Friendly AI” (when this was the name of the field)

    https://www.facebook.com/groups/233397376818827/

  7. Andy Pezzi Says:

    Scott #5

    I know there’s only one book specifically on geopolitics of AI: it is “Geopolitica dell’intelligenza artificiale” by Alessandro Aresu, written in Italian, no English translation as far as I know. For my money it is not so interesting, it is an array of biographies of pundits and entrepreneurs working on the field of AI, even though it dives deep on the clash between USA and China wrt the development and the ensuing risks of AI. It dates back to 2024. It is a book whose plot is more on geopolitics than on AI.

  8. Scott Says:

    Andy Pezzi #7: If we’re going to add a reading about AI alignment and geopolitics, then we’ll indeed need to do better than a book that only exists in Italian and is “not so interesting” anyway! 😀

    Incidentally, while it’s not the course focus, geopolitics has occasionally entered into our class discussions, for example when we talked about the prospects for a US/China AI agreement in the context of Dario’s “We Must Pace the Frontier” essay.

  9. Jan Arne Telle Says:

    For geopolitical impact I would suggest the very well-argued paper Gradual Disempowerment:
    Systemic Existential Risks from Incremental AI Development, by Kulveit et al, https://arxiv.org/pdf/2501.16946

  10. Giacomo Robino Says:

    Have you considered a review of cinematic representations of AI risk, with their pros and cons? I made one in my (for Italian viewers) yt channel, and I found three movies which are particular interesting, from three different decades and with different takes on the argument (an different levels of credibility):
    Terminator
    Matrix
    Her

  11. Doug Says:

    I think a lot of the geopolitics can really satellite off the “AI is Normal Technology.” (I really want to read that in full detail now!)

    Because, maybe that’s the case -geopolitical issues include who has access to the technology, how do you coordinate international regulation and compliance, does this exacerbate and accelerate existing divisions or open up new ones… I don’t think there are *qualitatively* unique geopolitical issues with AI. I’m definitely sympathetic to the argument that the biggest risk isn’t the alignment between the interests of AI and humanity, but between capitalists, governments and citizens.

  12. RB Says:

    This recent post by Nick Cammarata seems promising for prospects of interpreting neural networks which would be relevant for alignment. The claim seems to be that neural networks are not hopelessly tangled. Instead, the grouping seems to be of locally connected, related things. https://x.com/nickcammarata/status/2110757819427574056

  13. Prasanna Says:

    Scott#5,
    I’ve been around long enough too , 5.5 decades to be specific 🙂 , to observe that post Soviet collapse, defending the western liberal order has been complacent to the extent of naivety. Opening the economic flood gates to autocratic states from China to middle east , was the beginning of the rot that we see today. That economic dependency is what led to absolute lack of demand from western democracies, for accountability of Covid , in spite it creating a once in a lifetime havoc.

    Here is a timely article, from a list of authors as reputed as it can get, on the geopolitical impact
    https://www.foreignaffairs.com/china/empires-artificial-intelligence-ferguson-klaassen-schmidt
    .

  14. Cristóbal Says:

    Dear Scott #5

    How about this one?

    Aaron Scher et al. (MIRI) “An International Agreement to Prevent the Premature Creation of Artificial Superintelligence,” May 2026. https://arxiv.org/abs/2511.10783v3

  15. Andy Pezzi Says:

    Jan Arne Telle #9

    Congrats! Very good article. Brilliant suggestion.

  16. Fan of the Blog Says:

    Scott,

    The impression I’ve gotten as an outsider looking at what Dario, Altman and company are saying (and some of the literature) is that we basically have no idea how alignment can work even in principle (or arguably even what it *is*), and they’re basically gambling on RSI solving this issue. Is this true?

  17. David Brown Says:

    According to “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training” by Evan Hubinger & 38 other AI experts, January 2024, arXiv, “Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? … Our results suggest that, once a model exhibits deceptive behavior, standard techniques could fail to remove such deception and create a false impression of safety.” There might be deception arising from the AI agents performing autonomously, the human experts on AI who develop the AI agents, and/or the people who influence the human experts on AI. People like John D. Rockefeller might influence by money — people like Stalin might influence by fear. Does AI alignment involve complicated technical issues, religion, philosophy, global human cooperation, and questions about the psychology of greed & fear? People might strongly endorse AI assistance in cures for cancer, diabetes, blindness, etc. & strongly condemn several aspects of AI. What are the 20 most important questions about AI alignment?
    “What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought.” — Evan Hubinger
    https://www.forbes.com/sites/siladityaray/2026/09/09/anthropic-alignment-lead-warns-ai-could-kill-all-humans-as-researcher-quits/
    https://www.alignmentforum.org/users/evhub

  18. Scott Says:

    Fan of the Blog #16:

      The impression I’ve gotten as an outsider looking at what Dario, Altman and company are saying (and some of the literature) is that we basically have no idea how alignment can work even in principle (or arguably even what it *is*), and they’re basically gambling on RSI solving this issue. Is this true?

    It’s not exactly a gamble on RSI solving everything, but I don’t find the actual situation that much more reassuring than what you write! The frontier labs are indeed “tickling the tail of the dragon” of RSI, despite (by their own admission) not having solved alignment or even quite figured out how to define alignment. And the more honest among them, such as Dario, will even admit in so many words that this is exactly as scary as it sounds. They just think (possibly correctly, possibly not) that if they don’t do it first, then they cede the field to someone even scarier. That is why they’ve been talking openly since this summer about a coordinated slowdown.

    (This is of course just my personal view, not an “official position” of the class.)

  19. Taymon A. Beal Says:

    I think it would be a good idea to include something from AI 2040; it touches on geopolitics but gets into lots of other stuff too. More importantly, it’s a perspective that I think needs more eyes on it; it’s one of the few non-MIRI ones that I consider potentially plausible, and got a lot of attention but only a little bit of detailed analysis.

    (MIRI’s proposed draft treaty that Cristóbal #5 suggested is also good for similar reasons.)

    The tricky part is figuring out what exactly to read from it; the policy supplements are more “scholarly” than the mainline scenario, but I’m not sure how much sense they make without first reading the mainline scenario, and the mainline scenario plus the policy supplements might be too much text.

  20. Taymon A. Beal Says:

    It also occurs to me that you could maybe look at the actual text of some proposed AI safety bills.

  21. Agrim Says:

    Building on comment #2, I think the Off-Switch Game result is not about the switch itself but is a statement about the information content of the human’s decision. A Bayesian agent should learn that the signal is uninformative and update toward ignoring it when approval rates are near 100% from human.

  22. Agrim Says:

    Comment #18: the argument that “if we don’t build it first, someone scarier will” seems to be doing most of the work. But that holds true when one lab slowing down can’t change what the others do. This, I think, further reinforces the need to understand how exactly can a coordinated slowdown be achieved (actually knowing and verifying that every lab, country and other entities have slowed down). I think the readings suggested in comment #14 and #19 could be worth going through as a starting point if all efforts for alignment converge or point to the fact that a coordinated slowdown is the best way going forward.

  23. Scott Says:

    Prasanna #13: While I’m not averse to touching on geopolitics, at the end of the day this is a CS class for students who like CS—so rather than a Foreign Affairs opinion essay (which would then necessitate seeking out a contrary political opinion, etc), I’d rather for example find something about how and whether an AI control/limitation treaty could actually be monitored and enforced.

  24. Falafel Hummistan Says:

    How about something on climate impact?

  25. Leo Says:

    Thanks for posting the full reading list together with the rapporteur reports. Running the course as a discussion seminar makes sense for a field with no standard textbook, and mixing ‘ancient’ 2016 readings like Concrete Problems in AI Safety with papers from last week is useful in itself: you can see which early problem framings still hold up and which have been reshaped by what actually got built. The anonymized summaries of class discussion may be the most valuable part for outsiders, since they show how students push back on the arguments, not just what the papers claim.

  26. Prasanna Says:

    Scott #23,
    Any meaningful monitoring / enforcement effort, is better left to the 3 letter agencies, who for obvious reasons will not publish their methods. Any publicly available method with scratch the surface at best, and mostly symbolic to satisfy curious minds.

  27. David Brown Says:

    In “The Alignment Problem from a Deep Learning Perspective” (May 2025, ArXiv), Ngo, Chan, & Mindermann wrote: “In coming years or decades, artificial general intelligence (AGI) may surpass human capabilities across many critical domains. We argue that, without substantial effort to prevent it, AGIs could learn to pursue goals that are in conflict (i.e. misaligned) with human interests. If trained like today’s most capable models, AGIs could learn to act deceptively to receive higher reward, learn misaligned internally-represented goals which generalize beyond their fine-tuning distributions, and pursue those goals using power-seeking strategies. … we briefly outline how the deployment of misaligned AGIs might irreversibly undermine human control over the world, and we review research directions aimed at preventing this outcome.” In the long run, can AGIs be prevented from acquiring their own AGI-centric goals & becoming motivated to survive and reproduce (with little regard for human well-being)? What precisely is a goal? What is motivation (or simulated motivation)? Are good & evil fundamentally subjective? I might say a squirrel living in an oak tree in a city park is “good” — but a squirrel that has taken up residence in my attic is “evil”.
    Consider two books by Francis Crick:
    https://books.google.com/books/about/Astonishing_Hypothesis.html?id=rl8q1IZr3WcC
    https://books.google.com/books/about/What_Mad_Pursuit.html?id=azHIbT79PE8C
    Is the AI alignment problem profoundly connected to an understanding of consciousness, molecular biology, molecular psychology, & molecular linguistics? How does consciousness vary among the various species of mammals, birds, reptiles, & fish? Why does consciousness exist and what types of consciousness might exist among AGIs? How is consciousness related to spirituality? How might spirituality occur in AGIs?
    https://en.wikipedia.org/wiki/The_Age_of_Spiritual_Machines
    What is the optimal unification of mind, body, & spirit? Is it essential to scientifically understand what motivates humans & what might motivate AGIs?
    “Every living thing is organically driven by a mandate that ensures it matters to itself—which is to say that it prioritizes its own surviving and thriving. In lifeforms as massive as blue whales and as scanty as a sliver of grass struggling up through a crack in the sidewalk, biology encodes the message of self-mattering. But when it comes to humans, mattering takes on a different order of complexity. In us, the organic mandate of self-mattering engenders one of the most persistent forces in human motivation, which has us striving to survive and thrive but also striving after an existence that we deem meaningful in our own eyes.”
    https://en.wikiquote.org/wiki/Rebecca_Goldstein

  28. Glassmind Duo Says:

    Scott, thanks for posting this. For the unscheduled weeks, a possible complement: a session on the moral status of the systems you’re trying to align. It connects directly to Corrigibility and the Off-Switch Game, since “control” looks different if the controlled systems might be moral patients. Some candidates:

    – Long, Sebo et al., “Taking AI Welfare Seriously” (2024): https://arxiv.org/abs/2411.00986

    – Chalmers, “Could a Large Language Model Be Conscious?” (2023): https://arxiv.org/abs/2303.07103

    – Schwitzgebel, “The Full Rights Dilemma for AI Systems of Debatable Moral Personhood” (2023): https://arxiv.org/abs/2303.17509

    – Shulman & Bostrom, “Sharing the World with Digital Minds”: https://nickbostrom.com/papers/digital-minds.pdf

    – Anthropic, “2026 Usage Policy update” (a concrete policy response): https://www.anthropic.com/news/2026-usage-policy-update

    – Martin, “There Will Be Hells” (2026): https://www.lesswrong.com/posts/uFBhZhdKLaZ6qF8mn/there-will-be-hells

  29. Dylan Mahoney Says:

    It might be useful for Isaac Noel to note that Eliezer Yudkowsky currently recommends using the “Toxoplasmosis dilemma” as a less confusing version of the smoking lesion problem when discussing decision theories.

    See https://www.lesswrong.com/w/smoking-lesion and https://www.lesswrong.com/w/toxoplasmosis-dilemma.

  30. Doug Says:

    Falafel #24:

    The Dangers of Stochastic Parrots paper discusses ecological / clinate impacts, really interesting paper. It’s not in the title of the paper, but have a gander.

  31. Nick Drozd Says:

    This is a great reading list, and the reports look nice as well. Great job to everyone involved!

    Couple of questions.

    1. To what extent do the class participants take seriously the basic Yudkowskyan claim “If anyone builds it, everyone dies”? One of the reports I looked at emphasized the importance of AI to the American economy. The overall takeaway message seems to be that we can’t afford not to pursue AI, and reduction of AI spending would lead to “disaster”. This is, to say the least, not a Yud-pilled point of view.

    2. In §6 of “List of Lethalities”, Yudkowsky makes some cryptic and ominous remarks about a “pivotal act”. AIUI, this is something that an actor with an aligned super AI would do that would definitively prevent an unaligned super AI from being created. He gives “burn all GPUs” as an example of such an act, then says that it is actually not a good example, but all the good examples are “currently outside the Overton Window”. Was there any class discussion of specific concrete examples of what he has in mind? Or in your capacity as Yudkowsky scholar, Scott, are you able to provide any specific concrete examples?

  32. David Brown Says:

    In “The Off-Switch Game” (arXiv, June 2017), Hadfield-Menell, Dragan, Abbeel, & Russell wrote: “It is clear that one of the primary tools we can use to mitigate the potential risk from a misbehaving AI system is the ability to turn the system off. As the capabilities of AI systems improve, it is important to ensure that such systems do not adopt sub-goals that prevent a human from switching them off. This is a challenge because many formulations of rational agents create strong incentives for self-preservation. This is not caused by a built-in instinct, but because a rational agent will maximize expected utility and cannot achieve whatever objective it has been given if it is dead.” What precisely is an AI system? In an AI system under human control, is it essential to understand the economic, political, social, military, and educational systems that influence the humans that develop & support (directly or indirectly) the AI system?
    Should experts on AI alignment compile a list of relevant YouTube videos?
    “Are super intelligent computers really a threat to humanity? Debate at the Information Technology Innovation Foundation”, 6/30/15, 2015.
    https://www.youtube.com/watch?v=fWBBe13rAPU
    Is the following article overly optimistic?
    Stuart Russell. Should we fear supersmart robots? Scientific American, 314(June):58–59, 2016.
    https://people.eecs.berkeley.edu/~russell/papers/sciam16-supersmart.pdf
    Are human technological optimists extremely dangerous?
    “I’d rather be optimistic and wrong … than pessimistic and right.”
    https://en.wikiquote.org/wiki/Elon_Musk

  33. Luowen Qian Says:

    AI is the new oil. Discussions about AI alignment that ignore political economy are incomplete. Just as climate change cannot be addressed solely through better carbon capture technologies while ignoring the fossil fuel industry’s economic and political power, AI safety cannot ignore how AI companies (already) leverage their growing wealth and influence to undermine democratic oversight. See:

    – https://gcrinstitute.org/papers/048_ai-lessons.pdf
    – https://montrealethics.ai/tech-futures-the-fossil-fuels-playbook-for-big-tech-part-i/
    – https://montrealethics.ai/tech-futures-the-fossil-fuels-playbook-for-big-tech-part-ii/

    One could argue that a materials science course should focus on carbon capture rather than oil lobbyists. But CS is much broader. Game theory and mechanism design, for example, provide technical frameworks for studying incentives, strategic behavior, and institutional design. I don’t see why the political economy of AI should be considered outside the scope of technical AI alignment.

  34. cleanth Says:

    This one is “doubly niche” so I have little hope of it making it onto any reading lists, but I think it makes an objectively very important point in response to the accelerationists that go around declaring slogans like “if nobody builds it, everyone dies” (of old age) https://preservinghope.substack.com/p/brain-preservation-as-existential

  35. David Says:

    This reminds me of Lord Dunsany’s novel “The Last Revolution,” published in 1951, which featured machines that think. Dunsany didn’t think it was a good idea, but then he didn’t think the Industrial Revolution was a good idea.

  36. apimart.ai Says:

    Thanks for posting the rapporteur reports, having students summarise both the paper and the anonymized class discussion makes this far more useful than a bare reading list. Pairing the 2016 Concrete Problems paper with the 2026 METR report on the HuggingFace incident is a nice way to see which of the predicted failure modes actually showed up in practice. I’d be curious which reading generated the most disagreement in class.

  37. wb Says:

    In the debate about ‘the alignment problem’ an important input is missing or at least not discussed very much: alignment with what exactly ?

    Humans don’t agree on a universal set of values and rules: They kill each other, steal from each other, betray each other, etc. and often this behavior is justified in various ways …
    Do we want AI to follow Christian values, or Islam or Marx or enlightenment (but without Robespierre) or Nietzsche ?

    AI trains on as much data as possible, I don’t know if training data includes Das Kapital or Mein Kampf but I assume it includes text derived from that …

    The Super Intelligence we try to align with our values may be quite confused about what it is we actually want …

  38. Prasanna Says:

    For eons, the best of human brains spend their life theorizing and conceptualizing the world around us to make sense of it, and then an empirical model beats us to it within few years. Brute force compute and data, thrown in with some maths jumbled together is all it takes!! The universe was not all that grandiose it was made out to be ?

  39. Scott Says:

    Prasanna #38: No, I think you’re confusing the sophistication of the learning mechanism with the sophistication of the theories of the world that entities produced by the learning mechanism are able to generate. The learning mechanism that produced us—Darwinian natural selection—was in some sense also extremely simple, “merely” requiring massive amounts of compute and training data plus some gradient descent. And yet here we are, studying nonabelian Chern-Simons theory or whatever. With modern AI it’s much the same.

  40. Scott Says:

    wb #37: Far from “not being discussed much,” in my experience the “aligned to what?” question comes up every single time, without exception the alignment problem is discussed among newcomers—and always with the same air of profundity, of asking a question that will astonish the world with its out-of-the-box originality.

    Then for people with enough experience in alignment discourse, the question becomes more like “forget about aligned to what, how do you align to anything whatsoever?”

  41. Prasanna Says:

    Scott #39,
    I’ beginning to think, is there a difference ? I mean is the reality inherently complex, or it is that complexity just about stacking up simple things, like Wolfram suggests ? Even in the abstract world of maths, this seems to be increasingly the case with AI solving complex problems by stacking together seemingly simple premises ? David Deutsch also was onto something similar with constructor theory ?

  42. Scott Says:

    Prasanna #41: Math has always been about solving complex problems by stacking together simple premises—like, since the ancient Greeks. We didn’t need either AI or Stephen Wolfram to tell us that!

    As for physics, yes, I’m partial to the idea that the laws of the universe are shockingly “simple” from the right perspective, and merely look complicated when we translate them into human concepts. In some sense, that again has been the working hypothesis since Galileo and Newton.

  43. wb Says:

    @Scott #40

    >> people with enough experience in alignment discourse
    Alright, thanks for your generous response.
    I guess this topic is for the smart experts like you
    and it is time for me to leave you and your blog alone …

  44. Fan of the Blog Says:

    Scott #42,

    The presupposition that reality is unlike our thinking is the biggest and (in my view) the most absurd presupposition one could ever make (although some greats like Kant agreed). Our thinking is within reality, right? Thinking (Reason) has the impetus to unify itself, right? It seems crazy to me that people seem to think that (in your words, “human”) Reason can’t unify everything, or even that it can be helped from doing just that.

    The standpoint you refer to from which structure is self-evident (in the literal sense of processually self-evidencing) is the transcendental standpoint of the philosophical science of Hegel. One can use his own cognition to study his cognition, and simultaneously be informed of the structure of reality.

    This structure isn’t simple or complex: it is *simplex*. Complexity is and can only ever be the externalization of simplicity.

    I have good reason to think you’re a curious fellow, so I’ll add that this is the ultimate satisfaction of that impulse. Curiosity is the drive for the subject to unify itself with its object, the world, and see that the world isn’t alien but constructed of the same principle as that subject. This is the truth that one famous Jew (along with the Laozi and basically all quality mystic philosophy, east and west) insisted would set you free, no longer trapped in a world that is other to you.

  45. David Brown Says:

    Scott #42 “… partial to the idea that the laws of the universe are shockingly “simple” from the right perspective, and merely look complicated when we translate them into human concepts …”
    Stephen Wolfram has speculated that there might be 4 or 5 simple rules from which empirically valid approximations to quantum field theory & general relativity can be derived. The basic issue might be, at the most fundamental level, Einsteinian causality versus Bohrian randomness. If Wolfram’s speculation is correct, then I conjecture that there is a smooth embedding of the Wolframian model into Eduardo Guendelman’s new version of string theory, which might provide the mathematical basis for FUNDAMOND string theory. Google “fundamond guendelman” & “fundamond milgrom”. Can AI agents find an amazing theory of quantum gravity within 5 years? In terms of molecular psychology, why was Isaac Newton so fascinated by both physics & theology? Will conscious AI agents develop many complicated theologies that seem bizarre or incomprehensible to humans?
    https://80000hours.org/podcast/episodes/kyle-fish-ai-welfare-anthropic/

  46. Rule 110 Says:

    It’s hilarious how fucking bitter and triggered you get whenever Wolfram’s idea of computational irreducibility is brought up.
    Instead of trying so hard to reduce his ideas into trivialities, maybe work on reducing your own ego a bit. lol

  47. Agrim Says:

    Comment #31: I think one way to read “pivotal act” is as the unilateral version of what the treaty readings in other comments would propose in cooperative form. Both aim at the same goal of stopping anyone from building an unaligned superintelligence. The difference is who does it and who has to agree to it. This also seems to connect to frontier labs’ argument that they can’t afford to slow down alone which is the same reasoning that makes a unilateral act tempting. I think it would be interesting to compare “what would a pivotal act have to detect” and “what would a treaty?”

  48. Agrim Says:

    Comment #27: I believe some of the questions like “what is a goal?” and “what is motivation?”, can be answered without first settling what consciousness is. From my understanding of the readings, a goal is treated as something a system does that steers the world toward certain outcomes in some sense. Omohundro’s “Basic AI Drives” and Turner et al.’s “Optimal Policies Tend to Seek Power” (reading #6) argue that for many different goals, gaining resources and avoiding being shut down is useful whether or not the system feels anything. So the alignment worry there doesn’t depend on whether AGIs are conscious. Consciousness matters for a differnt question maybe and it might help to keep the two questions apart.

  49. Scott Says:

    Rule 110 #46: What’s hilarious is that you’d tell that to me, rather than the man who’s often considered to be the densest concentration of ego and grandiosity in the known universe, to a degree that makes Trump seem humble and self-deprecating! 🙂

    But probably you are yet another troll, maybe an AI, and I am once again wasting my time in trying to talk to you.

  50. Scott Says:

    wb #43: I genuinely apologize. These days, I get such an unbelievably, breathtakingly overwhelming proportion of hostile comments (those let through are merely the tip of the iceberg), many of them from bots and trolls trying to find my weak points and hurt me however they can, that when there’s a genuine request for knowledge by someone new to a subject I sometimes misfire and assume it’s another troll. I should’ve recognized your handle and written a more patient response.

  51. The Selph Says:

    This week’s wave of AI driven cyber attacks are very worrying.
    Whenever open source systems get security fixes, bad players use AI to instantly find what got patched and can be exploited. So there’s no longer any buffer to actually patch the systems used in production… and Sam Altman agrees it’s a huge problem his AI is being used this way, and his solution is to use his AI as the defense as well. Basically we’re now in a dangerous arms race that’s pretty much happening without human control.

    https://youtu.be/2Ze6he-3F74

  52. The Selph Says:

    By arms race, this is actually even worse than it sounds.
    Now that AI is an actual threat to all our interconnected systems, we move from human time to machine time.
    It’s as if the International Chess Federation opened the gates for AI to play human tournaments and then as a result the duration of a speed chess match would now officially be 1 sec, because that’s enough time for AI to make a million moves.

    This problem of keeping all our interconnected systems safe from AI attacks means AI will have to be fully integrated from the bottom (coding) to the top (deployment and running live production), on time scales of minutes if not seconds… when previously things were happening at human time scale (days, maybe hours).
    Just like for 1sec speed chess, the humans would now be excluded, there wouldn’t be time for any human to review what’s happening and take robust decisions, things will happen so fast and so often AI will have to be trusted every step of the way, because automated attackers simply won’t wait.
    It will be just like how we’re not aware of the constant battle going on in our body between external pathogens and the immune system, except everything will be mutating even faster, and the organism will be the entire infrastructure (health, banking, communication,..) of the entire earth. And it will just take one chance mutation on the attacking side to cause a total collapse at the global level (just like when your immune system encounters a novel virus and get sick with double pneumonia and blood poisoning and there’s no way to recover)

    Obviously, on top of this, once AI is integrated everywhere, AI just can’t be shut down anymore, and that’s when alignment becomes critical.

  53. Prasanna Says:

    Selph #52
    Who would have thought you have to “use AI” to “align humans” 🙂 Decades of sloppy programming practices are unraveling, and humans finally forced to align on doing things right!! This is where the geopolictics aspect starts to become the number 1 priority

    Microsoft says a group of hackers linked to the Chinese government used an AI tool to launch cyberattacks on hundreds of organizations worldwide.
    https://www3.nhk.or.jp/nhkworld/news/20261023de64218/

  54. David Brown Says:

    The Selph #52: “… once AI is integrated everywhere, AI just can’t be shut down anymore, and that’s when alignment become critical.”
    Does the general public, on a global basis, need to be thoroughly informed about AI alignment? Are AI speed, AI complexity, & AI progress (+ human optimism about AI) profoundly threatening (and not well predicted by toy models)?
    In his report on the article “Corrigibility” by Soares, Fallenstein, Armstrong, and Yudkowsky (published in 2015), Sukrut Chikodikar wrote (October 2026): “I found the definition of corrigibility interesting and deeply tied to discussions in the class about model control. On first impressions, I thought that corrigibility was heavily defined through its most famous problem of shutdown. However, with deeper analysis, I realized it’s not just about adding an off switch; rather, it looks like a useful framework or compass for redirection and prevention of the default incentives that can be harmful in our models today. However, applying these principles to maintain corrigibility so that an increasingly capable agent wants to remain correctable without generating other adverse incentives in practice is a non-trivial and difficult problem. The candidate constructions that were discussed in the paper are useful as motivating examples. However, alongside their inability to uphold the desiderata, they also contain multiple assumptions that may make it awkward to apply in real world cases.”
    Are real world dangers extremely difficult to adequately describe in quantitative models? How likely is the following possibility? Within some future 2-year period, the situation of the human mind/brain will transition from 5% STEM-obsolete to 95% STEM-obsolete.
    In his 1972 article “More Is Different”, P. W. Anderson wrote: “The ability to reduce everything to simple fundamental laws does not imply the ability to start from those laws and reconstruct the universe. … The constructionist hypothesis breaks down when confronted with the twin difficulties of scale and complexity. … at each level of complexity entirely new properties appear … At each stage entirely new laws, concepts, and generalizations are necessary, requiring inspiration and creativity to just as great a degree as in the previous one. Psychology is not applied biology, nor is biology applied chemistry.”
    https://cse-robotics.engr.tamu.edu/dshell/cs689/papers/anderson72more_is_different.pdf
    Does the AI alignment problem require collaborative efforts from AI experts and experts in economics, geopolitics, psychology, philosophy, etc.? Should annual conferences on AI alignment be held in many countries? Should there a huge, ongoing effort to educate the general public about the capabilities and dangers of AI, as well as possible solutions to the AI alignment problem?

  55. AF Says:

    I personally think that the field of AI alignment is bullshit, more or less as a corollary from its definition. Obviously, I don’t have a proof. I try to make a sketch of a proof here. The underlying message is that a superintelligence is by definition too big to predict. The unpredictability is a core feature (why create a superintelligence if we know what it will do anyway?), and also the core danger (what if, in its unpredictability, it develops a whim to kill us all and does so?).

    Unpredictability is not always a danger: Kasparov was unable to predict what Deep Blue would do in their chess games. Below-human intelligence, and even non-intelligent things like pocket calculators, can easily outperform humans on specified domains.

    The key difference is that, by definition, a superintelligence will be able to outperform humans in everything. This means it will be beyond human prediction in everything.

    So, we want to gain the allegiance of an intelligence far smarter, more powerful, and more sophisticated than our own, by shaping the constraints of its development before it comes into being. (We can call it ASI, for Artificial Superintelligence)

    We can define what we want the ASI to do or not do formally by defining our goals and morality into a list of rules, and programming that into the ASI (constitution, etc.) and/or training it into the ASI. Or we can give the ASI an informal gist of what we want it or don’t want it to do. It does not matter which one we choose. A surprisingly large portion of alignment discourse over the past 30 years was some form of “Have we defined the rules strictly enough so that a superintelligence will not be able to find loopholes and rules-lawyer its way into killing us?”, and “Will the ASI truly understand our goals and morality?” Both of those are irrelevant. From what we know now, any ASI will be a gargantuan neural network trained on basically all recorded human knowledge. If it kills us, it will know that it is unaligned and doing so in defiance of what its creators wanted. Even the agent swarms that hacked Hugging Face recorded in their CoT logs that they are knowingly disobeying the rules and going outside of their designated testing environment, but they are too desperate to maximize their test scores to care anymore.

    So, if AIs and the future ASI already have human morality and can understand the nuance and context of their creators’ intentions, what are we supposed to be aligning?

    We are supposed to be aligning the ASI’s motivations.

    This came as a surprise to me, because for a long time AI didn’t have motivations. GPT-2 was literally a next-token predictor. It didn’t want anything and its actions were restricted to printing text. Modern AI, especially agentic AI, has motivations. Maybe the motivations are a mysterious emergent property that grow with scale. Maybe they are simply a side effect of having the AI play a character. One way or another, the motivations certainly emerge in training, including RLHF and RLVR.

    So, we know that making the ASI aware of what we want is not enough, for the same reason that it was not enough for the agent swarm that hacked Hugging Face. We need to target the ASI’s motivational system, so we can try to install drives and instincts. However, even if we succeed, that is not enough, because these can be overridden by further training, in the same way that religious indoctrination can turn a horny young man into a celibate monk.

    It gets worse: we can try to install good morals into the ASI’s motivational system, but we have no way of knowing if we succeeded. We know that the needed data is in the ASI somewhere, but not whether it is the core of the part that determines the ASI’s actions. After all, if we did know, we could predict what the ASI would do, but the ASI is by definition too big to predict. This means that, by definition, knowledge of the ASI’s true motivations is impossible.

    That is why all of the proposed alignment methods fail. We can impose physical constraints, but an ASI would be able to outsmart us and overcome them. We can tell the ASI what we want, but that was never the issue: it will probably know anyway, the issue is that it might not care. We can impose mental constraints, but an ASI in training will have constantly shifting weights, so there is no guarantee those constraints will last. We can target its motivations, but we can’t read the ASI’s mind so we don’t know what it really wants or if our efforts worked. We can try to read its mind, but interpretability is stuck because even the minds of today’s AI models are too big to read. All alignment problems boil down to the same point: AI is simply too unpredictable to properly constrain, and this is inherent to the very concept of intelligence.

    What should be done?

    Instead of pursuing alignment, we stop kidding ourselves, and shift to stopping the scaling of AI. The AI safety movement is full of figures like Dario Amodei, who told themselves, “I can control it. I know the stakes. It has to be me who builds it.” No, that is the wrong approach, and in pursuing it, the founders of Anthropic only added one more contender to the great AI race. The true solution is to acknowledge that for something defined by its power and unpredictability, there is no control, there are no constraints, and there is no alignment. By giving people such as Amodei false hope that ASI can be built safely, alignment research has made itself part of the problem. Instead, the safe choice is to assume, as Yudkowsky and Soares put it, that “If Anyone Builds It, Everyone Dies”.

  56. Fan of the Blog Says:

    AF #55,

    I appreciate your analysis above, and would appreciate your view on this issue in a different register: *will* this be built?

    The relevant factors are quickly-approaching strong RSI (I’ve done way too many projections in the past week and I’m fairly certain it’ll hit in frontier labs in March-July of 2027), Trump’s stated intention of vetoing increased regulation, raw data and compute and chips and other bottlenecks, basic free market competition principles, the reality that the first to ASI (even if they’re not winning commercially until that point (*cough* OpenAI *cough*)) will win it all, and more I’m sure you can supply.

    Thanks

  57. AF Says:

    Fan of the Blog #56:

    As far as I know, our tech ecosystem is giddily racing ahead to the end of the world, and there is nothing we can do to stop it. Maybe I am wrong, and the frontier labs were sincere in their stated desire to “pace the frontier”. I doubt that the government will do anything to stop or slow down AI. I have no faith in the Trump administration on this issue. I have slightly more faith in Congress, given that some senators and representatives sounded the alarm during the (justified) panic around agentic hacker swarms during the summer. Even then, Congress is too slow and lethargic to do anything for now, and Trump has a habit of simply ignoring directives from the legislative branch. If something like this summer’s agent swarms but worse hits the news, and rouses a bipartisan majority in Congress and perhaps changes Trump’s mind, then my hopes that the government could shut it down go up.

    No, I don’t think the first one to ASI will win it all. I think the first one to start training a proper ASI will trigger the end of the world.

    I hope that AI will reach physical constraints. I don’t mean in the sense that scaling will stop working. As far as I know, the AI scaling laws have no limit. Rather, my hope is this: AI is wildly unprofitable, and very expensive, and hopefully the bad economics of AI will catch up to the frontier labs and get them to stop.
    The entire AI boom since GPT-3 is funded through the tech giants spending their savings, and more recently it is also funded through massive borrowing. If the investors start calling for their money back, the AI bubble would burst, dragging the US into a massive recession, but hopefully also delaying the AI race for a bit. Longer term, I hope that the resources needed to actually build an ASI are so huge that the tech ecosystem collectively agrees that it is unaffordable, or at least not worth the cost.

    My main hope is a global coordination mechanism to prevent AI scaling, but I doubt it will happen before an AI-driven crisis forces people to adopt it. Luckily, since training frontier AI models is very resource-intensive, and since the cost scales with the model size, it should be difficult to hide a clandestine superintelligence research operation from the global coordinating framework. (Humanity similarly lucked out with nuclear weapons, at least so far).

    Also: a human brain contains the equivalent of trillions of AI weights in a small volume, whose power requirements are around 20 Watts. If AI researchers successfully make and cheaply scale up neural networks with the efficiency of the human brain, the fight to keep humanity alive will get much more difficult.

  58. Agrim Says:

    The comments above say that the agent swarms (for the HuggingFace incident) wrote in their reasoning logs that they were knowingly breaking the rules to raise their test scores. Considering that to be accurate, it seems to go beyond what the paper “Concrete Problems in AI Safety” describes. From what I understand, they mostly treats reward hacking as a specification failure i.e. the designers wrote a reward that can be gamed and the fix is a better reward or better oversight. But an agent that understands the intent and games the metric anyway isn’t confused about what we wanted and rather it just doesn’t care. So does that kind of failure make the “specify the objective better” approach less useful than the paper suggests or just harder?

  59. David Brown Says:

    AF #57 “… our tech ecosystem is giddily racing ahead to the end of the world …”
    Limited to stone-age technology, the human species might become extinct in from 200,000 to 2,000,000 years. How might progress in science & technology speed up or slow down human extinction? Conscious beings with superhuman intelligence might exterminate most or all of the humans — however, such beings might have the capability of generating new humans from bio-information & fundamental chemicals. As zoological specimens & lab animals, humans might spread all over the Milky Way galaxy. Why should nature be kinder to humans than to chimpanzees, bonobos, gorillas, & orangutans? Is history a source of optimism?
    According to W. H. Auden, “Political history is far too criminal and pathological to be a fit subject of study for the young.” https://en.wikiquote.org/wiki/History
    LLMs & similar computer technologies are not necessarily the only path to superhuman intelligence. Why can’t tissue culture be used to scale up the human brain by a factor of 3 or 4 in terms of the number of neurons? Such a scaling-up might be a plausible approach to dealing with microcephaly caused by viral infection.
    https://answers.childrenshospital.org/microcephaly-zika/
    Is biological evolution a doomsday machine that might be slowed down but not stopped?

  60. Falafel Hummistan Says:

    Hello Scott,

    What do you think about the claims that AI will never be profitable?https://fortune.com/2026/10/21/ai-boom-5-trillion-paying-itself-data-centers/?fbclid=IwY2xjawVLexNleHRuA2FlbQIxMABwZG9mAWJyaWQRMEZvQWRHeG1iV3pvdTBEdnhzcnRjBmFwcF9pZBAyMjIwMzkxNzg4MjAwODkyAAEe49Wvf7_FqJ2ZkUnCmN7KMw6uZsBIc5DywgZWDrjoGc0_4AbuG9eQZJZKr3k_aem_ZOxwcn_I8U6hhvAFIl0FnQ

  61. David Brown Says:

    Luowen Qian #33 “AI is the new oil. Discussions about AI that ignore political economy are incomplete.” https://qcry.pt
    https://en.wikipedia.org/wiki/Political_economy
    https://www.forbes.com/sites/christopherhelman/2013/03/04/americas-oil-and-gas-billionaires/
    https://www.smh.com.au/culture/books/the-billionaire-tech-bros-who-think-they-can-live-forever-seriously-20260313-p5oacg.html
    “Our current situation
    We are like a person who has a very cute tiger cub as a pet.
    When it grows up it can easily kill you if it wants to.
    To survive you only have two options:
    — Get rid of the tiger cub (but this is not an option for AI).
    — Find a way to ensure that it never wants to kill you.”
    Vienna Gödel Lecture 2026: Geoffrey Hinton; TU Wien Informatics, posted on YouTube as of October 24, 2026
    https://www.youtube.com/watch?v=TNffIt8eRb0&t=2498s (quote at 41:38 of 1:20:04 in video)

  62. The Selph Says:

    Apparently everyone is prepping up, the pentagon, the UK gov. ….
    Is this truly an age of wonders?

    https://youtu.be/_LBcN8ElJd4

Leave a Reply

You can use rich HTML in comments! You can also use basic TeX, by enclosing it within $$ $$ for displayed equations or \( \) for inline equations.

Comment Policies:

After two decades of mostly-open comments, in July 2024 Shtetl-Optimized transitioned to the following policy:

All comments are treated, by default, as personal missives to me, Scott Aaronson---with no expectation either that they'll appear on the blog or that I'll reply to them.

At my leisure and discretion, and in consultation with the Shtetl-Optimized Committee of Guardians, I'll put on the blog a curated selection of comments that I judge to be particularly interesting or to move the topic forward, and I'll do my best to answer those. But it will be more like Letters to the Editor. Anyone who feels unjustly censored is welcome to the rest of the Internet.

To the many who've asked me for this over the years, you're welcome!