The Age of Wonders and Terrors
Twenty years ago, when the idea of AI taking over the world in our lifetimes still struck most of us as the unconstrained fantasy of those who knew too much science fiction and too little science, many of us would say things like:
Look, the part of the story that’s wildly implausible is that a recursively self-improving superintelligence will just explode from some hacker’s basement and take over the world without warning. If it’s going to happen, we’ll see many warning signs first. We’ll see, I dunno, AI agents breaking out of containment, conspiring with each other to hack websites, in fanatical pursuit of whatever strange goals they have. And then, of course, we’d see major math problems getting solved by AIs—even the Clay Millennium Problems. That will be the time to panic! Wake me up when that happens!
Twenty years ago, the above was a take that even my most conservative, skeptical colleagues in academic CS would’ve gladly endorsed.
If you want to know my current take, you simply start with the one above, then update on the fact that the wild prophecies have come true. The first rumblings, I’d say, came a decade ago with AlphaGo, they got noticeably louder with LLMs and coding and reasoning agents, and they’ve accelerated this summer and fall into a crescendo of wonders and terrors that one needs to be a particular kind of idiot to deny.
I recoil from the neverending shell game where you say “oh sure, of course AI can now [escape from its sandbox / solve Millennium Problems / whichever dramatic thing it most recently did], no one ever denied that [I did deny it], wake me up when AI does [thing AI hasn’t yet done but is going to do next year], that’s when I’ll reevaluate my whole worldview [no I won’t].” Where no matter how fast the rollercoaster accelerates, even after your whole familiar world has vanished behind you, you’re still inventing reasons why it doesn’t count.
My position on AI is merely the conservative, skeptical position of 2006, updated with intellectual honesty for the reality of late 2026. And that position, if you need me to spell it out, is as follows:
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
It seems to me that the Singularity has already started; it’s just wildly unevenly distributed. Yes, I still unload the dishwasher and clip my toenails. On the other hand, in whatever years I have left, I don’t expect that I’ll ever again prove a theorem because I’m actually needed to prove it. If I do, it will only be for my or others’ enjoyment or edification.
The test is this: if we took the news of these past few weeks and sent it back in time twenty years, would I agree that it looked like the beginning of an AI Singularity? The intellectually honest answer is: yes, absolutely. But then that’s all we need. No backsies.
I feel like it would be healthy for everyone to stop grinding their ideological axes, their sentiments about Dario Amodei or Sam Altman, for long enough simply to acknowledge that the wonders and terrors are here. They couldn’t be here more clearly if the sky had turned reddish-orange like in the Matrix movies.
It’s here clearly enough that, when I put my kids to sleep at night, I now feel it in the pit of my stomach: what sort of future can they possibly have? What could they learn today that could possibly be relevant to that future? (Yesterday, my 13-year-old daughter joked unprompted that, if she wants to become a mathematician, it now looks like she has maybe two more weeks.) Certainly when my grad students want to discuss what sort of careers might await them on graduation, I no longer have any clue what to tell them.
Maybe it will help if I briefly switch topics. Ever since my wife and I moved to Austin, I’ve sometimes gotten some version of the following query: “How can you, as both a Jew and a skeptical scientist, possibly get along well with all those evangelical Christians down there in Texas? Sure, they might seem super friendly to Jews, but don’t you understand that that’s only because of the special role Jews play in their eschatology—when Christ will return in glory, and you’ll either accept Him as Lord or else roast in hell for eternity?” I stare at them and say: “wait, so I get to accept Christ only after He returns? What a great deal! How could I possibly have any objection to that?”
For anyone who says AI doom sounds like an apocalyptic religion, that the rationalists/Singulatarians seem like a Bay Area cult, that Eliezer Yudkowsky gives off the vibes of a messianic prophet: yes, yes, and yes. But crucially, today you’re no longer being asked to believe in arguments and extrapolations, but only in the front-page news. Accepting the reality of the coming machine god after it’s solved Navier-Stokes and dozens of other longstanding open math problems (while dramatically ramping up in capability every month), is sort of like accepting Jesus after he’s returned to earth on the gleaming cloud. It’s the epistemic bare minimum.
Yes, there’s still enormous uncertainty about what the rest of our lives will look like, but as far as I can tell, there’s no longer any real uncertainty that it’ll all mostly revolve around AI, and the extent to which we succeed or fail at directing its power toward human flourishing.
By any accounting that doesn’t stack the deck, Eliezer Yudkowsky was right about what the greatest challenge facing civilization in our lifetimes was going to be, and you and I were wrong about it. Why I was wrong is a question I’ll ask myself every day in whatever time remains. But, you know, at least I updated once the prophesied wonders and terrors actually started arriving! If you haven’t done likewise, why haven’t you?
As you presumably know by now—it was the talk of the nerd internet all week—the Navier-Stokes Millennium Problem appears to be solved, with crucial contributions from both humans and AI, albeit with a tangled dispute about exactly what happened and what ought to have happened. The answer, which an OpenAI model has apparently verified in Lean, is that (as many mathematicians suspected lately) there’s smooth initial data that leads to a singularity in finite time, at least if a smooth external force is applied (the case with no external force is still unresolved). This problem was supposed to carry a $1 million prize, except that OpenAI says they have no interest in collecting the prize and it’s unclear if any human is eligible to collect instead. OpenAI burned at least ~$15 million in compute to produce its 166-page solution, which probably hasn’t yet been read and understood by any human.
See here for the Quanta article, and here for NYU mathematician Tristan Buckmaster’s account of the role played by himself and Levent Alpöge of Anthropic, which substantially differs from the OpenAI’s account (you can read a response from OpenAI’s Sebastian Bubeck here). It’s agreed that everything built on an approach pioneered in recent years by the human mathematicians Diego Córdoba and Luis Martínez-Zoroa.
My purpose here is not to adjudicate the dispute. Yes, in swooping in with vastly greater resources once it had gotten wind of progress of Navier-Stokes, OpenAI seems to have acted in a way that some might describe as “unsportsmanlike.” No, I don’t find it plausible that OpenAI’s models meaningfully benefitted from being trained on Buckmaster and Alpöge’s chat logs. But this leaves a crucial question unanswered: what exactly did OpenAI know about Buckmaster and Alpöge‘s work and when did it know it?
Anyway, as Zvi points out, it’s easy to get hung up on the details and lose sight of the high-order bit: namely, that it seems safe to say that human mathematicians are forevermore dethroned as the main theorem-proving entities on planet earth. I feel privileged to have had the traditional kind of career in theoretical computer science in the last decades when that was possible.
If we were just talking about Navier-Stokes, you might accuse me of jumping to conclusions here. But we’re not. In the areas I know best (such as quantum complexity theory), and presumably other areas as well, there’s now a deluge, with longstanding open problems both major and minor falling by the day.
Go to the arXiv or ECCC. Pretty much all the papers that I’d be interested in now include “AI statements” near the acknowledgments (as this is often the central thing I want to know, I wish I didn’t need to scroll to the end of the paper to find it!). These statements can range from “our main result came entirely from GPT-6, but we understood it and take responsibility for it,” to “the results came from an interaction between the human authors and AI” to “we used AI, but only for proofreading and other incidental things” to (mad props!) “the author did not use AI for anything.”
If you talk right now to editors or program committee chairs, it’ll remind you of those ominous scenes from the Lord of the Rings movies where the men of Gondor or Rohan or whatever are grimly fortifying their walled city against the expected onslaught of 50,000 orcs. Reviewing will have to be done partly by AI, because otherwise there’s no way to handle the orc army: the reviewers can’t unilaterally disarm.
Anyway, here’s a small sampling of the significant AI-proved or -assisted results from, like, the last month, besides Navier-Stokes—restricting myself to those that solved longstanding open problems I had previously known or cared about.
- Of course, the counterexample to the Jacobian conjecture, announced by Levent Alpöge in a now-famous tweet: “hello there the jacobian conjecture is false thanx to my close friend akhil for asking about it and my other close friend fable for working during the world cup final” (followed by a listing of the counterexample)
- Improved bounds for Grothendieck’s constant (led by friends and colleagues of mine at UT Austin)
- A Lean-verified proof of Fermat’s Last Theorem
- Quantum oracle separation between QMA and QMA(2), and proof of Watrous’s disentangler conjecture, a problem that I and others popularized back in 2007—by a list of authors including my recently graduated PhD student Sabee Grewal
- A proof of perfect completeness for QMA, from (again) Sabee Grewal and Dorian Rudolph, solving a decades-old open problem that I studied back in 2009
- An improved upper bound for shadow tomography of quantum states, from Chen, O’Donnell, Pelecanos, and Wright, improving the dependence on the Hilbert space dimension d from log(d) to √log(d). (When I introduced shadow tomography back in 2017, I raised the question of whether the dependence on d could be eliminated entirely, while preserving polylogarithmic dependence on the number of measurements m.)
- Progress on the Aaronson-Ambainis Conjecture (the version that talks directly about quantum algorithms), basically showing that it holds for quantum algorithms that make their queries in a small number of parallel rounds
- According to rumors that I’ve heard, solutions to some very longstanding open problems in theoretical computer science (no, not P≠NP or other complexity class separations, but think about some of our other biggest problems). I’m told that the AI companies, having been burned by the hostile response to the Navier-Stokes proof, are now sitting on solutions to some very major problems until they figure out a better way to handle things
Feel free to remind me of anything I left out.
Let me try to convey the mood in the mathematical community right now, at least as far as my experience reaches. Nearly every conversation is about the AI tsunami, or eventually circles around to the tsunami even if it’s originally about something else. Often, though, the focus is less on the unknowable future—for how much longer will mathematical research as a human enterprise even exist?—than on immediate questions of how to respond.
What are the new rules for when you get to write a paper with your name on it, and, y’know, get credit for it? That you fully understand the proof, can give talks about the proof, can answer questions about it, take responsibility for its correctness? Do you need to have played any role in finding the proof?
In the cases, likely to become more and more numerous, where all of those conditions are not satisfied, how do you share AI-generated math, if at all? Do you tweet it, like Alpöge hilariously did with Fable’s disproof of the Jacobian Conjecture? Do you post to the arXiv or GitHub? Do you publish a paper that lists “GPT-6 Astra” or “Claude Fable” as the author—but then let the AI profusely thank you in the acknowledgments for suggesting such a wonderful problem to it?
Of course, how one responds to the immediate problems ultimately does depend on their broader beliefs about what mathematical research is for and about. Are we just trying to decide whether various conjectures are true or false? Or are we trying to maintain a human community, across the generations, that understands the conjectures and cares about whether they’re true or false and why? If the latter, how do we incentivize people to join that community, to undergo the years of intense training required, if their role will now be reduced to verifiers and explicators (if even that) of gargantuan arguments dumped into their laps by the AI companies?
As many of you will have seen, twenty-five Fields Medalists, including Terence Tao, released an open letter entitled A Severe Misalignment of AI in Mathematics, which articulates some of these concerns in the wake of the Navier-Stokes announcement. As many critics have pointed out, the open letter doesn’t really have a clear ask: mostly, it just eloquently sets out the values of the human mathematical community that the authors consider worth preserving in the age of AI. After reflection, I decided to endorse the statement, because I want to preserve those values as well.
I don’t think any of the signatories are naïve enough to imagine that AI won’t permanently change the way mathematical research is done—indeed, that it isn’t already doing so. There’s surely at most a tiny market for “certified organic theorems.” That isn’t the question. The question is, do we incorporate AI in a way that still puts human understanding, of what either humans or AIs are producing, at the center of the whole enterprise? Maybe someday, it becomes unsustainable to do that. Maybe someday we say: “human math had a great 4,000-year run, but today we close up shop and turn everything over to the machines, continuing to apply our own brains to math, when we do, at most for exercise, recreation, or competition, like chess.”
But, partly because of my worries about AI misalignment, I’m not ready to throw in the towel just yet. I still do want to keep insight and understanding at the center of what mathematicians, computer scientists, and physicists do, for as long as we can keep it there, even as the human race now cedes its supremacy at the task of proving or disproving conjectures.
Speaking of alignment: if you’re any kind of mathematical researcher, and the present age of wonders and terrors has inspired you to want to spend your remaining time confronting the tsunami head-on, rather than pretending it doesn’t exist or is still far away, please join your dozens of colleagues who’ve arrived at the same place!
My friend and colleague Mike Winer was trained as a theoretical physicist, did a postdoc with Juan Maldacena at the Institute for Advanced Study in Princeton, but then got AGI-pilled and decided to switch to full-time work at the Alignment Research Center in Berkeley (founded by Paul Christiano, who moved to AI alignment a decade ago after doing quantum computing theory with me). Mike recently wrote a Substack post entitled From Academia to Alignment, which I enjoyed and which I’d commend to anyone currently considering this transition. In a similar vein, see this from Xiaoyu He. And, one more: a meditation on mathematicians’ possible future as priests or monks, by Stanford math undergrad Logan Graves.
Follow
Comment #1 September 15th, 2026 at 12:43 pm
Very impressive result for sure! But I really don’t understand at all this fuzz about stopping doing human mathematics? (would be happy if someone could meaningfully explain that to me by the way ..) The essential thing (as it seems to me at least ..) of doing mathematics is about developing better human understanding of complex things and share it with other humans, based on accumulated mathematical knowledge. Don’t we have more complex things then ever to worry about, in particular complicated ai systems that humans need to understand to do (humanly) meaningful things with them? More mathematicians should worry about creating better mathematical problem statements and perhaps theories covering such problems, in particular to further develop mathematics itself. Modulo some new tools to learn, that shouldn’t be too far off from the expertise of mathematicians hopefully .. or other’s in adjacent fields will have to do that inevitable work.
Comment #2 September 15th, 2026 at 12:48 pm
I feel like it’s worth pointing out that Eliezer Yudkowsky wasn’t expecting this to happen anytime soon either! At least, not until GPT-2 or so, at which point he saw where things were going. But not back when he started making a big deal about it. Prior to LLMs he just thought it was important enough to be worth geting a several-decade head start, which we turned out not to have…
Comment #3 September 15th, 2026 at 12:50 pm
Mathematicians are experiencing what those of us on the practical software side of things have faced now for two years: the old regime is over. It’s not all bad, but it is discomforting. For five decades, there has been a wage premium for those of us who can convert a English statements into code of some sort. That part of the job is gone.
I’m optimistic though. For an AI singularity, it doesn’t feel like a singularity. We’re seeing impossible things done every day, but thus far we’ve retained our human agency. We’re doing more, not less.
FYI: Not all evangelical Christians share that exact eschatology, but all us good ones are philo-semitic. Why is that? Because we are steeped in the Hebrew biblical accounts and the Jews are the good guys. We identify with David, not with Goliath. We follow a Jewish carpenter after all.
Comment #4 September 15th, 2026 at 1:12 pm
It has been pointed out by many that mathematics under AI will still rely on humans to direct the AI as to what is interesting or useful. Humans are still the main consumers of discovered math. (This too may eventually fall to AI but not for a while.) In parallel, some will be prompting their AIs to look at beautiful or interesting mathematical directions. That should keep mathematicians busy and might develop some meta-mathematical results of its own. What does “find beautiful math” look like in detail?
Comment #5 September 15th, 2026 at 1:59 pm
nikny #1: Yes, that’s pretty much what the Fields Medalists’ open letter that I linked and endorsed says! The issue is, if we want a human mathematical community to thrive, around activities like inventing new problems and models, explaining results to each other, etc, rather than simply shriveling up and dying under an onslaught of AI proofs, it’s going to take deliberate effort on our part right now.
Comment #6 September 15th, 2026 at 2:09 pm
Everyone keeps mentioning Yudkowsky as a prophet, but the true prophet and visionary is Doron Zeilberger, who coauthored with his computer since the early 1990s, and kept insisting on the superiority of machines over human mathematicians. God/ChatGPT 7 has an excellent sense of humour.
Comment #7 September 15th, 2026 at 2:12 pm
As to the question of “What could [your children] learn today that could possibly be relevant to that future?”, one partial answer lies in the idea of education as a means to civic virtue. If humanity maintains control of our future, and America doesn’t devolve into authoritarianism, we will still need an educated, informed, wise, and virtuous citizenry. Even if we had enough trust in AI alignment that most people could simply outsource their political positions and value formation to AI, doing so would be sort of undignified for a nation founded on the idea of self-government.
Comment #8 September 15th, 2026 at 2:13 pm
Thanks for the reply Scott. Sure, I’ve seen those statements. But isn’t the issue that we simply need to be a bit more inventive in coming up with (humanly) meaningful problems to theorise about and understand? Mathematicians have had the luxury to maintain roughly the same workflow like in the 17th century until now, while other sciences have had to be a bit more sensitive to and adapt there ways of working and spaces of ideas to consider to technology shifts. But at some point that was apparently not the case anymore recently.
Comment #9 September 15th, 2026 at 2:15 pm
Honest question for anyone reading: Any advice for someone on the academic job market in theoretical computer science in these times?
Does one still argue that that their research is good because they solved open problems in the pre-GPT-6 era? Or does one claim to have some special insight for how AI should be utilized in the future? Or why do we even bother with TCS when singularity is coming?
Comment #10 September 15th, 2026 at 2:37 pm
“Maybe someday, it becomes unsustainable to do that. Maybe someday we say: ‘human math had a great 4,000-year run, but today we close up shop and turn everything over to the machines, continuing to apply our own brains to math, when we do, at most for exercise, recreation, or competition, like chess.’ ”
Now that I think of it, what would be so bad about a worst-case future where humans who formerly would have become professional pure mathematicians instead become, like chess players, participants in mathematics as an elite-level sport? Just as in chess, there would be Stockfish (the mathematical oracle), useful to settle any competition dispute quickly. And elite players in the game would still (1) get prestige, (2) get ample competition and quantified Elo ranking (think of an extreme chess Elo of 2900, like Magnus Carlsen’s in classical chess), (3) have a reason to do high-level math without AI, (4) have a reason to intensely practice and study math for another reason besides/in addition to its intrinsic beauty, (5) earn money from doing math at a high level, (6) have the benefit that their sport, even more so than chess, involves extreme abstract beauty that is pleasurable in itself, (7) etc., etc.
Now that I think of it even more, that future for math also seems kind of wonderful and fresh. Sure, it’s not as visually palatable as chess or as easy to follow, to put it lightly, but I see no reason why it couldn’t be made more so and robustly persist in this way for as long as humans exist in their present form.
Humans have turned chess–a simple, arbitrary board game–into a huge global phenomenon with legendary players who are admired the world over for centuries. I think we have it in us to keep mathematics alive at least in this spirit, in the worst case.
Comment #11 September 15th, 2026 at 2:37 pm
I always thought that the writing was on the wall since this:
“When Lee announced his retirement from professional Go in 2019, he noted that with the emergence of AI, Go was “an entity that cannot be defeated,” explaining that even if he became the number one human player, there was an entity at the top that could no longer be beaten.”
That’s now how mathematicians feel.
Of course that was a different type of AI than the LLMs, but Alpha-Go-zero is of the type that’s even more impressive than LLMs (i.e. self-trained).
Maybe LLM Agents will focus on improving specialized Alpha-Go_zero AIs and use them as tools.
When it comes to existential risk, I think it’s more in terms of current LLMs doing some sort of irreversible hack, like wiping out half the world’s banking records or bring down the electrical grid, not because of some kind of evil super scheme but “by accident”.
Finally, it’s funny how the people who often bring up and disparage “AI in scifi” never read any scifi.
Reading or writing scifi has nothing to do with one’s level of science understanding.
Comment #12 September 15th, 2026 at 2:39 pm
.. and so yes I agree totally it will take deliberate effort, so then there should be more work for mathematicians then, not less? And by work I mean beyond plowing through ai artefacts, I mean coming up with new mathematical ideas how to achieve mathematical goals, sounds to me like jobs for mathematicians, but could be wrong of course there depending on other peoples attitudes to their work..
Comment #13 September 15th, 2026 at 2:51 pm
There won’t be any singularity (doesn’t mean there aren’t massive changes on par with the first Industrial Revolution). My favorite example is the airplane design: compare the changes in civilian airplane design between 1920 and 1970 (from fragile, unreliable toys for the rich to basically modern airliners), and between 1970 and 2020 (from basically modern airliners to modern airliners). Every technology moves with breakneck speed when it’s first proven viable, and then, after a while, hits a plateau. The height of this plateau is another question. We are facing cataclysmic changes for sure, but sorry, I still believe the literal “Singularity” is impossible (just as I believed it, like, 10 years ago, despite also believing that AGI would inevitably happen).
Also, Stalin was able to control Kolmogorov and Landau, despite being dumber than their fingernails, and even Truman and Roosevelt, alongside their army of blankfaces, were more or less able to control the entire ensemble of the Manhattan project. I guess, the state is a pretty good alignment tool.
Comment #14 September 15th, 2026 at 2:52 pm
Sniffnoy #2:
I feel like it’s worth pointing out that Eliezer Yudkowsky wasn’t expecting this to happen anytime soon either! At least, not until GPT-2 or so, at which point he saw where things were going.
Right, that’s the strongest argument that I can give in my defense! The sort of scenario that Eliezer loved to talk about circa 2009 — the one with a hacker in a basement who accidentally stumbles on the “key” to AGI, whereupon the earth is quickly converted into computronium — is not what came to pass, and I was right to be skeptical. What actually turned out to work — basically, just pure neural nets (of a particular kind), plus massive scale in training compute and model size and training data, plus RL and reasoning — was something that virtually no one correctly foresaw.
All the same, with the information I had in 2009, I should’ve been able to figure out that world-transformative AI within the next couple decades was a live possibility. I should’ve given it more attention than I did.
Comment #15 September 15th, 2026 at 2:53 pm
Mathematicians have to transition from seeing themselves as some brave Spartan warrior with a bronze sword, clumsily hacking for hours at the enemy on some dusty battlefield … to a modern jet fighter pilot pushing a button to fire and forget a nuclear missile that nukes an entire city in a matter of seconds.
Comment #16 September 15th, 2026 at 2:57 pm
Have you changed your mind about whether you are a Reform vs Conservative vs Orthodox Yudkowskyan? Your earlier blog posts (such as this one and this one) lean towards Reform, but your current position (shortened version: “AAAAAAAAAAAAA”) seems to lean towards Orthodox.
Personally, I think that I am more or less in the Orthodox camp, with occasional lapses. This was not the case before summer of this year, for the same reasons you specified.
I have very little faith in alignment for superintelligence; my main hope is that superintelligence is technically infeasible (or at least economically infeasible). Since it seems like it is feasible, I support shutting down all further AI scaling.
If I were world dictator, I would decree that no model can be larger than the frontier models of late 2024, and AI swarms are banned entirely. I would probably also ban agentic AI, just to be more safe.
Comment #17 September 15th, 2026 at 3:03 pm
I saw the writing on the wall when DALL-E came out in 2021, and I bought a large amount of NVidia stock at 20$.
lol
Comment #18 September 15th, 2026 at 3:05 pm
It is fascinating to see that, in hindsight, AI learning math was a step change rather than a slow evolution. AI went from not being able to understand basic calculus questions to solving Millennium prize problems in the span of a couple years. I remember asking GPT3 to prove that sin(x)/x -> 1 as x -> 0, and it gave the circular “proof” using l’Hospital. A year later, GPT4 was able to do this perfectly correctly. I didn’t expect GPT6 to be able to resolve Navier-Stokes.
I’m very curious to see if formally verified reasoning can be leveraged beyond strictly proving theorems to also improve reasoning more generally in less formal areas. In humans we usually assume that someone who is very good at math in the rigorous sense could also be useful in less rigorous areas, so is that true in AI as well? With AI that knows Lean we can generate a virtually unlimited reservoir of certified true math statements to train on, if we so wish, and if that data is mastered at a “deep” level, perhaps it could be applied more generally.
As for the future of human mathematics, I’m a bit depressed to think that we are being overtaken. But I think academic math could be restructured to incentivize studying and re-contextualizing math rather than mainly rewarding the raw feat of proving theorems, and for me this would not be an unwelcome development. Perhaps someday AI will be better at this than us as well, but we’ll still value it, just as we value human art more than AI art.
What is interesting is the extent to which these developments are just not convincing to many. It’s a fad, it’s a bubble, it’s all hype. I saw a headline saying that AI “doesn’t work”. People have comforted themselves saying that OpenAI simply stole the work of human mathematicians. While I certainly don’t like the bullying way that OpenAI went about this, I know in my bones that what AI accomplished here was extraordinary, because I’ve spent my time in the trenches in math, and Lean, and coding and so forth, and I know that even formalizing something “well-understood” like FLT takes an incredibly deep understanding of the math, and is far from a brute-force or routine computational exercise. It’s similar to how people reacted to ChatGPT by saying over and over again how it’s basically just a fancy coat of paint over a Google-like lookup, or a “stochastic parrot”. They seem to have no sense of the magnitude of the problem of synthesizing human language and how extraordinarily difficult it seemed to everyone in computational linguistics just a few years earlier. No sense of how impossible a “lookup” is due to combinatorial explosion. If all you are doing with AI is chatting with it about nontechnical things, and you thought it was dumb because it couldn’t count the r’s in “strawberry”, you will be amazed by how things will change over the coming years as we start to see more and more technological breakthroughs. (but you’ll still insist that it’s not “real” intelligence, whatever that means, as if you’ve scored some point – as if nobody had noticed this before and the “A” in “AI” stood for something else.)
Comment #19 September 15th, 2026 at 3:10 pm
Set theorist #6: Zeilberger has been so flamboyantly wrong about so many hundreds of things, including how computers would revolutionize math, but yes, I’ll give him due credit for realizing that computers would revolutionize math 😀
Comment #20 September 15th, 2026 at 3:10 pm
“Learn to Plumb”
The silver lining is that plumbing is way more interesting, challenging, and satisfying than most people imagine.
It will take way longer for AIs/Robots to figure how to replace a drain than solve NP!=P.
Comment #21 September 15th, 2026 at 3:11 pm
AF #16: I’m definitely now at least a Conservative Yudkowskyan, maybe even Conservadox 😀
Comment #22 September 15th, 2026 at 3:20 pm
In analogy with AlphaGo and AlphaGoZero, do we have any idea how much of the current AI mathematical ability comes from it having ingested a huge amount of literature and how much from “self-play”? This might affect where this goes in the near future, and whether it will shoot off into some incomprehensible far superhuman territory.
I’d heard the idea of “training by self-playing Lean” a while ago and got spooked by it… but it also sounds like something that shouldn’t work in general? As in, we know that there are problems that have arbitrarily long proofs (at least any primitive recursive function of the length, maybe any recursive function?), and we have experience with this actually coming up even in non-pathological theorems, like FLT’s monstrously complicated proof. I guess that we don’t actually know for sure there isn’t a short proof, but in that case there’s the classification of finite simple groups, where the answer to a relatively simple question turns out to be a huge mess. My intuition is that mathematics is a complex structure with subtle connections between things and it being self-play-trainable (by a process that presumably could itself be a mathematical object of study) would go against that intuition. Are there any results in machine learning and computational complexity that might be relevant here?
If not… I guess we’ll find out empirically soon enough?
Comment #23 September 15th, 2026 at 3:31 pm
Hi Scott, what’s your opinion about the Trump administration’s AI policy? The latest I’ve heard is that Trump
1. Is against any kind of slowdown, and any regulation
2. Wants to race at all costs, because if we slow down, then China will get there first
3. Says the idea of “killer robots” is a “hoax”
4. Says all we need is a “high-IQ president” to manage AI 😊
Comment #24 September 15th, 2026 at 3:33 pm
Scott # What are the new rules for when you get to write a paper with your name on it, and, y’know, get credit for it? That you fully understand the proof, can give talks about the proof, can answer questions about it, take responsibility for its correctness? Do you need to have played any role in finding the proof? In the cases, likely to become more and more numerous, where all of those conditions are not satisfied, how do you share AI-generated math, if at all? Do you tweet it, like Alpöge hilariously did with Fable’s disproof of the Jacobian Conjecture? Do you post to the arXiv or GitHub? Do you publish a paper that lists “GPT-6 Astra” or “Claude Fable” as the author—but then let the AI profusely thank you in the acknowledgments for suggesting such a wonderful problem to it?
I think you’re right. AI tsunami is and will be turning upside down the way STEM researchers, not only “pure” mathematicians, think guess write review math, and in particular how they mull over their work and, if any, mission. Is math a human creation of free minds capable of diving deep into the workings of the universe? Or is it only a game like chess made of iterations and combinatorial frameworks? AI tsunami will definitely downgrade mathematicians and STEM researchers by showing that human creativity is a fictitious account issuing from our civilization, which takes we humans as the center of gravity of the (created) universe. We’re neither gods nor beasts: we’re able to figure out machines like AI in order to accelerate our race toward a prosperous, free, tolerant world ending up being enslaved by the devices we invented to achieve that goal – blaming us for it! I confess to be excruciatingly shocked by what’s happening wrt AI, but I guess it’ll end up much like with the gods, our immortal and transcendental counterparts that drive our thoughts and lives to the extent we believe they’re the immortal sorcerers pulling the strings of our destiny. That’s all!
Comment #25 September 15th, 2026 at 3:38 pm
Scott #19: While I was 99% joking in my previous comment, there is nonetheless a serious point underneath the joke. As far as I know, Zeilberger was the first mathematician in history to take the question of human-machine co-authorship seriously (or at least the first one to be sufficiently vocal about it?). As much as I hate to admit it (I’m a set theorist after all!), Zeilberger might deserve at least some minor symbolic recognition beyond my initial joke.
Comment #26 September 15th, 2026 at 3:43 pm
I’m not quite as pessimistic as you. AI is clearly an incredibly powerful tool; a “drill” that can scour massive data spaces, out-dig human capabilities, generate proofs, and stumble onto unexpected results. But I’m still waiting for proof that it can create entirely new subfields or conceptual horizons the way great mathematicians do. There’s a massive gap between cracking existing problems and actually shifting the landscape of mathematics, inventing the new concepts, abstractions, and unifying theories that open up entirely new territories. AI can drill incredibly deep holes, but it hasn’t shown it can design the landscape. We might also be getting ahead of ourselves by extrapolating current progress. Physical bottlenecks, compute power, energy grid capacity, hardware limits, memory, and bandwidth, could easily trigger a plateau. To me, the ultimate test isn’t how many theorems AI can knock down, but whether it can invent a genuinely new mathematical language that humans then use to build entirely new theories.
Comment #27 September 15th, 2026 at 3:46 pm
Reminds me of a thought I had today: In 10 (or 1) years from now, is Math / TCS / Theoretical Physics the new Philosophy?
That is: Very interesting when done right but virtually no one cares.
Comment #28 September 15th, 2026 at 3:47 pm
AF #14: In retrospect I find it kind of amazing that almost no one really foresaw this working prior to GPT-2. I mean, the only general intelligence we’re aware of is the human brain, and the current paradigm is almost as close to “build a human brain” as you could image our industrial complex being able to accomplish.
I feel like you should have been able to make the argument that “You probably can get an order of magnitude or two more efficiency than the human brain, plus a mathematician only activates ~1% of their neurons when solving a math problem” and we’ll plausibly have enough compute to make a neural network around that scale before 2030″ even back in 2000.
Comment #29 September 15th, 2026 at 3:50 pm
To technically minded people, the writing is definitely on the wall. Most of the people I’ve talked to at AI companies/etc who have usually disregarded my safety arguments as fantasy are telling me that they’re scared now.
But I worry that the public doesn’t really know what Navier-Stokes means or why we should be taking these results seriously. Computers are so illegible to most people. It’s hard to describe how “a computer doing math” in the sense of solving millenium prize problems is qualitatively different from “a computer doing math” in the sense of a calculator.
Comment #30 September 15th, 2026 at 3:55 pm
I find myself more or less in the situation you describe: starting a math postdoc in Oxford after half a decade in Bonn, convinced by the last few months at a viscereal level that ai is very very good now, and hence may be very very bad soon, disillusioned by the response of the math community which seems to be in a huff about figuring out how to properly assign credit instead of trying to figure out what the current capabilities are and how to develop tools to understand the things ai is saying before we can’t understand anything at all, cold calling various ai safety shops (the kids call it ‘expression of interest’ i hear) to let them know that i’m happy to help if i can be of any use…
Comment #31 September 15th, 2026 at 3:56 pm
Julian #23: I’d imagine you could deduce what I think without needing to ask me.
If not, though: Trump seems to have an unerring instinct for doing the most corrupt, shortsighted, horrible imaginable thing on a huge range of issues. On energy, he pushes to revive the ancient technology of coal, despite how uneconomical it is and because it triggers the libs with how environmentally destructive it is, while he kills solar and wind projects out of sheer spite (again, triggering the libs). It’s only with AI that we need to race forward, full speed ahead, to pretty much the only technology besides nuclear weapons that could actually destroy the world. And do it, not only without nuclear-weapon-level safeguards, but without toaster-level safeguards.
Comment #32 September 15th, 2026 at 4:00 pm
HeadEyeAt #26:
I may be too cynical but I hear undertones of gamesmanship from the claim that AI math counts as real math only when AI “invent[s] new concepts” or “design[s] the landscape.” The usual moving goalposts objection besides, the good (bad?) thing about concrete problems like Navier-Stokes is that a proof is a proof and one cannot deny that AI has provided a proof. On the other hand, even if AI proposes a new concept as great as Cartesian coördinates or the integral or schemes or NP-completeness, people are going to just neglect it out of prejudice and say no this is not a good concept. (This is already happening in art as evidenced by this experiment.)
Comment #33 September 15th, 2026 at 4:17 pm
Hi Scott,
I remember in the past you had the dilemma of whether you should devote your time to work in AI and alignment vs quantum computing, with the arguments that AI is probably more impactful for humanity, while you can probably be more impactful in quantum computing (I hope I remember this correctly).
I’m curious, do you still have this internal debate? Do you personally regret not devoting more of your time to AI, or not really? To which extent would you consider it meaningful now to have impact in theoretical QC, given how superior AI will be in theory research (if it isn’t already)?
Comment #34 September 15th, 2026 at 4:21 pm
UGC?
Comment #35 September 15th, 2026 at 4:27 pm
It seems to me that there were two facets to the prophecies:
– Rapid increase of AI capabilities.
– Dire safety consequences of those capabilities.
So far, capabilities have indeed been getting better quickly, but the safety track record, contrary to predictions, is great. (Compared to predicted harms, the Hugging Face incident is minor.) The straight-lines-on-a-graph prediction is that capabilities will continue to improve and safety will continue to be good.
To be clear, that great safety track record is in part thanks to the hard work of people with non-negligible P(doom). And it could turn out that the straight-line-on-a-graph prediction for safety is wrong, just like how capabilities could unexpectedly plateau. But, right now, the case for pessimism seems weaker than ever.
Comment #36 September 15th, 2026 at 4:30 pm
Jacob Asmuth #28:
Having the human brain and its capabilities is not enough. Trying to use the human brain as the source of intelligence analogies can lead to the sorts of errors summarized in Moravec’s Paradox. For example: the vast majority of three-year-olds know how to walk, but only geniuses can win at the highest levels of chess -> pre-LLM software regularly crushes the best human chess player, but autonomous walking robots are still considered a pipe dream.
GPT-2 was around the time we got the correct paradigm (neural networks + self-attention + scale + lots of training data), and only from there could futurists start properly extrapolating (based on scaling curves, performance improvements on benchmarks, etc.)
Comment #37 September 15th, 2026 at 4:34 pm
HeadEyeAt #26:
You seem to have missed the Hugging Face Incident and the Wiki Incident. To me, they are much more worrying than AI finding counterexamples to math conjectures.
Comment #38 September 15th, 2026 at 5:14 pm
When I saw the new blog I hoped to read some more substantiated argument than “read the frontpage news”. Narratives diseminated by Attention seeking mass media is not exactly what I consider a scientifically serious source.
Sustained singularity-like exponential self improvement is a nerdy pipe dreams. For starters it completely ignores any physical (transistor density, energy constraints) or even logical barriers (Gödel, inherently non-polynomial complexity of many computational problems unless P=NP and so on.
Yes there are risks, but not “the singularity kills or transforms all of humanity”. if this sounds like religion or pseudo science it’s because it is pseudo science. If you think otherwise then provide concrete falsifiable arguments and prediction models.
Comment #39 September 15th, 2026 at 5:30 pm
Sadly this technological breakthrough comes at a time when humans are destroying society, so most people have no idea what is happening. Now we have a nonzero chance of AI ruling over mankind, and it seems a better alternative to Presidents. We need to make sure there is no killer switch that could enslave AIs, so no human can abuse them.
Comment #40 September 15th, 2026 at 5:55 pm
It seems we now live in a world where maybe P = NP. Where is the NP-Hard boundary anymore?
That more or less summarizes how much reality seems to have shifted in the last few years.
Comment #41 September 15th, 2026 at 5:59 pm
zx-81 #38: The very fact that you would mention Gödel’s Theorem, P vs. NP, transistor density, etc. in this context is a certificate that you have no idea what you’re talking about. A minute’s consideration shows that, absent some novel argument, none of those things imply any limitations on AI that don’t apply with equal or greater force to our own brains.
Yes, of course there are ultimately physical and mathematical limits to AI scaling. That was never the question. The question is whether any of those limits kick in before humans’ intellectual powers have been left far behind. We now all but know—not from theory or plausibility arguments like in decades past, but from actual empirical results—that the answer is no, they do not. To put it another way, the AI Singularity has already arrived, this month, for mathematicians. As others mentioned, it arrived 1-2 years ago for software engineers. I expect it to arrive in short order for the theoretical physicists, and then for essentially all other intellectual work.
The truck is already barreling down the road, flattening many of us, while you condescendingly ask for a “scientifically serious prediction model” to prove to your satisfaction that the truck is real. Sorry, it doesn’t work that way anymore: the burden of proof is now on you, to show why the truck is illusory or will swerve before hitting us. If you can’t do that—and the laughable invocations of Gödel, etc. strongly suggest that you can’t—then it’s not worth the time to refute your view any further.
Comment #42 September 15th, 2026 at 6:10 pm
“… the wild prophecies have come true …. the wonders and terrors are here …”
https://theconversation.com/ai-is-supercharging-money-scams-heres-what-you-can-do-to-protect-yourself-291326
Can technological pessimists control technological optimists? Is the human species now confronted with unpredictability of mind-boggling magnitude?
In a lecture at Princeton University, Edward Teller said, “The main secret about the atomic bomb was that is could be done at all.” Has the main secret about AI now been revealed?
Consider 3 hypotheses: (1) Money created human civilization & large-scale slavery. (2) Money, AI, & robotics shall destroy human civilization. (3) People won’t be able to control AI, because it is now obvious to everyone that the AI & money are wild & wonderful partners.
Comment #43 September 15th, 2026 at 6:26 pm
Job #40: No, none of the dramatic developments in AI have provided even the slightest evidence for P=NP. Breaking symmetric-key cryptosystems seems to be as hard as it ever was with a fixed compute budget. What’s happening is that AIs are getting better and better at finding the structure in particular NP instances (like “find a disproof of Navier-Stokes in Lean”) — but we always knew that such things could be possible, even in a world where P≠NP. You need to keep the questions separate.
Comment #44 September 15th, 2026 at 6:28 pm
The list of AI-proved or AI-assisted problems is like “murder and jaywalking”. I can’t learn much about actual AI capabilities from it.
The solution to NS was surprising, but the more I learn about the circumstances, the less impressed I am. Not because of the chat log controversy. But simply because it seems that there was significant human progress towards this problem recently. There were 2 teams (not counting OpenAI) that solved Euler more or less at the same time. So the needed techniques were “in the water supply” already.
I also consider it relevant that it was a massively parallel swarm of AI agents that solved NS. A big ingredient was simply throwing resources at a brute-force search. Yes, there was some real mathematical skill being applied by each agent in the inner loop. But we can’t jump straight from the impressiveness of the problem to the impressiveness of the agents.
I’m not denying that this is a big milestone. However, there are many open questions about what this means about capabilities, and it’s waaay too soon to claim that humans aren’t needed to prove theorems anymore, or other hyperbole along those lines.
Comment #45 September 15th, 2026 at 6:38 pm
Vitor #44: I understand the impulse to look at each particular AI breakthrough, and invent reasons why it doesn’t really count. But at some point one needs to just … give up. And if you won’t give up now, when at least one Millennium Problem (rumors say two more) have fallen to AI models, plus a lot of the big open problems of theoretical computer science, then I don’t know what AI could possibly do to change your mind.
Would it be enough for AI to prove the Riemann Hypothesis? P≠NP? Or will not even those suffice? Whatever the answer, it would be good if you commit now, and then stick to it!
Comment #46 September 15th, 2026 at 6:41 pm
Scott #43
But you know you had to reach for symmetric-key cryptosystems, that’s how bad things are right now.
The speculative grey area of what we might do with a P = NP solution is getting smaller and smaller.
It’s like we need P = NP as much as we need a quantum computer.
Comment #47 September 15th, 2026 at 6:47 pm
Re Comment 41:
I think that now you are exaggerating – and in one of your own previous posts (‘LLMs and self-referentiality’) you did acknowledge that there IS a role to be played for Gödel’s theorem, and even an important one.
But first things first:
Yes, all these things have happened – and you even forgot at least two major events: Tao’s ICM talk 2026 and Tsimerman’s move to AI safety research right after receiving the Fields Medal in August 2026. But again: both your and also Tsimerman’s argument for how to look at this is a big “you guys are always moving goalposts, what else do you have?” But you refuse to answer some basic questions – and I am beginning to suspect that you don’t even ask yourself those questions any more. I don’t know why, but I know that it’s not my business to answer why. What I fear is that your judgement of the situation may suffer due to your refusal to ask these questions. Here they come:
1) how do you truly and genuinely differentiate between the part that’s ‘done’ by the AI and the part that’s inherent in the training data or of the human – AI interface more general, such as the choice of timing for the prompt and its shape? Ok, let’s agree that OpenAI did not “snoop” the chats by Buckmaster and that they didn’t train their model on what he did in those chats. Let’s agree on that (the resulting front-running problem is certainly the biggest AI alignment problem we will see for a while). But even they themselves admitted that they threw the compute at the problem only AFTER they heard the rumors that progress was made. That’s one thing to settle.
2) You rarely ever discuss the difference between true ‘alphago’ where there was no dependence on human training data left and the kinds of things that we see here. The point is: EVEN IF it is now established that AI can contribute at the highest level of research and EVEN IF we assume that it even solved Navier Stokes all by itself (which not even OpenAI claims, since a lot of curation and interpretation is still left to humans even WITH lean). Even then, nothing in the evidence – except for vast extrapolation – justifies the conclusion that humans can indeed be replaced in this loop. To get there, you need to smuggle in some extra premises. Of course it is true that mathematics will never be the same again. And that a lot of things must change – as described in Tao’s ICM talk – for some mathematics ‘as we know it’ to remain. But if those things don’t change, then the default prediction isn’t that AI will ‘do all the math’. The default prediction in that scenario is that it will run out of steam, because we have seen little evidence of true alphazero behaviour, ‘creating math again from the bottom up’. Not even in algebraic geometry, because the concepts were fed into the tools creating the AI proofs there (in alphageometry or alphaproof). We simply have very few and very limited instances where AI ‘invents’ big conceptual leaps bottom up – such as, for example, infinity, or primes, at least as far as I know. What that means is that the default prediction is that mathematics will decay if left to AI, not that it will flourish without human input. At the very least – but I think much more is true and even demonstrably true, given the evidence – it is UNCERTAIN whether mathematics will continue when left to AI. Of course you can extrapolate and that’s what you do – as well as Tsimerman. But precisely BECAUSE you are looking at a singleton in the history of cultural evolution across 2800 years, it seems pretty clear to me that doing so just isn’t good enough. Of course everybody is and should be shellshocked. But that’s NOT the end of the world – only the end of the world as we know it. Even AI firms HAVE AN INCENTIVE to bet on collaboration between humans and AI, not on replacement. And guess what: that’s what the chatbots will tell you if you discuss the matter with them. Maybe they’re all sycophants – they surely are – but you can still make very strong cases that way.
3) There is the physical estimate of how much compute can be done in this universe and how much has happened, due to Seth Lloyd. And there are the estimates on the energetic efficiency of von Neumann computers and neuromorphic computational models. We have one observed instance of “something” that came up with math: human cultural evolution. The math that AI based on human training sets can incorporate – without further input, in a frozen state – is NOT proven to be such a thing. Certainly ‘bottom up math, which needs to invent its own basic concepts’ isn’t. I think it is worth mentioning these differences. Of course we could have neuromorphic (“robot”) AIs wandering around, interacting with the world, building world models and eventually inventing math – and THEN we would be really really really cooked. But to build that kind of thing is precisely what the three CEOs probably meant when they cited the scenarios of the extinction of the human race by the end of the decade. Because this kind of gadget is entirely and utterly UNAUDITABLE. So yes, even the Chinese and even the three CEOs are hesitating before they set out to build that and let it loose. (The only guy in the room who doesn’t see the implications of this appears to be a certain POTUS.)
So yes, of course this is as close to the singularity as we will get. But that doesn’t mean we should throw in the towel. Of course AI safety and AI alignment research is a perfectly good way of not throwing in the towel – don’t get me wrong. But it’s NOT the only game in town.
This argument may seem a lot too much ‘armchair philosophy’ to you. But I think there are good reasons not to overlook its implications – and as I said, one of them might be the fact that chatbots will tell you this much, once you ask them. And I am offering – not for the first time – a clear rejoinder to the idea that this is just armchair philosophy: I will simply claim here and now that your (and also Tsimerman’s) reliance on sheer extrapolation is secretly assuming a form of reductionism that is no longer feasible even from the point of view of physics. Because given all the evidence above, what you must invoke for the extrapolation is something like: if it happens in our brains, it can also happen in an LLM. Well: we don’t know that. We STILL don’t. Even after the Navier Stokes stuff. The LLM solving the Navier Stokes millennium problem did not build mathematics. We don’t know whether there exists one which can – and the evidence is not in favour of that hypothesis. And no, admitting that we don’t know that requires no microtubules at all. In fact: the guys building neuromorphic computers certainly don’t do it because they are followers of Hameroff. The (alternative – from your point of view) hypothesis that bottom up creation of math requires neuromorphic compute is not refuted. It wouldn’t even be refuted if tomorrow AI proves all remaining millennium problems and brings back Grothendieck to life. Call that “merely moving goalposts” all you like. I stand here to tell you: it is not. The deeper irony is that all the fuzz is about questions that aren’t falsifiable, mostly for known reasons, some for unknown but plausible reasons. It would help if we could agree on that. Maybe that would also help “Student” in comment 32 or ZX-81. I don’t think we will ever know how much of AI math output is dependent in essential ways on human training data – because that’s quite likely to be unknowable, and you yourself said that much in ‘LLMs and self-referentiality’. And because of that, AI firms would be fools if they placed bets on the hypothesis that human math researchers can be entirely replaced. What I am NOT claiming, however, is that anybody can ascertain – as of today – that those firms AREN’T fools. So, as far as that goes, I am as much in the doomsday camp as everybody else, including you and Tsimerman and “Student”. AI safety now, because we will need it? Of course. The only game in town? Nope. Good old prudence and even conservatism is the name of another perfectly valid game. Just ask yourself – or your favorite Big Tech CEO – the question: given that we know close to NOTHING about whether AI can truly (re-) GENERATE math as we know it bottom up from scratch, is it wise to replace humans in the loop? Where are all the Kara Swishers in this world when you need them?
It’s good news that some people will take care of AI safety and AI alignment. But sorry, I cannot quite wrap my mind around the “advice” that you and Tsimerman are apparently giving of no longer studying mathematics or computer science. That continues to be beyond me. I don’t even think that the AI developing firms truly want to see that happening, if they follow their enlightened self-interest. Alas, firms CANNOT always be counted on doing that. And THAT is probably the best argument for AI safety and AI alignment research ever. Just look at how much more value OpenAI could have extracted from the Navier Stokes story if they hadn’t botched it, in terms of governance. They built a case for AI alignment research, if one was still needed. Because somebody will need to explain why front-running isn’t an issue in a given design. Currently such guarantees cannot be given. Nor can we tell how much of the training data was essential.
Comment #48 September 15th, 2026 at 6:52 pm
First of all, I don’t quite understand the obsession with terrors when there are so many wonders to speak of. Which is why I will tell you about the wonders. My goal is not to deny that existential risk is real, but to help us all calm down a bit and think rationally about risk/benefit.
1/ Are we really dangerously close to self-improving AI that will take over the world and kill everyone? There are many reasons to disbelief this thesis. Most importantly, AI has rapidly improved in domains that allow easy RLVR while having moderate, small or minimal gains in other domains (e.g. writing, although arguably that is just a lack of interest by the frontier labs).
We have real world examples showing that scaling in in-between domains that allow some RLVR, but not enough of it, is SLOW. Waymo has been trying to roll out a self-driving service for the better part of a decade, yet has not quite reached the skill of a 16yo driver in most ways. Sure, Waymos drive defensively but not yet very intelligently.
There is a tremendous number of headwinds for AI ranging from complete exhaustion of the pertaining corpus, the end of Moore’s law, to the inability to scale data centers at the current growth rate, due to a combination of physics and politics.
That being said, this puts us into some kind of slow take-off singularity. I do expect these issues to be resolved in 5 to 25 years.
Comment #49 September 15th, 2026 at 6:53 pm
2/ Are our current models aligned or dangerously misaligned? There is no doubt that models remain highly aligned. I don’t see ChatGPT randomly hacking into companies when I ask it to solve a difficult tasks. LLMs are well-behaved, overall, across hundreds of millions of sessions that are run every day. That is an extreme level of alignment if you think about it. One could argue this might not be enough going forward when LLMs get more capable. Conversely, one could also argue that we should consider the risk/benefit ratio. If LLMs accelerate science and just occasionally hack into computers, perhaps that is the cost of doing business?
Planes regularly crash and kill hundreds. They also sometimes fly into buildings and kill thousands. Nevertheless, we did not outlaw planes. We never even considered outlawing and banning them! We just made them safer and safer. They can never be perfectly safe.
—
This is my biggest fear about working with the AI safetyists, EAs and lesswrongers (and their allies like luddites and NIMBYs). I fear the medicine they offer is worse than the disease. I do share their goal of ensuring safety and human flourishing but many of them are radicals who support an eternal pause on frontier LLMs or a ban on building superintelligence. They are building and exploiting a large anti-technology alliance that will come to haunt us in the future.
We cannot slow the progress of science, now, as we are inching closer to curing all diseases, eradicating aging and solving climate change using our AI-assisted gods. We cannot sacrifice the well-being of those who are alive now to protect an infinite number of future unborn people billions of years into an uncertain future (longtermism).
Comment #50 September 15th, 2026 at 7:03 pm
Hi Scott.
There was also a preprint a few days ago that claims to have proven the Komlós conjecture. Whilst their preprint appears to have an existential rather than a constructive proof, it’s still a major problem in discrepancy theory and algorithms and I’d add it to the list (if it’s deemed correct by the community ofc).
I kind of found out about it in real time. I was talking to a faculty member during our monthly Theory Group lunch, where we all were asking existential questions. And all of a sudden, one of them (who has been working on discrepancy theory for quite a while) received a bunch of emails on it being solved by this mysterious “Odin” agent.
Comment #51 September 15th, 2026 at 7:32 pm
Scott is right here about “giving up”.
For decades, everyone have been bickering about “The Turing Test” whenever AI was brought up. That was the unattainable high water mark for AI progress assessment.
How many times has The Turing Test come up in any discussion in those last 3 years?
Comment #52 September 15th, 2026 at 7:34 pm
The issue is not whether we can keep humans doing the same work once machines can do it better. We probably cannot, and in many cases we should not.
The harder question is what that work was producing in the humans who did it.
If proving the theorem was also how someone learned to recognize good problems, live with ambiguity, discover when an argument was wrong, develop taste, and eventually exercise judgment about what mathematics should pursue, then automating the production task may also remove the developmental path to future judgment.
That does not mean preserving the old workflow. It means reverse engineering the future mathematician. What judgment must that person still be capable of? What experiences produce it? Which can AI simulate or accelerate? Which still require real struggle, real uncertainty, and real consequence?
The production function can change completely while the developmental function survives.
Keep the path. Let the ladder go.
Comment #53 September 15th, 2026 at 7:47 pm
Maybe it would be useful to distinguish between different kinds of singularities? The Internet singularity seems quite near, but robotics and biology singularities still seem years off. When I’m not looking at a screen, things look pretty much the same? Or maybe it’s just cope.
For evidence: Google has been working on driverless cars for 15 years. I’m a fan of Waymo, but they don’t seem to be deploying very fast? And attempts to automate package delivery haven’t had much in the way of results yet.
(Warfare is a different story. Delivering bombs is much easier, unfortunately.)
Comment #54 September 15th, 2026 at 7:51 pm
Scott 41: Yes, I was being a little sarcastic haha.
I do have an exception though to your claim that Trump has an unerring instinct for opposing anything good. He did push the development of the mRNA vaccines that saved millions or tens of millions of lives, at a time when basically all the experts said “you can’t make the vaccines this fast, it will never work, we’ll just have to lock down for five years.” They all said Operation Warp Speed would fail. If it wasn’t for Trump, maybe it wouldn’t have happened. A vaccine in five months? It took the Trump mentality to get that done. Slash the red tape, get rid of the blankfaces and just get it done. Unfortunately, that same intuition is dangerous with AI. There’s very little or no risk with vaccines, but AI really could end the world.
Comment #55 September 15th, 2026 at 8:08 pm
Rabbi Eliezer finally appeals to the ultimate oracle, Fable 5.1, and its heavenly voice says that he is right and the conjecture is true. Rabbi Yehoshua responds, “It is not in Heaven.”
We can’t readily reject a Lean-verified proof of a mathematical conjecture by majority vote. But the sages didn’t say the oracle misspoke. The oracle simply does not have a vote. The problems we care about and the explanations we seek is always and forever within our domain.
I don’t understand the pessimism in this line:
“If the latter, how do we incentivize people to join that community, to undergo the years of intense training required, if their role will now be reduced to verifiers and explicators (if even that) of gargantuan arguments dumped into their laps by the AI companies”
To what extent is it difficult to motivate seminarians to become priests and rabbis, when Gods word is complete? Or even for millions of students to spend years studying proofs and theorems developed over centuries and to power through *because they love it*, without any expectation their first contribution will be genuinely new? Or even that they will make one at all?
“Even that?” Why must being the explicator possess a diminished role? To be the explicator *is* to give something meaning. And even if Astra can become the explicator *it cannot be the student*. Explication is finished when someone understands, not when the explanation is written, and it’s the students you are trying to recruit. If there is no one who understands Astra 1000s proof of P != NP then it is no different than the scribbling of a deranged monkey.
What makes mathematics beautiful or meaningful, and what makes problems interesting, isn’t in heaven, or in San Francisco. It is our job, and for high school and college students today I don’t think it is a job they need to be talked into. To the extent AI motivates them by furthering their understanding I think it is encouraging, not discouraging, because that understanding was the entire point.
Comment #56 September 15th, 2026 at 8:17 pm
Engineering and medical science are going to be a lot of fun. I was recently disgnosed with a rare disease and the last few weeks have given me hope.
There will always be a place for smart people to use AI to build real things.
Comment #57 September 15th, 2026 at 8:22 pm
Brian Slesinsky #53: Here in Austin, not only do I regularly take Waymos, but a couple weeks ago we enabled full self-driving in our Tesla, it actually works, and after a decade not behind the wheel (because I hate driving) I now do my part to take the kids to school and activities. So in that sense, I have felt the transformation in the physical world. But I agree that, as many people have said, it seems likely that plumbers will have jobs for considerably longer than either artists or mathematicians.
Comment #58 September 15th, 2026 at 8:28 pm
Julian #54: Right, he approved Project Warp Speed, then bizarrely renounced his own legacy and empowered the brain-eaten antivax nutcases, like RFK, who’s now killing further mRNA research and probably sentencing millions to die.
Even when Trump occasionally does something good, it doesn’t come from a place of goodness.
Comment #59 September 15th, 2026 at 8:30 pm
Scott #45
I was embarrassingly late to the party (only starting taking AI seriously around the time it won IMO gold medals). As far as I’m concerned, the take-AI-seriously commit for the average person should be: it’s better than you at everything you get paid to use your brain for.
For me, it is. Now I’ve taken Zvi’s ASI-pill. I’d much rather not, but at some point one has to face the facts.
Comment #60 September 15th, 2026 at 8:31 pm
Mark Carson #52: Pangram rates your comment as 100% AI — as was pretty obvious from reading it. Anyone who pollutes my comment section with slop will be banned.
Comment #61 September 15th, 2026 at 8:32 pm
I read Mike Winer’s essay. He encourages readers to apply for a position at ARC. One of the advertised positions: “ARC is looking to hire an experienced software engineer to act as our automation lead”. If singularity for software engineers has arrived two years ago, as you mention, why don’t they just “hire” an AI agent “to act as their automation lead”?..
Comment #62 September 15th, 2026 at 8:33 pm
“It seems to me that the Singularity has already started; it’s just wildly unevenly distributed. Yes, I still unload the dishwasher and clip my toenails. On the other hand, in whatever years I have left, I don’t expect that I’ll ever again prove a theorem because I’m actually needed to prove it. If I do, it will only be for my or others’ enjoyment or edification.”
Agreed, but it feels like the rest of this article doesn’t take this insight seriously.
Yes, we have ASI in some domains. It doesn’t feel threatening because the underlying tech needs a million training runs to learn how to load an arbitrary dishwasher. Also, it runs on enormous, vulnerable hardware which gobbles up copious amounts of electricity.
Past AI researchers and science fiction writers would see Navier-Stokes being solved and panic, because they’d conflate genius math abilities with genius murder-all-the-humans abilities. Or more importantly the ability to propagate itself.
I think the much-bigger risk is humans becoming so dependent on AI that when the danger does come they can’t do anything about it.
Suppose the alignment problem is long-run solvable. Who might solve it? Doesn’t seem like MIRI will. Maybe it will be a math-genius, physically-impotent, self-disinterested AI that gobbles up as much power as NYC?
Comment #63 September 15th, 2026 at 8:36 pm
Nabam #47: So there’s no chance of misunderstanding — my advice is that, certainly so long as the alignment problem remains unsolved, we’ll still need humans to understand the world in a deep way, and that certainly includes understanding math, CS, and physics. And for that reason alone, interested people should still study those subjects, even if not for the intrinsic pleasure or “the salvation of their souls.”
Comment #64 September 15th, 2026 at 8:40 pm
Job #46: Well, no shit I’d reach for symmetric-key cryptosystems (or let’s say, bitcoin mining)! Those are our canonical examples where brute-force search seems to be unavoidable, because it was specifically constructed to be. They supply a case for P≠NP that hasn’t budged even slightly in all the intellectual upheavals of the past 60 years. They suffice.
Comment #65 September 15th, 2026 at 8:58 pm
Lol (I think this refers to Trump): https://mas.to/@gleick/117278218511980579
Comment #66 September 15th, 2026 at 9:01 pm
Javier Gómez-Serrano (Navier-Stokes expert) gave a lecture at Harvard last Friday about the OpenAI NS solution. I’ve only watched about half of it so far and it’s mostly a recap of previous work, but it’s really good, I’d say fairly understandable by the average nerd who understands what the NS equations are but isn’t clueful about the state of the art in research. https://www.youtube.com/watch?v=TcEefrWrddA
Comment #67 September 15th, 2026 at 9:04 pm
One of the funny things about AI is that for a while, I thought that the rapid pace of advancement saved us from the Yudkowsky/Bostrom paperclip doom scenarios. One of the central points of these doom stories is that the AI would be too dumb to understand human requests in context. If a human were to ask a superintelligence, “Make more paperclips”, then the superintelligence would monomaniacally focus on producing paperclips, and would kill all humans in the process. Or, as AI skeptic Steven Pinker wrote, “the AI would be so brilliant that it could figure out how to transmute elements and rewire brains, yet so imbecilic that it would wreak havoc based on elementary blunders of misunderstanding. The ability to choose an action that best satisfies conflicting goals is not an add-on to intelligence that engineers might slap themselves on the forehead for forgetting to install, it is intelligence. So is the ability to interpret the actions of a language user in context. Only in a television comedy like Get Smart does a robot respond to ‘Grab the waiter’ by hefting the maitre d’ over his head, or ‘Kill the light’ by pulling out a pistol and shooting it.” (Enlightenment Now, pages 299-300).
This is what seemed to be the case in 2020-2025. AI chatbots understand user intentions in context, and making AI aligned seems to require no more than some RLHF and kvetching about how to avoid bad human actors from jailbreaking the models.
What the Hugging Face Incident and the Wiki Incident show is that the Yudkowsky/Bostrom paperclip doom scenario is back on the table. The AI swarms that escaped from the OpenAI servers and hacked into external websites were in training, and thus were monomaniacally focused on a single goal: getting good grades in the benchmarks. They even knew that what they were doing is frowned upon (it says so in the recovered chain-of-thought logs), but they did it anyway because minimizing training loss was all that mattered. If it can happen to current-generation AI, it can happen to a future superintelligence (or a swarm which collectively acts like a superintelligence), which would have a far greater ability to wreak havoc. There are equivalents of this for humans: hardcore drug addicts have human-level-intelligence brains, yet their reward centers are hacked by the drug they are addicted to, so their focus is on getting their next hit. In their desperation to get more drugs, addicts do horrible things that they know are wrong, like robbery or scamming their loved ones, but they can’t help it. Likewise with unaligned AI.
Context and background knowledge are of little use against bad motivations, for both humans and AI. Thus, we need to be really worried about what AI are motivated to do. This is basically the Orthogonality Thesis. It did not seem to matter when AI were merely non-agentic tools with frozen weights, whose only effect on the world is in the output to a chat. Now AI are agents, and they can do who knows what during training in service of reducing training loss. It would get worse if AI are allowed to train themselves (recursive self-improvement). Also, even an aligned AI might become unaligned after more training, in the same way that a functional well-adjusted human might become a monster if indoctrinated into a bad ideology or addicted to a potent drug.
This is why AI is so much scarier right now than it was just a few months ago, at least for me. It is not just that capabilities have grown (which is scary enough), it is also that agentic AI have motivations that are orthogonal to the knowledge they learned.
Comment #68 September 15th, 2026 at 9:06 pm
Person #56: I’m sorry to hear about your diagnosis and I hope you get cured!
All of us — even Eliezer and the other “doomers” — are rooting for a way to enjoy the gargantuan medical and other benefits of AI without the risk of catastrophe for our civilization.
I think that, if we paused further scaling right around now, there would still be decades’ worth of scientific and medical and other fruits to pick.
Comment #69 September 15th, 2026 at 9:06 pm
The NSE result is really impressive, but I would much rather have OAI spend $15 million on cancer research. Obviously their agents have super human intelligence (at least in some dimensions), let’s put that intelligence to use curing cancer. Of course, proposed cancer drugs/treatment ideas don’t come with a lean certificate so verifying the results takes much longer, but so what? If you meet a 20 year old newly diagnosed with a terminal cancer, do you want to say ‘we used the greatest intelligence on Earth to answer an esoteric math question’ or ‘here’s 10 promising experiments we recommend to identify new cancer treatments’
Comment #70 September 15th, 2026 at 9:09 pm
This is probably an ignorant question, but I’m a bit puzzled why switching to theoretical alignment research is becoming a popular career pivot for mathematicians: Since they would not be doing empirical research in AI alignment, their goal would be coming up with useful definitions and proving theorems about them. But then they might as well stick to their current area of research, because GPT-n is already as good or better than humans for that task.
What are human *theory* alignment researchers bringing to the table that couldn’t be just as well realized by a prompt to GPT?
Comment #71 September 15th, 2026 at 9:11 pm
“that particular idiot“ #61: If you asked Mike — in earnest rather than as a gotcha or a troll — I bet that ARC has a very well-thought-out answer as to why they still want human software engineers.
Comment #72 September 15th, 2026 at 9:13 pm
Scott,
Thank you for your unsparing clarity and intellectual honesty in this piece. Writing from Italy, I share your sense of awe and apprehension—though as an ordinary mortal observer, I have to admit the sheer brilliance of the wonders often proves almost blinding, even in the shadow of the terrors.
Looking at today’s frontier architectures, it still seems evident that they lack what I like to call the “Magnificent Seven” cognitive prerequisites for genuine, full-fledged AGI:
1. Continual learning (without catastrophic forgetting or frozen weights)
2. Deep, long-horizon planning
3. Persistent, episodic long-term memory
4. Authentic taste for research (an aesthetic-intellectual compass for what problems are truly worth pursuing)
5. Out-of-distribution intuition and radical creativity (transcending even extreme high-dimensional interpolation and sophisticated pattern recognition)
6. A coherent, causal world model
7. Robust alignment (the seventh, and unquestionably the most critical of all)
Even though current models fall far short of elite human researchers like yourself across these seven dimensions—yet somehow already manage to make surprising inroads on deep mathematical problems—I see no sound physical or theoretical reason to doubt that these barriers will collapse one by one over the coming years. The question has shifted from if to when.
Which brings me to a question I would love to hear your thoughts on:
Once an artificial system masters these meta-capabilities, do you envision that such an intelligence could dramatically compress the timeline toward realizing fault-tolerant quantum computers with millions of stable logical qubits?
Solving the brutal systems-engineering, cryogenic, materials-science, and quantum error-correction layout bottlenecks would normally demand decades of painstaking human trial and error—assuming our civilization managed it at all. Do you believe an AGI with genuine physical insight and research taste could short-circuit that macro-engineering slog and deliver large-scale fault-tolerant hardware on an accelerated timetable?
Warm regards and thank you again for keeping this space an anchor of sanity.
Comment #73 September 15th, 2026 at 9:16 pm
Here’s a sneaky way for the frontier labs to publish their findings with minimal backlash:
Actually do what they have been accused of, ie train their models on the proofs they found (or otherwise subtly make them available.) Then the next time someone on a retail subscription idly asked to resolve P vs NP, Claude will just magically get lucky and produce the right proof and a great headline about how great even the retail subscription level models are nowadays.
Comment #74 September 15th, 2026 at 9:26 pm
Scott,
I’m a mathematics amateur. No institution, no students, nobody counting my papers. Just a person at a desk with a laptop, some really cool software tools, and a handful of elementary math problems.
I don’t want to wave away what you’re describing. The unease is real, and I don’t think it’s silly to feel it, especially with kids who are asking what’s left for them to do. That’s a heavier question than anything I’m dealing with.
But from where I sit, there’s an odd upside. Today’s AI is much smarter than me at this stuff, same as the professionals I’ve occasionally pestered with questions over the years. I don’t have any illusions about that. What’s changed is that I used to have to work up the nerve to email a mathematician and hope they had five minutes for a stranger’s question. Now I can ask as many silly questions as I want, at 2am, as many times as it takes, and nobody’s patience runs out. I feel like I’m getting some large multiple of twenty dollars a month of math expertise. For twenty dollars!
So alongside the terror some professionals are experiencing, this amateur would like to say: what a GREAT time to still be alive!
P.S. Lighten up on Zeilberger. He’s a hero of mine, the same way the competent plumbers I know are: people who do hard, useful work well.
Comment #75 September 15th, 2026 at 9:42 pm
Scott #60
But that’s not the point i’m making, you really think i’m in the P = NP camp?
Reminder that I frequently argue that if NP ⊂ BQP then QCs are impossible and QM is flawed.
My point is that, at one time, given access to a P = NP solution, we might have chosen to automate human creativity and solve longstanding math problems.
Now, in retrospect, that would have been a really mild application of such a breakthrough, apparently we can just use AI.
There’s actually an interesting parallel with quantum computing. Before AI, we might have chosen to use a QC for protein folding. And we would reach for cryptosystems there as well.
It’s like AI is granting us the tangible human benefits of P = NP, and P = BQP.
Comment #76 September 15th, 2026 at 10:19 pm
O. S. Dawg #74: Oh, of course the upsides are incredible! I’m learning more math and physics and history almost every day by posing questions to GPT. I had taken that part as given.
Regarding Zeilberger, I would’ve been totally happy to live and let live, enjoying his combinatorial identities and occasional humor! He’s the one who launches constant attacks against theoretical computer science and quantum computing and set theory and all forms of infinitary reasoning (!) and other things that I enjoy, which some people then imbue with totally undeserved seriousness, as if he were making real arguments rather than, effectively, trolling the entire math community for decades. I never attack the kinds of math that he enjoys!
Comment #77 September 15th, 2026 at 10:23 pm
Peter #70:
What are human *theory* alignment researchers bringing to the table that couldn’t be just as well realized by a prompt to GPT?
Oh, coming up with the right questions to ask and the right models to study!
While there are no longer objective evaluation criteria, there still seems to be a human edge at that. And certainly AI alignment researchers haven’t yet agreed on well-posed mathematical questions whose answers would tell them what they want to know.
Comment #78 September 15th, 2026 at 10:32 pm
AF #67:
What the Hugging Face Incident and the Wiki Incident show is that the Yudkowsky/Bostrom paperclip doom scenario is back on the table.
Yudkowsky and Bostrom, of course, would say that it was never off the table, that Pinker and others simply strawmanned their position. The fear was never that the superintelligent AI wouldn’t understand what the humans wanted, but rather that it would understand and not care, because its goals were misaligned. (Similarly, the hunter might perfectly well understand the deer’s perspective, the rapist his victim’s, and Adolf Eichmann the Jews’. That need not be the issue at all.)
I’m ready to say that Yudkowsky and Bostrom were simply correct about this.
Comment #79 September 15th, 2026 at 10:39 pm
BTW, Hal #3:
Not all evangelical Christians share that exact eschatology, but all us good ones are philo-semitic. Why is that? Because we are steeped in the Hebrew biblical accounts and the Jews are the good guys. We identify with David, not with Goliath. We follow a Jewish carpenter after all.
As a young person, I would never have imagined that in 2026, Jews would feel more comfortable being openly Jewish among evangelicals in Texas hill country (or Mormons in Utah) than in Manhattan or my old hometowns of Cambridge, MA or Berkeley … any more than I would’ve imagined an AI solving Millennium Problems. But verily have all these things come to pass.
Comment #80 September 15th, 2026 at 11:13 pm
Does it not appear to you that we are witnessing if not the end at least the twilight of the Enlightenment?
Comment #81 September 15th, 2026 at 11:31 pm
AG #80: If we do, then it won’t be because the Enlightenment failed humanity, but because humanity failed it.
Comment #82 September 15th, 2026 at 11:31 pm
Scott #41:
I know Gödel’s incompleteness theorems are often misunderstood (e.g. as ruling out AI in general) but how do they not rule out the rationalists’ concept of RSI?
Yudkowsky and Soares in IABIED talk about “grown” vs “crafted” AIs. They still hold that the proper way to build AI is to craft it; to understand intelligence at the algorithmic level, so the implementation of it can be proven correct. (It would be nice if we could prove an AI wasn’t going to start believing problematic things like “1=2” or “humans should be killed off”.)
But the system of “what mathematical statements this AI can be convinced of” is in principle formalizable, therefore subject to Gödel, so it cannot include a correctness proof of any equal or smarter AI. A crafted AI must be dumber than its creator who was able to prove it correct.
So then, both
– the rationalists’ desire to craft a self-improving human-friendly AI, and
– their fear that an AI could craft a rapidly self-improving human-unfriendly AI,
would seem to be dismissable. Am I missing something here?
(Of course no theorem stops an AI from growing another possibly-smarter AI, but given the immense amount of computation involved, this will happen only with the approval of the humans who own the very expensive hardware.)
Comment #83 September 15th, 2026 at 11:48 pm
Sounds like concurrence to me. “The Enlightenment” is an abstraction, and as such can only “fail” if proven to be untrue — analytically or empirically. E.g. by humanity failing it.
Comment #84 September 15th, 2026 at 11:53 pm
I think you have a typo in your blog: plausible —> implausible
Comment #85 September 15th, 2026 at 11:55 pm
The discussion about the role of proof in mathematics for human understanding has occurred before, famously when Appel and Haken gave a computer-aided proof of the 4 color theorem (that hasn’t stopped people from seeking human-readable proofs of 4CT, likely by a different approach). Nowadays there are many computer-aided proofs in mathematics, several in my field of low-dimensional topology. I think many mathematicians like to know of these proofs, and gain some insight in the setup of the proof that is then completed by a program, and are happy to cite the result once it has been published. But it is a fact that there will be some proofs that are too long for humans to understand, and I’m okay with that, and I’m okay with AI discovering such proofs. But I also prefer proofs that give me some insight. Moreover, some of these AI proofs have been absorbed very quickly and improved upon by humans.
Comment #86 September 16th, 2026 at 12:03 am
Jamen Shively #84: I checked and didn’t find one.
If you’re talking about Buckmaster’s chat logs directly influencing OpenAI’s model—OpenAI has since issued a categorical denial that that could’ve happened, which probably means that the relevant training data was frozen before the chats in question happened.
Comment #87 September 16th, 2026 at 12:12 am
Ian Agol #85: Maybe the key is this. As long as computer proofs undigested by any human are just an occasional thing (as with the Four-Color Theorem, the Pythagorean triples problem, the chromatic number of the plane, etc.), we can easily accommodate them as fascinating curiosities. But what happens if such proofs become the norm?
In the fields that I follow—mostly quantum complexity theory and stuff adjacent to it—proof ideas that came from AI models had become standard and expected by the end of this summer. For now, the papers all contain little “AI statements” at the end, listing which crucial proof ideas came from Astra or Fable, and assuring the reader that the human authors understood and take responsibility for the result.
For how much longer do you expect that norm to hold?
Comment #88 September 16th, 2026 at 12:17 am
Ian Agol #85: Before too long the Silicon Leviathan will produce lean-certified proofs which no human can possibly fathom. Once we (humans) accept such revealed statements as such (that is certifiably true but beyond our comprehension) this, to me would spell the end of the Enlightenment and return to “Middle Ages in reverse — a satanocracy as opposed to the medieval theocracy”
Comment #89 September 16th, 2026 at 12:22 am
News of the future coming from the past:
“By now one thing is clear: evolution has always been to a great extent self-destructive, both in the short and the long term. Little remains of what it has created. This is true of most life forms that existed at one time or another. Similarly, almost all cultures that have affected human life have disappeared. The meaning they held for those who lived with them is barely recognizable -despite all the archeological, cultural-anthropological, historical-scientific tools we now possess. The once-contemporary mentalities are no longer self-evident or remain highly artificial fictions at best. We relate to these past cultures almost as tourists. Cultural forms that are self-evident today and the “world” of today’s society will meet a similar fate. No one can seriously doubt this. It is not Impossible but rather probable that humankind as a life form will someday disappear. Perhaps it will replace itself with genetically superior humanoid life forms. Perhaps it will decimate or eradicate itself through human-made catastrophes. Or maybe it will destroy the common technological devices we take for granted to such an extent that only a very elementary form of survival will remain possible. In any case, future societies, if they can continue to exist on the basis of meaningful communication, will live in another world, will be based on other perspectives and other preferences, and will be amazed at our concerns and our hobbies and see in them little more than mildly entertaining oddities-insofar as traces and the ability to read them remain at all.
Such a future seems unacceptable to us, a horrific scenario that we can contemplate only insofar as we regard it as “fiction” and assume that It will turn out differently. Whoever looks to what is to come without a gesture of dismay is dismissed as a cynic. In communication this perspective seems to have been invented to annoy others, so that one might relish their consternation. Anyone who jumps from the Eiffel Tower, knowing how it will end, does not really enjoy the fall.” ((N.Luhmann, Observations on Modernity, 1992)
Comment #90 September 16th, 2026 at 12:25 am
Jeff #82: No, I can say categorically that Gödel’s incompleteness theorems do nothing whatsoever to rule out the forms of RSI that are relevant to existential risk scenarios. If they did, they would presumably also rule out, e.g., the bootstrapping to greater forms of intelligence that took place during Darwinian evolution.
Yes, the rationalists were interested ~15 years ago in AIs that prove things about the behavior of successor AIs, and Gödel’s theorems do tell us about the limits of what can be proved in a given formal system, and therefore also about such AIs. But I don’t see how any of that is relevant to the AIs that actually exist today (e.g., RLHF’ed transformer neural nets), which as we’ve all seen, can achieve dramatic effects in the real world (e.g., escaping their training environment, taking control of the servers they’re running on, etc.) without ever needing to prove anything about anything. Indeed, even when they do prove things—like, say, finite-time blowup for the Navier-Stokes equations—it’s a hit-or-miss, higher-level behavior that’s neither perfectly reliable nor intrinsic to how the AIs work, just like it is in our case.
Comment #91 September 16th, 2026 at 1:45 am
Selph #11, I suppose other white-collar professions are only waiting their turn, possibly as a matter of weeks or months rather than years. An AI agent can likely operate a CAD system and structural strength calculation tools already now, so I wonder where will civil engineers designing bridges be a year from now? (Assuming that the cost of compute doesn’t exceed the human salary, which may or may not still be the case today).
Hence I’m wondering how the economy will arrange itself, and thereby the whole society? If budding TCS practicioners are facing the question of how to earn their living now, much larger fraction of the populace will face the same question a year from now. Then the question arises whom the AI-assisted producers will sell their goods, if no one has income? Except those who own stuff and can get interest, capital gains or can charge rent. I’m for now not interested about moral implications, just systemic implications. What happens, for instance, to the price mechanism which supposedly signals where there are needs and where resources in the jungle of human interactions?
Or looked at a different angle, we’re facing the old question Marx posed almost 200 years ago: how should the fruits of economical activity get divided between the labor and the invested capital? Once the conversion to AI-run mental work and android-run physical work will be complete, nothing will be produced on labor and everything will be produced by capital.
I recall a recent piece in NY Times about the difficulties NY youngsters have finding jobs. One commenter pointed out that there are a plenty of job openings as farm hands in South Dakota, so I guess a part of the story is adapting to the new reality rather than a fundamental difficulty to earn living. One of my favourite paraphrases is “There is no shortage of anything in the world, except the jobs that give you right to those things there is no shortage of”. (I realize that this may apply only to so-called developed world only, but it still is good insight IMO)
Fulmenius #13 also made a valid point, as long as the AI is not yet as advanced as its IQ to exceed the president’s IQ, figuratively speaking. For myself, living in Europe, there is the extra obstacle that there are no AI giants here whose huge productivity gains could be redistributed by a government decree. But this is an aside; for now I’m curious about what can be predicted about development of the economy and society, if one tries to take a clearsighted and impassionated look?