AAUP-NC Critical AI Policy Project

We Can Totally Just Unplug It, and Also We Should

Hello all! Apologies for the radio silence; I hope you’re doing well! I’ve been working on a longer thing specifically pertaining to higher education policy for the last few weeks, but it’s not quite ready; in the meantime, this piece is a follow-up to the TESCREAL piece I wrote earlier.

I want to be clear: I don’t think it’s exactly the most urgent problem we’re staring down right now with regards to AI in NC, but I do think it’s important to keep an eye on, especially because I’ve been hearing more lately about astroturfed “AI Safety” advocates appearing on campuses. (I get into those a bit near the end, but I’ll write more about them in the coming week or two.)

In any case, we’ve got lots of exciting stuff coming up. See you soon!


Last year, the writer and “AI safety” advocates Eliezer Yudkowsky and Nate Soares published a bestselling book called If Anyone Builds It, Everyone Dies, about the dangers of “superintelligent” AI. I've been meaning to review this book for months now, but I keep putting it off, in large part because – spoiler alert – I find it immensely frustrating. So I am going to instead review a representative video made by Soares, one which purports to answer some the most common objections to his concerns.

Soares broke up his claims into ten questions, so I'm going to take them one-by-one. This is going to take multiple letters; I only get through the first of his questions here. (The others should go faster, though. I hope!)


Preliminary note about “general intelligence”

Here's something to lay out at the start. The stronger claims Soares will make here – particuarly the ones about AIs “getting smarter” of their own accord – are broadly predicated on the belief that there exists something we could call “general intelligence.” I don’t think Soares uses the term in his video, but this concept of “general intelligence” underpins the concept of “AGI,” or artificial general intelligence. AGI – specifically, “misaligned” AGI, or AGI that doesn’t share “our” values – is the thing Soares and Yudkowsky are afraid of.

The easiest way for our purposes to describe general intelligence (known in the psychometric literature as the “g factor”) is that it’s the thing IQ purports to measure. More concretely, the g factor is an attempt to explain why students who performed well on a specific test of cognitive ability performed well on other, seemingly-unrelated tests; it does so by positing the existence of a single underlying capacity: a general intelligence.

Like the concept of IQ, the concept of general intelligence is – to put it mildly – fraught. To put it less mildly: I personally believe both IQ and general intelligence are unrigorous fictions whose persistence is largely attributable to the eugenics movement1. The methodology underpinning both the original findings and later attempts to reproduce these findings has been brought into question; more to the point, it’s a huge leap to say that these correlations, if they do exist, are expressions of a single inborn genetic factor and not, for example, a result of the fact that well-resourced schools tend to have good teachers across disciplines.

To be clear, I don’t lead with this in order to argue that Soares or others concerned about AGI hold straightforwardly eugenicist beliefs.2 Nor am I making an argument about whether all learning is domain-specific.

Rather, I believe that the pseudoscience of general intelligence has led “doomers” like Soares astray in at least one specific way: namely, in their foundational assumption that improvements in one intellectual domain presage improvements in others. For our purposes, the basic comparison with human learning stands: getting better at generating solutions to math problems might make you better, say, formal logic. But it’s not going to teach you anything about history or how to negotiate interpersonal conflict, let alone the more dramatic things Soares and his cohort speculates about.

And so, to his first question:


1. How can something so error-prone be so dangerous?

In a cursory sense, I agree: something error-prone can be very, very dangerous. But closer attention to the way Soares articulates his concern makes it clear that his interests lie elsewhere. The question he’s really asking, I think, is something like: “how can something so error-prone be capable of achieving something like the superintelligence I am worried about?” In order to understand how I got to this interpretation, I’m going to have to explain a bit about his intellectual milieu.


“The concern is that they’re making these AIs smarter”

At the beginning of his answer, Soares takes a tack that should be familiar to anyone who’s spent time listening to advocates of AI:

A couple years ago, people thought it was going to take decades before AI could solve the Math Olympiad gold medal challenge, which is one of the most challenging math competitions for teens. That was solved in 2025. It was not solved in the mid-40s like these predictions said. It was solved in 2025. Then in 2026, there's claims that AIs are solving Millennium Prize problems, which are some of the most important math problems that have been standing open for decades with million-dollar prizes attached to each of them. So, the technology is improving and it's improving fast. The issue is what happens if the AIs keep improving.

This is a by-now classic story: AI is improving at an astonishing clip. Whatever it can’t do for now, just wait until two years from now!

To be clear, the customer experience of LLMs has improved over the past couple of years. This is mostly because AI wrappers – the software designed to mediate between the user and the model – have improved a great deal; the most notable of these is probably still Anthropic’s TUI-based coding assistant wrapper Claude Code).

But recent “frontier” model releases would not necessarily corroborate the intuition that they’re getting better at a shocking clip. As the economist Paul Kedrosky argues, “rolling relative model gains have fallen from their 2024 peak.” The most famous recent dud was probably OpenAI’s GPT-5.

But even the most spectacular rollout of the year — Anthropic’s allegedly too-dangerous-to-release Mythos model — wasn’t all it was cracked up to be. All eight of the flagship security vulnerabilities cited in the Mythos system card as evidence of its superpower were also discovered by cheaper, older open-source models. (We’ll get more into Anthropic’s cybersecurity theater in the next letter.)

I believe that this is the case because the really big improvements in LLMs – the kind we saw from about 2018 to 2022 – resulted from our then-new capacity to train models on massive corpuses of training data. But scaling seems to have hit the wall: a paper from the journal Neurocomputing put it in March, “as model size increases, performance gains could become unpredictable and non-monotonic, indicating diminishing returns from scale alone.”

All of which is to say: there’s a lot of reason not to believe Soares’s claim that LLMs are still getting better as rapidly as he fears they are.

Let’s return to his claim, then: that LLMs are hitting milestones faster than people thought they would. The logical next question: which milestones? And, more to our point, which people?


“People thought it would take decades”

In the video description, Soares provides two citations for his claim that this is all moving faster than we thought it would. Neither of them is about the Millennium Prize problem. (He offers one source about the Millennium Prize, but it doesn’t include any predictions.) Because I addressed this whole deal in an earlier letter, I’m not going to dwell on it for too long, but it’s worth discussing for a second:

The findings from OpenAI seem potentially remarkable. Minimally, they seem like evidence that LLMs may have a role to play in mathematical proof generation in the future (assuming that they become cheaper to use; the Millennium Prize proof, according to their per-token pricing, took upwards of $20,000,000 to solve).3 Computer-based proof assistants have been a thing in math for decades now; LLMs seem like they may be a neat addition to the toolkit.

But OpenAI’s math work ultimately reinforces something we’ve known about LLMs for a long time: they perform better (a) with highly-structured, human-constructed languages such as programming languages and formal math, and (b) in instances where binaristic output can be checked for correctness, because this is very helpful for reinforcement learning. (I've talked about this vis-à-vis computer programming, but it applies to math, too.)

To return to the point I made about “general intelligence” earlier: formal math, while very cool, is about as specific an intellectual domain as it gets. (I say this with nothing but love, I promise, but anyone who’s known a mathematician knows that being very good at math won’t necessarily make you very good at anything else.) And when it comes to LLMs, researchers have confirmed this finding: as that same paper from Neurocomputing puts it, “the emergence of advanced abilities is highly task-dependent.”


Winning the Olympiad

Ok, so let’s return to the milestone predictions Soares does cite. There are two of them; both have to do with the Math Olympiad contest.4

TechCrunch

The first is a TechCrunch article about the Math Olympiad test. This article is mostly about how OpenAI maybe cheated (not a surprise to anyone observing the controversies around their more recent math findings). But the piece doesn’t cite any specific predictions: both assertions about the pace of development are editorial commentary, so it’s not going to be much help for our purposes.5

Metaculus

His second citation is a page from a website called “Metaculus,” which describes itself as “an online forecasting platform focusing on topics of global importance.”6 It is, in essence, a predictions market without the gambling.

Before we unpack the (numerous) implications of that, let’s take a look at the specific page he cited: “When will an open-source AI win a Gold Medal in the International Math Olympiad?”

Unfortunately, this page raises many, many more questions than it answers. For one thing, it’s a completely different question, because it’s limited to open-source models – not the closed-source “frontier” models cited in the TechCrunch article, which are the ones he’s worried about. (As of the writing of this piece, the date they were supposed to have won by passed just over a week ago, meaning either that the website hasn’t verified the claim or that AI is, in fact, developing more slowly than we thought.)

Because I am such an excellent and good-faith interlocutor, I looked around the website for a question that better resembled his. The only thing I really found was this far less popular one – “When will a publicly-available LLM achieve gold on the International Math Olympiad?” – but because the models which won in 2025 weren’t publically available yet, the accepted forecast on this page comes from this past July, a full year after the announcements cited in the TechCrunch article.

Even granted that YouTube videos typically do have lower standards for citations than papers or books7, this is, frankly, a total mess. I can’t do anything with any of this. It seems like classic motivated reasoning: like he started from the argument and then backfilled a couple of sources without checking to see if they were actually relevant. (In fairness, as someone who’s gotten into quite a few arguments on the internet myself, I’m sympathetic to the impulse; still, it’s shoddy stuff.)


Metaculus and co.

Let’s circle back to “Metaculus.” Metaculus is a website where anyone can sign up and make a prediction: it is, as I said, a prediction market without the gambling.

But prediction markets’ claim to utility comes from the (misguided) belief that putting money on the line incentivizes rigor, coupled with a Hayekian faith in the capacity of price signals to disclose otherwise-inconceivable truths about reality.

So how does Metaculus claim to keep people honest, if not by gambling? What motivates people to play this game?

Their FAQ claims the same noble goals as Kalshi and their ilk: “aggregated predictions of strong forecasters offer one of the best ways of predicting future events.”8 But I think a more honest answer comes further down the page: “As well as being worthwhile, Metaculus aims to be interesting and fun, while allowing participants to hone their prediction prowess and amass a track-record to prove it” (emphasis mine).

Reputation makes more sense as a motivator to me – especially given how small the community seems to be. Only 300 people voted in the forecasting poll cited by Soares. (Only 30 voted in the other one I found.)

And this community is a familiar one, at least to those who read the earlier TESCREAL newsletter: namely, the community of “effective altruists.”


Next time, on the newsletter

Metaculus was founded by “two researcher scientists, Anthony Aguirre and Greg Laughlin.” Aguirre was a co-founder of something called The Foundational Questions Institute, as well as the better-known Future of Life Institute.

It is not much of a stretch to say that both of these “institutes” are appendages of the broader “EA” movement. It’ll take more time than I have here to lay out the tangled financial and ideological web supporting this claim.

But this movement, as I’ve discussed before, is disproportionately influential; while small, insular, and fringe, a shocking number of Silicon Valley’s elite have been affiliated with it.

And here we arrive at the ultimate reason I’m writing about all these weird “AI Safety” guys: this movement is reaching university campuses. The not-at-all-shady-sounding AI Safety Seeding Initiative provides free mentorship to anyone interested in starting an AI Safety club at their school; they work to connect people with funding from money-granting organizations like the Pathfinder group. The first page alone of search results for “effective altruist student fellowship” reveals uncannily-similar eight-week discussion-based “fellowship” programs at Harvard, Yale, Stanford, UPenn, Cambridge, and Cornell. I’ve personally heard reports from people at multiple universities discussing the sudden presence of well-funded representatives and clubs on college campuses.

Soares is a relatively minor player in the whole thing, but it’s really worth hammering home his reach. Apparently, within six months of its release, the book has sold more than 150,000 English-language copies; it’s already available in English, Dutch, Portuguese, and Japanese, and there are apparently plans to translate it into Korean, Mandarin Chinese, Russian, Ukrainian, Vietnamese, Polish, and German. Perhaps more concerningly, despite being the less famous of its two authors, a cursory search of YouTube shows that in the last couple of months he’s been on CNBC, Fox News, Sky News, and official podcasts affiliated with Bloomberg and Newsweek – as well as a bunch of podcasts I’ve never heard of, each of which has 50k-250k views. Harrowingly, his two-hour interview with Tucker Carlson has received 1.5 million views in two months; a panel discussion featuring him on a channel called “The Diary of a CEO Clips” somehow has 3.2 million.9

Here’s the takeaway: Soares is reaching literally millions of people with his message about the coming AI apocalypse. He says preventing AGI is most important thing humanity has ever faced; as we’ll get into later, this priority actively impedes his and his cohort’s willingness to fight against actually-existing problems with actually-existing “AI.” Still, his cohort’s interpretation of developments has dominated the discourse around AI. And after all that, in the video in which he is claiming to answer all the most pressing questions about rogue AI for a popular audience, the citations he provides for the central claim underpinning his entire life’s work are a misleading link to an effective altruist gambling website and a TechCrunch article from a year ago that’s mostly about something else. Anyway, the next nine claims he makes shouldn’t take quite as long to respond to as this one. See you next time!




Footnotes

1 The classic treatment of this is probably Stephen Jay Gould’s The Mismeasure of Man, but a lot of people have written well about this, especially in the wake of the controversy surrounding Charles Murray and Richard Hernnstein’s The Bell Curve. For a convenient rundown of how terrible that book is, here’s a good YouTube video about that book by the YouTube creator Shaun.

2 Not that AI researchers have done much to beat the allegations. Nick Bostrom, the author of the massively influential Superintelligence, infamously wrote (in an email which also used the n-word) that “blacks are more stupid than whites.” While he apologized for it, a careful reading of his apology shows he’s perhaps less recalcitrant than one would hope: he apologizes for the racial slur and for his “provocative communication style,” acknowledges that “unequal access … leads to inequality in social outcomes, including sometimes disparities in skills and cognitive capacity” (sic, emphasis mine). He defers on the question of “genetic contributors to differences between groups in cognitive abilities,” saying it’s not his area of expertise and it doesn’t interest him, then says “eugenics” means a lot of things, actually, and while of course he doesn’t support “horrific atrocities”…etc.

More directly, here’s a bit from an earlier letter I wrote about one of the most important papers ever written about the prospect of AGI: “While at Microsoft, [Sébastien Bubeck, the researcher currently in charge of OpenAI’s math department] was the main author of the highly influential, intellectually worthless “Sparks of Artificial General Intelligence” paper, notable for, among other things, sourcing its definition of intelligence from an op-ed defending the racist work of pseudoscience The Bell Curve.” For a broader overview of the relationship between the historical project of AI and eugenics, the movie Ghost in the Machine is worth watching.

3 As I lack subject matter expertise and have obvious ideological biases – and as the whole point of my argument is that it doesn’t actually matter how good LLMs are at finding math proofs – this is my official stance on the question. Unofficially, however, I think the plagiarism allegations make it highly likely that these results are far, far less impressive than they seem. Even more than this, I’m sympathetic to the arguments that proofs like these are themselves highly overvalued.

4 Nota bene: While I have no experience with cool fancy math like the Millennium Problem, I did come in second in my school’s fifth-grade Math Olympiad contest, so this one is completely in my wheelhouse.

5 But it’s also worth noting that the specific innovation was in the realm of natural language processing – the fundamental thing LLMs are good at:

Last year, Google scored a silver medal at IMO using a “formal” system, meaning it required humans to translate problems into a machine‑readable format. This year, both OpenAI and Google entered “informal” systems into the competition, which were able to ingest questions and generate proof‑based answers in natural language.

6 Metaculus was founded by a co-founder of the Future of Life Institute – which is to say, a member of the same social and financial circles as Soares and Yudkowsky. This is not a “gotcha,” per se, but a reminder that this is a very small circle of people making and propagating this story.

7 I’m not getting to this part until next letter, but it’s worth noting here that the book he coauthored with Yudkowsky is, if anything, even less rigorously sourced.

8 They cite “the political scientist Philip Tetlock” here, though they don’t link to anything. Come on, guys! Gimme a link to click!

9 I am not going to look up what “The Diary of a CEO” is. This newsletter should make clear that my head is filled with enough cursed knowledge as-is.

Thoughts? Leave a comment