Superduperintelligence

Which way would Pascal wager?

Yesterday Safe Superintelligence Inc., cofounded by Ilya Sutskever, Daniel Gross, and Daniel Levy, was announced. Sutskever’s remarkable history—from the paper that kickstarted modern AI to chief scientist at OpenAI—needs no introduction, and I know Daniel Gross slightly and seriously respect him. These are serious people, unironically trying to create “a safe superintelligence.”

…So what does that mean, exactly? For the industry, and for, y’know, us?

“Draw me a picture of a safe superintelligence” followed by”A little less twee and more mechanical?” You don’t want to see the first one, trust me.

Let’s talk supersports

To talk about superintelligence, you must define what superintelligence is … and therefore what intelligence is … and one of the dark ironies of this AI era is that discussion of human intelligence has become one of the Internet’s flaming toxic waste dumps, laced with bigotry, hatred, and bad faith. So let’s just stipulate that intelligence is not a single trait but rather a linked bundle of traits — I think that’s not too controversial? — and then fall back on a handy sports metaphor.

When serious people analyze human physical excellence, in this case soccer, they make pizza charts like this:

The bigger the slice, the better you are, until at the limit you are the 100th percentile for humanity for that particular metric. (Kevin De Bruyne is very good at soccer.)

Now imagine a comparable pizza chart for intelligence. There are many ways you could divide its categories—fluid vs crystallized, language/wordcel vs. math/shape-rotator—and then itemize its particular fields: memory, arithmetic, pattern recognition, self-awareness, ability to turn a novel situation to one’s benefit, etc.

Intelligence discourse includes claims that a characteristic known as g exists, which is correlated with all of these things and also IQ, such that it’s meaningful to have a single metric for intelligence. (Amid counterclaims re all of these things.) When discussing sports it’s much less controversial to say that there is such a thing as “athleticism,” which is at least somewhat hereditary, and that it is related to — but not dispositive of! — all the metrics above. Jonathan Kuminga is acclaimed as the Golden State Warriors’ most athletic player, but definitely not their best player. That said, Steph Curry is still pretty damn athletic.

Accepting that all metaphors are wrong but some are useful, let’s pursue this one. It seems superathleticism could reasonably mean several different things:

• Being better than human, i.e. scoring above 100, at any one metric. This is very boring; computers have had this kind of superintelligence, e.g. super-memory and super-arithmetic, for a very long time.

• Being better than human at an entire category of metrics, such as “attacking,” “defending,” or “possession.” (Note that the above metrics are all in-game, implying a certain baseline level of performance; an immobile robot that kicks goals with perfect accuracy every time would still get 0 in all shooting metrics, because it would never be in position to take a shot in a real game. This deals with the “can pass the bar exam but is still a 0 as an actual lawyer” problem; such tests are merely predictive of in-game capabilities, for, and only for, humans.)

• Being more athletic than any human, i.e. having more of that raw single characteristic necessary for athletic greatness … but not necessarily superhuman at any given metric, or even overall (again, Kuminga vs. Curry.)

• Being better than human at most of the given metrics.

• Being better than human at all of the given metrics.

All of the last four sound like reasonable, but distinct, definitions of superathleticism. Worth noting: we can also distinguish between this and artificial general athleticism, the latter being maybe something like “at least the 10th percentile in every metric.”

Note that, following this metaphor, using three of those four definitions, it’s entirely possible to reach superintelligence without achieving artificial general intelligence! Possible, and to my mind, highly likely; I’ve argued before that when we think about AI we “are prone to a Fundamental Extrapolation Error, in that we extrapolate assuming modern AI is on a path to intelligence like ours.”

OK. We’ve defined terms, ish, which is a start. Now let’s talk about consequences. In particular, the most consequential thing that many people postulate superintelligences will eventually be able to do: recursively self-improve, i.e. make themselves smarter. That obviously doesn’t apply to superathleticism … right?

Stares at you in practice

Except of course sports are actually all about recursive self-improvement. You go to the gym to get into better shape and practice, and the better shape you’re in the more you get out of the gym, and as Gary Player famously once put it, “The more I practice, the luckier I get.” Professional athletes literally spend most of their time recursively self-improving! …The problem is, of course, as any gym-goer knows, is you soon enough hit diminishing returns.

This is true of human intelligence as well. Whenever anyone tells you that the purpose of education is not to inculcate facts but to teach students how to learn, they’re saying real education invariably comes from recursive self-improvement. Classes, readings, question sets, internships, proofs of concept — recursive self-improvement, one and all. No matter what our g, if there even is such a thing, education makes us smarter through recursive self-improvement.

…But there are limits, and there are diminishing returns. You can argue most humans don’t push the limits of their intelligence hardware, but you can’t argue those limits don’t exist. Today’s AI already has vaguely comparable limits: training compute, inference compute, data, context windows, (lack of) memory, etcetera. AI cannot recursively self-improve, yet … but even if we develop a superintelligence-as-defined-above which can, it will hit some sort of hardware limit eventually, after which its returns will diminish.

All of which means that the real unanswered philosophical future-of-the-future question is: will such a superintelligence be able to bootstrap itself into another, superior, hardware substrate entirely? And then obvious corollary is: will we?

Stand back, I’m a professional science fiction author

(No, really, I am. And I gotta say, the fact that we’re talking about this as all is a reassuring testimony to the power of science fiction!)

OK, now we can finally start talking about the consequences of superintelligence. One being: knowing whether / when we manage to create a superintelligence does not, by itself, actually tell us all that much about the future! Instead there are at least four disparate futures:

1. Superintelligence is hard.
   Here, the scaling hypothesis hits a wall. Over the next few decades AI keeps getting better at a somewhat disconnected grab bag of abilities, generating ever more complex and consistent outputs, but never quite fuses these abilities together into any kind of coherent approach to understanding or problem-solving the world, much less exceeding the human capacity to do so. It remains an increasingly and ultimately incredibly useful tool, accelerating our recursive self-improvement called “progress” in a myriad of ways, but does not become an entity one could imagine taking the reins of progress itself.

2. Superintelligence is easy (ish), recursive self-improvement is hard.
   We build something(s) intellectually superior to ourselves in every way … but not unrecognizably so. It has its own limits, can see farther than us but not outrun us; is probably very expensive to build or run, or both, such that superintelligence is the purview of superpowers and megacorps only; and is accepted as an oracle, not a threat, in the same way that nobody was particularly worried about John von Neumann conquering the world, even if he had really wanted to. But this oracle is incredibly valuable. With it, scientific and technical advances happen far faster … as do military ones, increasing Cold-War-esque tensions. This is an interesting future in that we technically attain superintelligence but it isn’t really transformative — the world is still recognizable, changing faster, but still at a manageable / comprehensible rate. At least at first. Although the temptation to use these oracles for policy planning, too, will become irresistible, for both better and worse … and because recursive self-improvement is hard, in this future, many will seek to actually use our new oracle to work on making it easier, an initiative that perhaps could — eventually — lead to

3. Superintelligence is easy (ish), recursive self-improvement is easy … until it’s not.
   Here we pass very quickly through the above “oracle” stage into a kind of “intelligence demigod” stage; terawatt-consuming frontier systems become much more intelligent than us, such that we don’t really know what they’re up to or how they work … but at the same time they have many decidedly mortal limitations, we have some insight into their workings and motivations and alignments, their advances underpin a new era of spectacular human wealth, and maybe even new definitions of what it means to be human as these demigods accelerate biotech progress, and we soon exist in a kind of symbiosis with them, each species (because we’re definitely talking about a new species here) reliant on the other to maintain their way of life. Furthermore, mere gigawatt-systems are widely accessible, relatively cheap, also beyond human intelligence (though not nearly as smart as the frontier demigods), and difficult to monitor at scale. Essentially every aspect of life, including what “life” even means, is transformed.

4. FOOM!
   This is the classic “fast takeoff” scenario; recursive self-improvement turns out to have no meaningful limits at all, because … uh … reasons. (Yes, I too find this very unlikely.) Within months of achieving superintelligence, or even faster, we have bootstrapped not just a new species but a new pantheon of gods into being, who will have powers of nanotechnology, superphysics, biotech like putty in their hands, etc., and we will just have to hope that they judge their creators with mercy rather than rendering us all molecule by molecule into a Dyson sphere with which to accelerate their own ever-exponentiating godhood. Fine, I exaggerate … but not by much.

Pascal’s Strawmen

Lots of very smart people, especially those in the field of AI, think superintelligence is, if not “around the corner” exactly, at least “within reach.” (Though I sure wish they would all clarify which definition of superintelligence they’re using exactly.) Lots of other smart people, some of them also in AI, think this is … very much not so. (Although again, I sure wish, etcetera.) (My former employers) Metaculus show a median forecast of 2032, but a mode of 2027, for AGI, which is again not really superintelligence, but is at least directionally linked:

The obvious question is “who to believe?” A more nuanced question is “how best to think about this?” An even more nuanced and interesting one is “What are the stances that we would predict humans beings to take, independent of what is actually correct?”

We can postulate the human beings are bad at extrapolating from exponential growth. Thus, many people would predict scenario 1 above — superintelligence is hard — even if they were to be shown evidence that it is incorrect. So we should bear in mind that this stance aligns with a known cognitive bias. This doesn’t mean it’s wrong! But it’s a valuable data point.

However this is also true of “superintelligence is imminent.” Human beings have an extremely strong and well-established tendency to believe in predictions of sudden massive, and especially potentially eschatological, transformations. I don’t know your religious faith or lack thereof, but I do know that whatever it is, the implication is that you believe most of the rest of humanity, including many extremely intelligent people, holds completely false and ridiculous beliefs. This doesn't mean the inescapable religious overtones of “superintelligence is coming” mean it’s wrong! But it’s a valuable data point.

In the 1970s, techno-pessimists thought that the world would be doomed by overpopulation, citing Malthus and the Club of Rome, while techno-optimists thought we were on the verge of conquering space, and would be living in lunar cities and O’Neill colonies by the end of the decade. Both groups, as it turned out, were completely, utterly wrong. This doesn’t mean either group is wrong today! But it’s a valuable data point.

People always tend to overestimate the significance of what they do. Journalists are convinced their work is the lifeblood of a civilized society, whereas society … increasingly disagrees. (I agree with the journalists, but then I would, wouldn’t I?) This doesn’t mean people in the AI industry are wrong about the significance of AI progress! But it’s a valuable data point.

Pascal’s Upshot

So, given that there’s no group that we should clearly believe, how should we think about this? Perhaps Blaise Pascal, and his famous wager, gives us an answer.

If transformative superintelligence really is only a few years away, the world will soon be completely unrecognizable, where “unrecognizable” means “there’s no good way to prepare for that tomorrow today; even if the tsunami of change is small enough to surf, you won’t know how it hits until it does. If it’s a tsunami of good, you’ll enjoy it so much you won’t care you didn’t really prep, and if it’s bad … it won’t much matter what you do.”

But if this kind of world-shaking transformation isn’t immediately en route, and you act as if it is — by cashing in your retirement savings, quitting your soon-to-be-laughably-obsolete job, not encouraging your kids to learn how to use/program “classic” computers today, all the things that people would do if they really believed transformative superintelligence was around the corner — then you face a ton of downside if you’re wrong, and very little upside if you’re right.

So here, at long last, is my answer to my opening question. I’m very confident AI will become a more important and more powerful tool; don’t let the existence of overenthusiastic “this is a cool new hammer, therefore everything is a nail” people obscure that! I have no especially informed opinion about whether this means transformative superintelligence is, for better or worse, truly in our reach. (I do have strong feelings, in that I’m skeptically hopeful / hopefully skeptical.) But I am pretty convinced that the way to live — for almost all people, and almost all businesses — is to act as if it’s not.