Extropia's Children, Chapter 4

What You Owe The Future

(Each chapter of Extropia’s Children can stand alone, but see also Chapter 1, “The Wunderkind”; Chapter 2, “This Demon-Haunted World”; and Chapter 3, “Extropicoin Extrapolated”)

i. Certainly better than ineffective

Effective Altruism, aka EA, is unquestionably the most prominent of Extropia's children. Major media outlets such as The Economist and The New Yorker write about it frequently. Its principles now guide an estimated $30 billion of philanthropical donations, disbursed at a rate of about $600 million/year. It is also increasingly controversial. Still, puzzled readers may well ask: what even is effective altruism, and what does it have to do with AI, rationalism, and the ancient extropians mailing list?

Much has been written about effective altruism, but, briefly: at root, EA is about focusing on altruism / philanthropy which a) treats all human lives as equally valuable b) actually works, efficiently and effectively, as opposed to feeling good or seeming like it ought to help. Put another way: “Using evidence and reason to figure out how to benefit others as much as possible, and taking action on that basis.”

The effective part is not particularly controversial. I count myself among the many who strongly agree, having spent years traveling in the developing world / Global South mostly via public transit, during which I developed a healthy contempt for most forms of development aid. Some might cavil that saying “altruism which leads to better results is better” implies that ineffective ways of helping people are therefore bad, which … yes, it does, but that's just part of open-eyed consequentialism.

More controversial is that if you take “as much as possible” literally, you get what some call “Strong EA,” which means not just helping others effectively but dedicating your entire life to it. Some live at the poverty line despite substantial salaries because they give so much away. Others worry about enjoying art, or eating ice cream, or having children, when they could be spending that time and effort on helping others. While this seems simultaneously "admirable in a disturbingly ascetic way" and "unlikely to scale," one's altruism can still be relatively effective with or without that degree of dedication. It is notable however that effective altruism is unusually correlated with this kind of monklike or even — to its critics — cultlike abnegation.

Most controversial by far, though, is that EA often, and seemingly increasingly, focuses on ‘longtermism,’ meaning altruism which strives to benefit the far future and its inhabitants, rather people alive today. (Focusing on e.g. global poverty and/or climate change is called ‘neartermism.’) Longtermists argue that, in the same way the health of people in Malawi should matter greatly to Americans, the wellbeing of people centuries from now should matter greatly to us today; that when it comes to giving and helping, displacement in time is not meaningfully different from displacement in space. In practice this means an intense focus on existential risks, aka x-risks. i.e. those which might threaten humanity itself and therefore extinguish all future people. Those risks are, in turn, headlined by — you guessed it! — AI risk … i.e. the creation of a superintelligent AI which might exterminate us all.

ii. Rationalism is EA is EA is rationalism

The effective altruism community traces its roots back to 2011. Since then it has grown to such significance that some might dispute it was born from Eliezer Yudkowsky's rationalism, the LessWrong community, and ultimately the extropians; and it's true that rationalism is not its only antecedent. (Another is Peter Singer.) To others, the idea that effective altruism has always been deeply entwined with rationalism may seem a long deceased equine. (Both LessWrongers and effective altruists commonly call themselves "EA/rationalist.") Kindly indulge a few more obligatory kicks, just to make their intertwining very clear.

So: Open Philanthropy, aka OP, essentially a vehicle for donating Facebook cofounder Dustin Moskovitz's fortune, is explicitly an effective altruism organization, and — given we're talking about ~$11 billion — certainly one of the most significant. Much of OP's team hails from the rationalism movement and/or remains connected to it.

The most obvious example is Luke Muehlhauser, a Senior Program Officer at OP, who was MIRI's executive director for years. But there's much more. For instance, Rebecca Baron, a Grants Assistant, attended an “AI Safety Camp” which counts MIRI's Evan Hubinger among its mentors. He in turn helps fund projects at the Long-Term Future Fund … which is “managed by a team led by Asya Bergal, a Program Associate at Open Philanthropy” … who previously worked at AI Impacts, funded by MIRI.

In 2019 that same LTFF fund (despite dissent) recommended a (tiny) donation to print copies of Yudkowsky's Harry Potter fanfic for Russian math medalists. This was recommended by Oliver Habryka, who has been Director of Strategy at the Center for Effective Altruism US, and the Project Lead for LessWrong 2.0, while apparently still disbursing funds for LTFF.

Claire Zabel, another Senior Program Officer at OP, seems to be a frequent co-author and/or interlocutor of Buck Shlegeris, who worked at MIRI, and now works at Redwood Research, “a longtermist organization working on AI alignment” … whose board includes Holden Karnofsky, the co-CEO of OP … whose “Senior Research Analyst” Ajeya Cotra recently recorded a podcast about AI progress with none other than Yudkowsky himself. Etcetera, etcetera, I could go on, but the horse is dust. One final note: Open Philanthropy is MIRI's top donor, to the tune of $13.5 million and counting ... not counting the $3.5 million also donated to CFAR.

Perhaps the best analogy is to think of rationalism, MIRI, CFAR, effective altruism, LessWrong, Slate Star Codex / Astral Codex Ten, and other associated communities as islands of varying size in an archipelago. Each is distinct, but at the same time, it often makes sense for the rest of the world to approximate them as a single entity.

iii. From Berkeley to Oxford to Oxford to Berkeley

It is true that Yudkowsky’s Bay Area is not the only pole of effective altruism. The other is Oxford, home to William MacCaskill, author of the recent bestseller What We Owe The Future. His recent New Yorker profile says of EA's origins:

That club grew into an organization called GiveWell, which determined that, for example, the most cost-effective way to save a human life was to give approximately four thousand dollars to the Against Malaria Foundation, which distributes insecticide-treated bed nets. In the Bay Area "rationalist" community, a tech-adjacent online subculture devoted to hawkish logic and quarrelsome empiricism, bloggers converged on similar ideas. Eliezer Yudkowsky, one of the group's patriarchs...

So far so unsurprising. But then another familiar name appears:

The Centre for Effective Altruism now dwells in Trajan House ... Nick Bostrom, a philosopher whose organization, the Future of Humanity Institute, also shares the space...

To which you might say: wait wait what? Nick Bostrom? The very same Nick Bostrom who often interacted with Yudkowsky on the extropians mailing list, subsequently co-authored papers with him, and was on the board of the Singularity Institute for Artificial Intelligence, which became MIRI? And the answer is: That's right. The shadow of that ancient 90s mailing list is very long indeed.

Bostrom's story is more European and (slightly) more conventional than Yudkowsky's, but no less extraordinary. A Swede more fully named Niklas Boström, he too is something of an autodidact; his own 2015 New Yorker profile explains “To his parents' dismay, Bostrom insisted on finishing his final year of high school from home by taking special exams, which he completed in ten weeks.” After acquiring a Ph.D from the London School of Economics, he taught at Yale, all the while keeping up a voluminous correspondence on the extropians list.

In 2005 he founded Oxford's Future of Humanity Institute, which he continues to direct today, “investigating big-picture questions about humanity and its prospects.” Its team includes extropians-list alumni Anders Sandberg, Eric Drexler, and Robin Hanson alongside formative effective altruists Toby Ord and MacCaskill himself. In 2014 Bostrom published his bestseller Superintelligence: Paths, Dangers, Strategies, which popularized the idea of AI risk (and, incidentally, profoundly influenced my own novel Exadelic.) His most poignant work, though, is the “Fable of the Dragon Tyrant,” which laments the inevitability of death ... and suggests that if there is any chance it is not inevitable, we are making a terrible moral mistake by not overthrowing its tyranny as soon as possible.

Bostrom's proximity to effective altruism and rationalism is far from merely geographical. He is cited often in MacCaskill's profile. His own said, seven years ago: “The institute shares office space with the Centre for Effective Altruism, and both organizations intersect with a social movement that promotes pure rationality as a guide to moral action.” And, notably, Nick Bostrom is also sometimes called “the Father of Longtermism.”

iv. Nearterm vs. longterm, bailey vs. motte

MacCaskill's New Yorker profile observes:

In retrospect, "Doing Good Better" was less a blueprint for future campaigns than an epitaph for what came to be called the "bed-nets era." During the next five years, a much vaster idea began to take hold in the minds of the movement's leaders: the threat of humanity's annihilation.

This in turn is followed by much discussion of AI risk. (As well as risks like pandemics; but AI gets most of the attention.) There are two overlapping interpretations of this transition. One is that effective altruism started with Peter Singer, whose philosophy argued for distributing malaria bed nets, but then, as people thought harder about the mission, evolved towards Yudkowsky and Bostrom, and began to focus on existential — and especially AI — risk.

The other, far less charitable, interpretation is that modern effective altruism is a motte-and-bailey argument. This metaphor, widely popularized in the EA/rationalist community by Scott Alexander (about whom more anon) is “when you make a bold, controversial statement [the bailey]. Then when somebody challenges you, you retreat to an obvious, uncontroversial statement [the motte], and say that was what you meant all along, so you're clearly right and they're silly for challenging you. Then when the argument is over you go back to making the bold, controversial statement.” In this interpretation, “helping as many people as possible with maximum efficacy” is the motte ... and longtermist focus on x-risks, especially AI risk, is the bailey.

I realize this is contentious, maybe even pejorative. But there does seem to be a significant mismatch between EA's distinct roots in Singer and Yudkowsky, i.e. neartermist “help people today” vs. longtermist “help people tomorrow, mostly by working on hypothetical existential risks and especially AI risk.” (Even Singer makes this point.) The New Yorker's description of MacCaskill as

careful to sidestep the notion that efforts on behalf of trillions of theoretical future humans might be fundamentally irreconcilable with the neartermist world-on-fire agenda

does not exactly help resolve matters. (Trillions is, incidentally, actually a gross understatement; Bostrom has speculated that the future could include trillions of trillions of trillions of trillions of humans, mostly living in computer simulations.)

Ultimately, asking effective altruists what their top priority is seems like a pretty reasonable question, especially since kind of its whole point is to focus on the most efficacious initiatives. If its advocates can't or won't answer, a motte-and-bailey interpretation does not seem completely unreasonable. This is doubly true given e.g. The New Yorker quoting Nick Beckstead of Future Fund:

it now seems more plausible to me that saving a life in a rich country is substantially more important than saving a life in a poor country.

That ... probably isn't what most people think of when they hear “effective altruism.”

v. Wrestling with the future

In fairness, much of the EA community recognizes this mismatch and seems uneasy about it. A recent highly-voted post on the EA forum says: “Effective altruism is no longer the right name for the movement ... Trying to prevent humanity from going extinct does not match people's commonsense definition of altruism.” This parallels Scott Alexander's recent argument against EA's focus on longtermism. (He suggests that instead they clarify that AI and other x-risks are imminent threats to the lives of people today.) In a comment, MacCaskill largely concedes the mismatch. Open Philanthropy openly wrestled with it some years ago, and seems to have temporarily resolved it by giving a lot to both neartermists and longtermists, easy when you have $11 billion to play with.

In general effective altruism is quite good about trying to recognize and admit its errors (viz. “adjusting its priors in response to new evidence,” as a Bayesian would put it.) The “Our Mistakes” page on the Centre for Effective Altruism's web site is voluminous, and includes:

In 2018, we published the second edition of the Effective Altruism Handbook, which emphasized our longtermist view of cause prioritization, contained little information about why many EAs prioritize global health and animal advocacy, and focused on risks from AI to a much greater extent than any other cause. This caused some community members to feel that CEA was dismissive of the causes they valued.

From late 2017 until early 2021, the Reading List on the EffectiveAltruism.org homepage included content on a variety of longtermist cause areas but not global poverty (which was the community's most popular cause at the time per the EA Survey). Similarly, the Resources page was disproportionately skewed toward longtermist content.

Our joint work with Leverage [yes, that Leverage!] was a source of significant disagreement among CEA staff. As there was turnover in our staff and leadership, we decided not to work with them again, and haven't worked with them since 2018.

It's also worth noting that back in 2012, Holden Karnofsky, then the Co-Executive Director of GiveWell — still apparently in its Peter Singer days — wrote a detailed and quite critical analysis of the Singularity Institute (which had not yet changed its name to MIRI.) This is largely a critique of the institute's effectiveness and outputs, not its philosophy, but I don't want to leave you with the incorrect impression that MIRI and EA have always been one-voice shoulder-to-shoulder. The longtermist vs. neartermist tension continues.

vi. Supporters and critics

As effective altruism's profile has grown, so has that of both its supporters and critics. Its supporters include, to varying degrees, Facebook co-founder Dustin Moskovitz; cryptocurrency billionaire Sam Bankman-Fried, who briefly worked at the Center for Effective Altruism; Skype co-founder Jaan Tallinn; and world's richest man (and OpenAI founder) Elon Musk. (MacCaskill, somewhat bafflingly, acted as a go-between for Bankman-Fried and Musk regarding the latter’s acquisition of Twitter.) On the other hand, Timnit Gebru, formerly of Google, calls it a cult; Charity Navigator's leadership has written at length about “The Elitist Philanthropy of So-Called Effective Altruism”; and even LessWrongers have suggested it's hypocritical.

More interestingly, the former / lapsed extropian Jean Guerrero recently wrote:

The New York Times, the New Yorker and other media have given longtermism fawning coverage this year with little or no mention of its deranged core […] Longtermism is often framed as a way to protect Earth. But its architects care less about ecosystems than about making sure nothing stops humanity from reaching what Bostrom calls “technological maturity.” That’s a nice way of characterizing that moment when people turn into bits.

Émile P. Torres, a philosopher who studies existential threats and has extensively investigated longtermism, warned that the traction longtermism is gaining makes it “the most dangerous secular belief system in the world today.” Leading longtermists have arrived at abhorrent conclusions, such as that philanthropy should focus on saving and improving wealthy people’s lives more than poor people’s.

Widely-respected-in-Silicon-Valley physicist / programmer / author / Y Combinator Research Fellow Michael Nielsen has written not so much criticism as thoughtful nuanced analysis of effective altruism in general:

I'm cautiously enthusiastic about EA's moral pioneering. But it is potentially a minefield, something to also be cautious about. [...] the reason many people are bothered by EA is not that they think it's a bad idea to "do good better". But rather that they doubt the ability of EA institutions and community to live up to the aspirations. [...] if your social movement "works in principle" but practical implementation has too many problems, then it's not really working in principle, either. The quality "we are able to do this effectively in practice" is an important (implicit) in-principle quality.

[Note that this is not the first theory-vs-practice criticism encountered in this series.]

There is a related attitude toward the arts common in EA. Singer is blunt about this: you can't really justify the arts.

(Tangentially: it is interesting to ponder what truth there is in past UN Secretary-General Dag Hammarskjöld's statement that: "It is more noble to give yourself completely to one individual than to labor diligently for the salvation of the masses." This is, to put it mildly, not an EA point of view. And yet I believe it has some considerable truth to it.)

Less centrally, the part of the principle about "using evidence and reason" is striking. There is ongoing change in what humanity means by "evidence and reason", with occasional sharp jumps. Indeed, many of humanity's greatest achievements have been radical changes in what we mean by evidence and reason.

Disappointingly, Nielsen's thought-provoking essay barely touches on longtermism, but it's worth considering that Hammarskjöld quote in depth. If it's wrong to give yourself to an individual rather than labor for the salvation of the masses ... then isn't it just as wrong, for just the same reasons, to disproportionately help the people you love, when you could be aiding a larger number of desperate strangers? If so, what Nielsen calls “Strong EA” does not sound advisable or sustainable or scalable or, well, frankly, human.

Longtermism is thornier yet. MacCaskill writes “I now believe the world's long-run fate depends in part on the choices we make in our lifetimes,” but lately that's been true of every lifetime; you’ll notice we didn't all die in a Cold War nuclear apocalypse. The difference is that there was (and is) nothing hypothetical about those thousands of nuclear warheads. Longtermists treat the many trillion humans to come as no less real than those struggling in Haiti and Ukraine and Bangladesh today ... but I take my definition of reality from Philip K. Dick, who knew a thing or two about the subject: “Reality is that which, when you stop believing in it, doesn't go away.” Does that really apply to the those future trillions? Really? Which ones, exactly?

Even if you accept that our lifetimes are a crux period on which all human futures will hinge, isn't that actually an argument for neartermism? Don't we want to help as many people as possible now, and lift them out of poverty, to increase their ability to help save the future? Doesn’t longtermism take money away from them to squander it on highly uncertain hypothetical futures and technologies we (quite openly) don’t even pretend to understand? Isn’t that at least potentially extremely inconsistent with “doing the most good possible”?

Isn't it suspiciously convenient that a cohort of theoretician graduates of elite universities, who overwhelmingly staff EA organizations, have concluded that one of the cohorts of people most important to the future are ... more graduates of elite universities, upon whom think-tank money must be showered, so they can write theoretical AI-risk papers and AI-alignment proofs-of-concept which the future will very possibly look back on as hilariously crude and irrelevant to actual AGI?

Some longtermist concern and altruism seems called for, to be clear. I'm broadly supportive of being good ancestors to our future (as you might expect; few people alive have directed longer-term projects than I.) But the kind of large-numbers logic sometimes used to quantify EA longtermism, the “1 in 1,000,000 chance of exterminating 1,000,000,000,000,000,000,000 future people,” is so handwavey you can always rationalize adding or removing a few zeros to taste, to support any conclusion you want. Peter Singer himself makes a similar point.

The fact Nick Bostrom “considers a distant future in which trillions of digital minds merge into an enormous cognitive cyber-soup” speaks well of his intellectual candor, and is what you'd expect from a community whose origins go back to the extropians list. You'd similarly expect the understanding that those hypothetical future minds, if they come into being, will not necessarily be human as we understand the term. But all this highlights the fact that the futures which effective longtermists plan for are extremely speculative ... especially if you assume, as I do, that the future will always be much weirder than we expect.

As I'm sure is apparent, I'm skeptical of both Strong EA and longtermism. Orwell once wrote “Saints should always be considered guilty until proven innocent.” Like rationalism, effective altruism seems strongly correlated to quasi-religious thinking: ascetic, messianic, eschatological. It makes me very uneasy to see longtermist beliefs (and they are beliefs; see again PKD’s definition of reality) framed as the inescapable outcome of correct thinking, rather than a contentious worldview based on highly arguable foundations. I grant that paying some heed to longtermism seems right, but even so, it's still basically impossible to determine whether those efforts will be effective, or useless … or even counterproductive. That inability to iterate based on outcomes is a huge red flag.

vii. Extropialtruist

In 2003, 23-year-old then-pro-AI autodidact Eliezer Yudkowsky wrote to the extropians (who discussed altruism quite a lot):

As far as leveraged altruism goes, nothing is within the same league of the Singularity, for either efficiency or efficacy, by N orders of magnitude. I really don't see why rational discussion would tend to linger anywhere else.

At the time that was probably not viewed as a seminal example of New Altruist thinking which would come to dictate the distribution of tens of billions of dollars ... but here we are. Let's pause to marvel, once again, how extraordinary it is that this major new social force / moral movement / multibillion-dollar philanthropical strategy can connect both its primary centers of activity, Yudkowsky's Berkeley and Bostrom's Oxford, directly back to the obscure 1990s extropians mailing list.

Up next: What was the great rationalist diaspora? What has the Machine Intelligence Research Institute actually, well, done? And why — given the amazing rise of his philosophies — is Eliezer Yudkowsky now seemingly wracked by utter despair?