Ozy writes that Effective Altruism avoids the typical failure modes of people in developed countries intervening in developing ones, because it is evidence-based, humble, and respects the autonomy of the recipients of the intervention. The basic reasoning is that Effective Altruists pay attention to empirical evidence, focus on what's shown to work, change what they're doing when it looks like it's not working, and respect the autonomy of the people for whose benefit they're intervening.
Effective Altruism is not actually safe from the failure modes alluded to:
- Effective Altruism is not humble. Its narrative in practice relies on claims of outsized benefits in terms of hard-to-measure things like life outcomes, which makes humility quite difficult. Outsized benefits probably require going out on a limb and doing extraordinary things.
- Effective Altruism is less evidence based than EAs think. People talk about some EA charities as producing large improvements in life outcomes with certainty, but this is often not happening. And when the facts disagree with our hopes, we seem pretty good at ignoring the facts.
- Effective Altruism is not about autonomy. Some EA charities are good at respecting the autonomy of beneficiaries, but this is nowhere near central to the movement, and many top charities are not about autonomy at all, and are much better fits for the stereotype of rich Westerners deciding that they know what's best for people in poor countries.
- Standard failure modes are standard. We need a model of what causes them, and how we're different, in order to be sure we're avoiding them.
- 1 Effective Altruism neither can nor ought be humble
- 2 Effective Altruism is based on weaker evidence than EAs think
- 3 Effective Altruism doesn't prioritize autonomy
- 4 Effective Altruists don't change our minds
- 5 Effective Altruism quacks like a duck
- 6 People who throw stones should live in glass houses
- 7 Disclosure
Effective Altruism neither can nor ought be humble
Effective Altruism is often pitched through the narrative that if we sincerely try to do good better, we will end up doing the most possible good one can do. The narrative is not that there's a particular type of doing good that we're specializing in, that other people happen to have neglected; it's a universalist movement. Either we're claiming to be more altruistic than all the other effective people or more effective than the other altruistic ones, or both. This is not a humble claim.
We should expect to have to go out on a limb in order to find extraordinarily effective interventions. If Effective Altruist interventions are so obvious that anyone can easily verify that it's better, and they're still neglected by the rest of the world, that implies that the rest of the world is stupid, evil, or both. If we don't believe one of those things - and in general it seems bad practice to blithely assume that everyone else disagrees with you for one of those reasons - then we should not expect it to be something that's uncontroversially good and cost-effective. We should expect that when we do find an intervention that seems clearly better to us, it will be somehow difficult to explain our reasoning or require some sort of expert knowledge to understand or violate people's moral intuitions or in some other way seem weird or speculative or just not the done thing.
Effective Altruism is based on weaker evidence than EAs think
First, effective altruists tend to emphasize evidence. It’s all too easy to make assumptions that people in developing countries really want what you think they ought to want, regardless of their stated preferences. It’s all too easy to blunder into some complex system you don’t understand and mess everything up, because you have a PhD and none of these people have a fourth-grade education and how could they possibly know more than you do? It’s all too easy to tell yourself a beautiful story about the grateful natives and ignore the facts on the ground. It’s all too easy to decide that clearly what would really benefit people in the developing world is whatever benefits you.
[...] There’s a reason that a lot of recommended effective altruist charities are in public health. Economic development involves a host of assumptions about how economies work and how to trade off different values and so on, any of which can be mistaken and then you have an utter disaster on your hands. Deworming only requires the assumptions that deworming medicine works the same in Africa as it does in Europe and that most people do not like being infested with worms, both of which seem to be on fairly firm ground.
This does not seem like an opinion unique to Ozy - Ozy is just nice enough to write this up on their blog where I can critique it. I used to talk this way too. Many Effective Altruists I've spoken with seem to think that the case for deworming as a high-value intervention is a slam dunk, based on GiveWell's research. They seem to think that we can be very confident that we're making a huge difference in human well-being. Unfortunately, I do not think that this is borne out by the facts. And GiveWell agrees. Publicly.
GiveWell doesn't think deworming is a slam dunk; Effective Altruists ignored this
Let's talk about deworming, a star intervention among GiveWell's top charities. I get the sense that people think there's decisive evidence from randomized controlled trials that clearly shows that deworming charities have huge effects on life outcomes. GiveWell recently published a blog post clarifying their position on this:
Our cost-effectiveness model implies that most staff members believe there is at most a 1-2% chance that deworming programs conducted today have similar impacts to those directly implied by the randomized controlled trials on which we rely most heavily, which differed from modern-day deworming programs in a number of important ways. [...]
The “1-2% chance” doesn’t mean that we think that there’s a 98-99% chance that deworming programs have no effect at all, but that we think it’s appropriate to use a 1-2% multiplier compared to the impact found in the original trials – this could be thought of as assigning some chance that deworming programs have no impact, and some chance that the impact exists but will be smaller than was measured in those trials. For instance, as we describe below, worm infection rates are much lower in present contexts than they were in the trials.
That GiveWell blog post was published after Ozy's post, but GiveWell's uncertainty about the magnitude of deworming's impact was already publicly available on GiveWell's website. It's a bad sign about Effective Altruists' attitude towards evidence that GiveWell needed to yell this loud to have people take what they're saying seriously.
If you read GiveWell's intervention report and follow the links, you will find that there's a sort of evidence pyramid. There's strong base of empirical evidence for the comparatively modest result that deworming pills kill worms. There are two controlled trials (one randomized, the other one close enough) that show effects on school attendance and test scores but not IQ, but they're in areas nearly adjacent to each other during similar time periods, in which infection rates were probably unusually high due to El Niño. Only one of these studies measures the thing we're most interested in, a long-term life outcome improvement: an increase in income as a higher share of treated than untreated kids were able to get comparatively high-wage manufacturing jobs.
This is not terribly weak evidence. Personally, I think this is enough evidence to justify visiting schools in poor villages and making kids take gigantic bitter poisonous pills that often make them feel sick twice a year every year. But let's look at this with clear eyes. "Deworming kills worms" is not what makes this charity exciting to Effective Altruists - people are excited about the Schistosomiasis Control Initiative and Deworm the World because their programs look like they might provide huge improvements in life outcomes, relative to costs. And this depends on the tip of the evidence pyramid. Medicine's track record is not that great. It would not be terribly surprising if under the more ordinary circumstances of areas treated, routine deworming did more harm than good.
The cost-effectiveness spreadsheet linked to in GiveWell's intervention report implies something like a 70% chance that the single study relied on for the long-term life outcomes result is wrong - and another 70% chance that even if the effect is real, it doesn't generalize outside the period and population studied. I would prefer if the cost-effectiveness model were made explicit in the text of the intervention report rather than only in a linked spreadsheet, but it's available, publicly, on GiveWell's website. Some Effective Altruists just seem intent on assuming that the underlying uncertainty doesn't exist.
SCI isn't a slam dunk as an organization either
There are two huge gaping holes in the case for the effectiveness of giving to GiveWell's top-recommended deworming charity, the Schistosomiasis Control Initiative (SCI). (Note that Deworm the World is also a GiveWell top charity, but GiveWell does not believe that they are funding constrained.) The first is that they seem not to be on top of how much money they have or where it's going. From GiveWell's page on SCI:
We wrote in our November 2015 review that SCI's accounting system seemed ill-suited to its needs and that SCI's financial information was limited in scope and prone to containing errors. In December 2015, we spoke to SCI's former Finance and Operations Manager, who confirmed that the financial systems SCI was using were not well suited for NGO grant management, and that SCI had not allocated sufficient staff capacity to managing its finances. [...]
Based on our experiences with SCI's financial reports, we believe that they are prone to containing errors. We detail the errors we have learned about in this footnote (including some errors we found in 2016). Of particular note: (1) a July 2015 grant from GiveWell for about $333,000 was misallocated within Imperial College until we noticed it was missing from SCI's revenue in March 2016; and (2) in 2015, SCI provided inaccurate information about how much funding it would have from other sources in 2016, leading us to overestimate its room for more funding by $1.5 million.
It's also hard to get clear comprehensive data on what they're actually doing:
The evidence we have seen on SCI's track record of reaching those it has targeted for treatment is fairly limited. We have seen results from about half of the countries SCI has worked in and, for those countries from which we have seen results, we have generally seen one year of results, though SCI has worked in the country for several years.
To be clear, I'm not sure that this makes SCI exceptionally sloppy as an organization. Most charities don't seem to try to be even this transparent, and the fact that we have observed all these flaws in an organization willing to talk to GiveWell strongly suggests that similarly respectable charities we aren't looking at in detail are doing even worse.
Effective Altruism doesn't prioritize autonomy
Second, effective altruists care a lot about autonomy. Give Directly does what it says on the tin: it gives poor people in Africa unconditional cash transfers. Much to the surprise of burden-carrying white men everywhere, it turns out that if you give people money they make basically reasonable decisions about what to spend it on: they buy livestock, furniture, and iron roofs. It turns out that people generally know their own needs better than you do, and you should generally trust them instead of assuming that you know better. Consider the effective altruist proverb: if your intervention cannot outperform giving poor people cash, you should just give them the cash. This sort of essential respect for the preferences of people in the developing world speaks well for our ability to actually improve things, instead of just making ourselves feel better.
However, to remind you - SCI is also a central EA charity. Their mission is to go into villages full of poor people, round up the kids during schooltime, and make them swallow gigantic bitter pills full of poison that often make them sick, twice a year, every year, for their own good. Then sending another team in to check the pill-givers' work, to see whether the kids actually got the pills, because we don't even trust the pill-givers. As I said above, I think that the expected value of this program is strongly positive - but it's sure as heck not about autonomy.
If you want a program that delivers medicine in a way that respects autonomy, you might want to look at Living Goods, a GiveWell "standout" charity that distributes medicine (including deworming pills!) and other goods through trained local salespeople who sell them door to door.
To be clear, I don't think that prioritizing estimated effectiveness over autonomy is a mistake in the case of SCI. It looks like the expected value of the best known interventions that optimize for autonomy is not so large as to make earning to give a terribly appealing option. The very best interventions - the ones orders of magnitude better than giving to GiveDirectly - involve some steering power on the part of the donor and the organization.
But we shouldn't pretend that a single giving opportunity exists that combines SCI's expected value, the Against Malaria Foundation's high confidence that the intervention improves outcomes a lot, and GiveDirectly's respect for autonomy and high confidence that more money means more program. These are different things.
Effective Altruists don't change our minds
I don't want to let GiveWell off the hook entirely. Something about their rhetoric and their place in society seems to reliably cause positive portrayals in mainstream publications to say things like:
The most common refrain from experts I consulted was that my priorities pointed in a clear direction: If what you want is to save lives with certainty, several people said, you have to go to GiveWell.
You can't buy certainty. GiveWell agrees - you just can't do it. At least not without a lot more personal involvement than giving to some of the GiveWell top charities. The fact that many Effective Altruists persistently think you can is a bad sign.
Effective Altruism cannot be humble - it demands high-value giving opportunities
Peter Singer persuaded a bunch of people to give all they can to the developing world, under the premise that you can save a life for not very much money. A lot of Effective Altruists were basically persuaded to devote their careers to doing the most good by this argument:
To challenge my students to think about the ethics of what we owe to people in need, I ask them to imagine that their route to the university takes them past a shallow pond. One morning, I say to them, you notice a child has fallen in and appears to be drowning. To wade in and pull the child out would be easy but it will mean that you get your clothes wet and muddy, and by the time you go home and change you will have missed your first class.
I then ask the students: do you have any obligation to rescue the child? Unanimously, the students say they do. The importance of saving a child so far outweighs the cost of getting one’s clothes muddy and missing a class, that they refuse to consider it any kind of excuse for not saving the child. Does it make a difference, I ask, that there are other people walking past the pond who would equally be able to rescue the child but are not doing so? No, the students reply, the fact that others are not doing what they ought to do is no reason why I should not do what I ought to do.
Once we are all clear about our obligations to rescue the drowning child in front of us, I ask: would it make any difference if the child were far away, in another country perhaps, but similarly in danger of death, and equally within your means to save, at no great cost – and absolutely no danger – to yourself? Virtually all agree that distance and nationality make no moral difference to the situation. I then point out that we are all in that situation of the person passing the shallow pond: we can all save lives of people, both children and adults, who would otherwise die, and we can do so at a very small cost to us: the cost of a new CD, a shirt or a night out at a restaurant or concert, can mean the difference between life and death to more than one person somewhere in the world – and overseas aid agencies like Oxfam overcome the problem of acting at a distance.
But the moral force of the argument depends on the strong contrast between the magnitude of what is given up, and the magnitude of what is saved. (Jonah Sinick makes a similar point here.)
GiveWell can be thought of as a test of the hypothesis that we can find exceptionally good giving opportunities, if we use a certain type of high standards of formal rigor and accountability. It turns out that when you try to do that, you can't judge most programs, most organizations don't want to talk to you, and when you finally find something you have moderate confidence in working and an organization you're moderately sure is actually doing the intervention, the expected value is nowhere near the fantastically high results we were expecting.
When conventional giving opportunities that are easy to verify don't turn out to be extremely high expected value, you have a choice. You can accept a lower payoff, or you can start going out on a limb and making personal judgment calls in situations with higher uncertainty, or you can play pretend.
With its SCI recommendation, GiveWell took the second approach. But it seems that lots of EAs are taking the third.
Effective Altruists are chasing yields
In the investment world, there is a concept called chasing yield. In an environment where asset prices are high and the yield on investments is low, investors believe that they ought to be able to find better returns, so they buy assets with higher returns - pretending that they're not taking on more risk by doing so:
“More money has been lost reaching for yield than at the point of a gun.”
Raymond DeVoe, Jr. (February 1995)
Investors may be very tempted to reach for yield in today’s low interest rate environment. [...]
There is no shortage of headlines in the financial press instructing investors where they can find high-yield investment opportunities. I’m not going to debate the investment merits of the various strategies being touted for yield, except to remind you there is no such thing as a free lunch. Investments don’t pay 6% in a 2% environment unless risk is involved. If you don’t understand the risk or don’t think any risk exists, you should run the other way.
Investors chase yield by lowering the credit quality of their fixed income investments. BB rated junk bonds are yielding 4.75% versus the 10-year Treasury note at 1.9%. They also chase yield by extending maturities. A 30-year treasury bond is yielding 2.7%, almost a full percent above the 10-year. There are also income funds with creative strategies, preferred stocks, master limited partnerships, leveraged income funds, etc. all promising better yield for the income-starved investor.
It's worth noting that investors are humans, and anything that will tempt an investor in investment-world will tempt you in your world as well. When you buy into a narrative that a certain sort of opportunity is available, and make life decisions based on that assumption, it's very hard to notice when the evidence suggests that this situation is no longer true - or maybe that it never was.
It can hurt to learn that you're not going to be able to keep a promise. It's really bad news, and at first it feels to hard to bear. It hurts to learn that you're not going to be able to help others as much as you'd hoped. And it's the most understandable thing in the world to flinch away from that.
This seems like basically what's going on in Effective Altruism. Public EA figures tacitly promise that you can help poor people, with high confidence, cheaply. Effective Altruists make life choices based on the assumption that there are opportunities to save a live for a thousand dollars or less with near certainty, so there's some motivation not to notice when reality fails to live up to our demands.
But when we succumb to that motivation, we are not responding to the evidence. Because of this, Effective Altruism is not as evidence based in practice as we like to think.
Effective Altruism quacks like a duck
Ozy's description of the demographics of Effective Altruism don't look promising:
Indeed, its success in avoiding these failure modes is striking, given that its membership is overwhelmingly white and from the developed world and has a collective racial politics that can be best described as ’embarrassing.’
The prior on this sort of group causing harm through well-meaning interventions in the lives of distant poor people is pretty high. High enough that I feel like there's some Chesterton's Fence analogue that should be invoked here - if there's a pattern that reliably produces some failure mode, you should really be sure you understand how it produces that failure mode before blithely assuming that you'll do better just because your inside view doesn't predict those failure modes. The inside view of the people who came before also didn't predict those failure modes!
People who throw stones should live in glass houses
This blog post has mostly been a criticism of good people trying to do good things. I feel that it's good form for me to offer up some constructive suggestions for other people to criticize.
Recommendations for GiveWell
- Make cost-effective analysis models more explicit, in the body of the intervention report, and again in the body of top charity reports:
- Not because the exact model is likely to be a very precise fit, but precisely to make it clear that you are relying on a model and making inferences, where you believe there's uncertainty, and where you're more confident.
- Filling in the model with numbers will help, not because the exact quantitative result is very meaningful, but because "70% chance" is expressive in a sense that "seems likely but some reservations" isn't.
- This will also make it easier to argue with GiveWell's assumptions, which will create more opportunities for GiveWell to learn, faster.
- Givewell publishes a lot of long pages, that necessarily end up burying very interesting information. Replacing a long blog post with a series of shorter ones published one at a time as info becomes publishable would invite more feedback, for a few reasons:
- Often in very long blog posts and reports, very interesting info isn't headlined up top, which makes readers less likely to notice it.
- This might help with a faster thought-to-publication cycle, which I think would be very high value on its own as it would encourage more feedback (since it's less motivating to comment on stuff when the decision was long ago). This would also increase the timeliness of the external feedback GiveWell gets.
Recommendations for Effective Altruists
- Stop making claims you haven't been personally persuaded of by reviewing the evidence.
- Stop treating GiveWell like a divine revelation of the one true giving opportunity. They're not, and they know it, and they keep saying so. Believe them.
- SCI still seems like a fine organization to donate to if you're short on research time right now but have money to give.
- In the long run, the way to have a high impact is to do your own thinking. Favor understanding what your likely effect on the world is, over abstract considerations of "bestness." The former will cause you to apply more thought, and the latter will create subtle mental pressure to look away from problems with your system.
Reach recommendations for people who want to decisively contribute
- GiveWell's done a great job putting a lot of important information out there, but it's in a fairly dense form and it's often difficult to figure out the exact reasoning behind a recommendation. (Case in point: GiveWell's recent blog post was surprising to many people.) Start organizing this information into a wiki on the effectiveness of interventions and charities. (If you decide to pick up this project, I invite you to ping me or comment below to help coordinate with other interested people. This probably overlaps in ambition with Arbital but is feasible now without the completion of someone else's technical project.)
Next actions for me
- Offer up a more specific critique of GiveWell's and Effective Altruists' intellectual processes and propose alternatives that they can adopt or argue with. I'm throwing stones, so I should show people the basic courtesy of living in a glass house.
I've worked for GiveWell and Open Philanthropy Project in the past. I'm still in touch and on good terms with people there, but the opinions here are my own.