We were only pretending to engage with each other. But it wasn’t our fault. We had to be, because talking about bad faith is Not OK. Continue reading
I'll pretend to ask, and you'll pretend to answer
5 Replies
Author Archives: Benquo
The order of the soul
Our higher cognitive functions have two modes: a drive to bias nature towards certain outcomes, and an appreciation of structural symmetry in the arrangement of the universe. In standard three-part models of the soul, bias maps well onto the middle part. Symmetry maps well onto the "upper" part in ancient accounts, but not modern ones. This reflects a real change in how people think. It is a sign of damage. Damage wrought on people's souls – especially among elites – by formal schooling and related pervasive dominance relations in employment. Continue reading
Geometers, Scribes, and the structure of intelligence
When people talk about general intelligence in humans, they tend to talk about measured IQ. While a lot of variation in IQ is really just variation in brain health, and probably related to variation in general health, there are at least two distinct modes of general intelligence in humans: fluid intelligence and crystallized intelligence.
Fluid intelligence is pretty much anything you can use a spatial metaphor to think about, and is measured pretty directly by Raven's Progressive Matrices. It's used for puzzle-solving.
Crystallized intelligence, on the other hand, relies on your conceptual vocabulary. You can do analogical reasoning with it – so it lends itself to a fortiori style arguments.
I don't think it's just a coincidence that I know of two main ways people have discovered disjunctive, structural reasoning – once in geometry, and once in the courts. Continue reading
Oops Prize
if you don’t correct errors, you don’t get anything done, because you stay wrong. I don't think we do enough to reward saying oops.
Lately, I’ve been complaining about ways the EA community’s been papering over problems in ways that forgo this sort of learning. But while complaining is important, on its own it doesn’t offer any specific vision for how to do things. At the recent EA Global conference in Boston, I was reflecting with a friend on what sorts of positive norms I would like to see in the discourse.
One example of something I wish I saw more of, is people publicly and very clearly saying, "we tried X, it didn’t work, so now we’re stopping.” Or, “I used to believe X, and as a result asked people to do Y, but now I don’t believe X anymore and don’t think Y is a particularly good use of resources.” People often invest a lot of social capital in their current beliefs and plans; admitting that you were wrong can cost you valuable social momentum and mean you have to start over. You might worry that people will associate you with wrongness. We need communities where instead, clear admissions of error or failure are publicly acknowledged as signs of integrity, and commitment to communal learning and shared model-building.
So I'm offering a prize. But first, let me give an example of the sort of thing we need to be praising more loudly more often. Continue reading
Is Silicon Valley real?
Scott Alexander argues that Silicon Valley's reputation - that it's decayed into bubbles around nonsense products such as the San Francisco-based Juicero - is undeserved, that there's still plenty of innovation around. But I think what's really going on is that Silicon Valley doesn't exist anymore. Continue reading
Bad intent is a disposition, not a feeling
It’s common to think that someone else is arguing in bad faith. In a recent blog post, Nate Soares claims that this intuition is both wrong and harmful:
I believe that the ability to expect that conversation partners are well-intentioned by default is a public good. An extremely valuable public good. When criticism turns to attacking the intentions of others, I perceive that to be burning the commons. Communities often have to deal with actors that in fact have ill intentions, and in that case it's often worth the damage to prevent an even greater exploitation by malicious actors. But damage is damage in either case, and I suspect that young communities are prone to destroying this particular commons based on false premises.
To be clear, I am not claiming that well-intentioned actions tend to have good consequences. The road to hell is paved with good intentions. Whether or not someone's actions have good consequences is an entirely separate issue. I am only claiming that, in the particular case of small high-trust communities, I believe almost everyone is almost always attempting to do good by their own lights. I believe that propagating doubt about that fact is nearly always a bad idea.
It would be surprising, if bad intent were so rare in the relevant sense, that people would be so quick to jump to the conclusion that it is present. Why would that be adaptive? Continue reading
Actors and scribes, words and deeds
Among the kinds of people, are the Actors, and the Scribes. Actors mainly relate to speech as action that has effects. Scribes mainly relate to speech as a structured arrangement of pointers that have meanings.
I previously described this as a distinction between promise-keeping "Quakers" and impulsive "Actors," but I think this missed a key distinction. There's "telling the truth," and then there's a more specific thing that's more obviously distinct from even Actors who are trying to make honest reports: keeping precisely accurate formal accounts. This leaves out some other types – I'm not exactly sure how it relates to engineers and diplomats, for instance – but I think I have the right names for these two things now.
Summary
Everyone agrees that words have meaning; they convey information from the speaker to the listener or reader. That's all they do. So when I used the phrase “words have meanings” to describe one side of a divide between people who use language to report facts, and people who use language to enact roles, was I strawmanning the other side?
I say no. Many common uses of language, including some perfectly legitimate ones, are not well-described by "words have meanings." For instance, people who try to use promises like magic spells to bind their future behavior don't seem to consider the possibility that others might treat their promises as a factual representation of what the future will be like.
Some uses of language do not simply describe objects or events in the world, but are enactive, designed to evoke particular feelings or cause particular actions. Even when speech can only be understood as a description of part of a model of the world, the context in which a sentence is uttered often implies an active intent, so if we only consider the direct meaning of the text, we will miss the most important thing about the sentence.
Some apparent uses of language’s denotative features may in fact be purely enactive. This is possible because humans initially learn language mimetically, and try to copy usage before understanding what it’s for. Primarily denotative language users are likely to assume that structural inconsistencies in speech are errors, when they’re often simply signs that the speech is primarily intended to be enactive. Continue reading
Effective Altruism is self-recommending
A parent I know reports (some details anonymized):
Recently we bought my 3-year-old daughter a "behavior chart," in which she can earn stickers for achievements like not throwing tantrums, eating fruits and vegetables, and going to sleep on time. We successfully impressed on her that a major goal each day was to earn as many stickers as possible.
This morning, though, I found her just plastering her entire behavior chart with stickers. She genuinely seemed to think I'd be proud of how many stickers she now had.
The Effective Altruism movement has now entered this extremely cute stage of cognitive development. EA is more than three years old, but institutions age differently than individuals. Continue reading
An OpenAI board seat is surprisingly expensive
The Open Philanthropy Project recently bought a seat on the board of the billion-dollar nonprofit AI research organization OpenAI for $30 million. Some people have said that this was surprisingly cheap, because the price in dollars was such a low share of OpenAI's eventual endowment: 3%.
To the contrary, this seat on OpenAI's board is very expensive, not because the nominal price is high, but precisely because it is so low.
If OpenAI hasn’t extracted a meaningful-to-it amount of money, then it follows that it is getting something other than money out of the deal. The obvious thing it is getting is buy-in for OpenAI as an AI safety and capacity venture. In exchange for a board seat, the Open Philanthropy Project is aligning itself socially with OpenAI, by taking the position of a material supporter of the project. The important thing is mutual validation, and a nominal donation just large enough to neg the other AI safety organizations supported by the Open Philanthropy Project is simply a customary part of the ritual.
By my count, the grant is larger than all the Open Philanthropy Project's other AI safety grants combined.
(Cross-posted at LessWrong.)
OpenAI makes humanity less safe
If there's anything we can do now about the risks of superintelligent AI, then OpenAI makes humanity less safe.
Once upon a time, some good people were worried about the possibility that humanity would figure out how to create a superintelligent AI before they figured out how to tell it what we wanted it to do. If this happened, it could lead to literally destroying humanity and nearly everything we care about. This would be very bad. So they tried to warn people about the problem, and to organize efforts to solve it.
Specifically, they called for work on aligning an AI’s goals with ours - sometimes called the value alignment problem, AI control, friendly AI, or simply AI safety - before rushing ahead to increase the power of AI.
Some other good people listened. They knew they had no relevant technical expertise, but what they did have was a lot of money. So they did the one thing they could do - throw money at the problem, giving it to trusted parties to try to solve the problem. Unfortunately, the money was used to make the problem worse. This is the story of OpenAI. Continue reading