Sarah Perry is a contributing editor of Ribbonfarm.
Author’s note: The thinking that gave rise to this essay was committed in collaboration with St. Rev. Errors, suspicious implications, and dubious conclusions are my own.
On March 12, 1928, the St. Francis Dam, just north of Los Angeles, California, failed catastrophically and sent a wave of water through the valley that caused the gruesome deaths of hundreds of people. No one had predicted the disaster, but after an investigation, it was decided that the dam was built on inadequate soil; the disaster was, in theory, predictable – after the fact. People thought they had control over a massive force (the water), but their control turned out to be illusory.
Considering political and social disasters like the famines of the Great Leap Forward in China or the French Revolution, a similar explanation for the resulting piles of bodies seems apt: social forces over which humans thought they had control (in the sense of being able to coordinate with each other for well-being and sustenance) turned out not to be under their control. No one ever sees it coming, but after the fact everyone is anxious to demonstrate how inevitable it was. If mass violence and destruction seem impossible in our time, consider that everyone who was about to experience revolution felt pretty much the same way. Even the revolutionaries themselves often think they have little chance of success until the revolution is already underway.
Why the surprise? Why does a phenomenon so seemingly inevitable in hindsight go unforeseen? Just as the water in the St. Francis Dam was slowly, imperceptibly undermining the stability of the dam in the weeks preceding the catastrophic failure, the private opinion upon which the success of a revolution depends goes unobserved. It is difficult for anyone to gauge the true popularity of either an incumbent government or the revolutionary opposition, because of a phenomenon Timur Kuran calls preference falsification (in his book Private Truths, Public Lies: The Social Consequences of Preference Falsification – a concept briefly mentioned in an earlier installment).
Preference falsification is an information theory term for the tendency for people to express a public preference that is different from their private, interior preference. For various reasons, certain preferences may not be publicly acceptable to express; they may be punished by execution, or labor camps, or exile, or social exclusion, or at the very least suspicion and a risk of some of these things. When people do not express their true preferences, they are deprived of the opportunity to coordinate with each other to create a more preferable outcome for both. Preference falsification is not just a political phenomenon, but a product of our dual nature, experiencing ourselves on the one hand from the privileged first-person perspective, and on the other hand from the imagined perspective of others. Pretending to have different preferences than one really does may be necessary to maintain a sense of safety, social belonging, and status.
People’s expressed, public preferences are a function of both their interior preferences and the perceived acceptability of revealing them; other people’s expressed preferences serve as a guide for measuring acceptability. So people’s expressed preferences are in part a function of other people’s expressed preferences. Under certain circumstances, when the distribution of preferences is right, a domino effect may be begun by a single dissenter, toppling the status quo of preference falsification. One dissenter may embolden others, and then together with them give the impression that it is acceptable for others to express their true preferences. On the other hand, people whose preferences are satisfied by the status quo may find it wise to begin to falsify their preferences when a revolution begins to look imminent.
Kuran’s primary examples of the preference falsification dynamic are revolutions and counter-revolutions, revolutionaries unseating the old order, then the revolutionaries themselves being unseated, as in Russia, Iran, and Eastern Europe. One repeating feature of regime change is that even as an old preference falsification equilibrium collapses, a new one is erected in its place. For a nearer and fresher example, consider the case of gay marriage in the United States. Gay marriage is a very new issue; there was little support for it in the 1990s, and it was almost unthinkable in the 1980s. Until 2003, it was constitutional for states to criminally prohibit gay sex. Gays experienced the opposite of legal protection, and their social status was poor. Many gay people engaged in preference falsification by hiding their sexuality (i.e., stayed in the closet), and many straight people felt socially obligated to denigrate gays even though they might have secretly preferred not to.
But then the balance of opinion shifted very quickly. The debate on gay marriage is now over, and a new preference falsification regime is in place. Just as there used to be serious social consequences for being openly gay or supporting gay rights, there are now serious social consequences for expressing an opinion against gay marriage (and this applies retroactively to opinions expressed in the time before the current preference falsification equilibrium was in place, as Brendan Eich found out). Defenders of this new order argue that expressing a preference against gay marriage is harmful, and that the moral and social harms vastly outweigh any restriction on expression, especially since no good people really want to express that anyway. This may be true. The important point is that it functions as a new sacredness, something that is so important that we agree not to examine it too closely, and to only speak of it in respectful, ideologically correct terms. But it is disturbing to watch a new sacredness be born, no matter how benign it seems, because like the water locked up for now in a dam, the path it might take in the future is inscrutable and hard to control.
Social forces can have powerful, dangerous, unforeseen effects. The social force that I have been calling sacredness is one of the most powerful. As indicated in the title, it can be weaponized and turned against humans. It can be an engine of cooperation, or a destructive plague, and often displays both natures at once. It travels through many channels, via all forms of speech or human communication, and it operates at many scales. Its language is symbol, entangling map with territory, fusing the word spoken with the thing signified.
Preference falsification is a mechanism by which sacredness can operate. But sacredness is not limited to causing people to lie; it can actually change their underlying, interior, private preferences. We might call the process of weeding and shaping individual preferences “preference husbandry.” This can be performed either by the individual himself, or by social forces acting on him.
How does sacredness operate, and what are its tendencies? Here I will outline thirteen observations about sacredness warfare:
1. “Sacredness binds and blinds.” This is Jonathan Haidt’s mantra from The Righteous Mind, suggesting that sacredness has an emotional component that encourages social bonding and protective outrage, as well as a cognitive component that induces “blindness” – to counterarguments, or to the humanity of heretics.
2. Sacredness implies an in-group and an out-group. In-group members are perceivers of the sacredness (or competent pretenders); out-group members are non-perceivers, heretics, enemies of the group.
3. Sacredness offers belonging to the in-group, but threatens exile for violations of sacredness or insufficient piety.
4. Sacred values must be signaled as valuable in a sufficiently costly manner that sincerity is assured (or a believable public demonstration of sincerity, which anyway has the same effect on both members and outsiders).
5. Ritual energizes the maintenance of sacredness and its power, a costly signal displayed to all (sincere believers or otherwise).
6. Attacking rival sacrednesses or heretics provides evidence of sincerity or commitment to the sacredness. Attacking other believers for insufficient piety will do if heretics are not available.
7. A sacredness battle is won when expressing contradictory or disrespectful thoughts is effectively prohibited by a preference falsification equilibrium; people must either learn to feel the new sacredness, or pretend to.
8. But even as individual battles may be won, new challengers will appear; something that remains sacred for a long time has likely happened upon (evolved, that is to say) defenses against potential rival sacrednesses.
9. Genuinely perceiving sacredness is probably the most reliable, believable way to signal respect for the sacredness and stay in the in-group.
10. Most people are capable, to a limited degree, of altering their perception of sacredness based on social cues; that is, something that is sincerely experienced as sacred, may years later be come to be experienced as silly or mundane, or, more commonly, as evil or vile.
11. Irony and sincerity are not mutually exclusive; it is difficult to measure the components of each in an action.
12. Anything attacking or threatening a preference falsification equilibrium usually wants to replace it with a different preference falsification equilibrium.
13. The new order brought about by a change in sacredness may make everyone worse off than before, and it is impossible to predict its effects before the fact.
The Egregore
Sacredness may perhaps be best understood, by our peculiarly constructed human minds, with reference to egregores – autonomous psychic entities made up of, and influencing, the thoughts of a group of people. Egregoric entities, whether gods or demons or dictators at the center of a cult of personality, are powerful entities, even as they are imaginary – wholly created by and consisting of thoughts, speech, and behaviors. The human mind is a powerful entity, and becomes more powerful in coordination with others. To say that an “imaginary” entity, existing only in human minds, has agency, is not much stranger than suggesting that humans themselves, inscrutable piles of preferences that we are, have agency. Agency is no more than “a quick and dirty way of contextualizing the temporal activities of entropy climbers,” as St. Rev puts it.
Multiplying entities in this way helps make salient how unknowable these entities are – how little we know about the outcomes of social policy and change ahead of time, and how little we can know about them even after the fact. Here be dragons.
These entities vary in how demanding they are – how much they terrorize their component humans. They also vary in terms of what they are able to get out of their humans in terms of productivity. Productivity (measured in harvests, cathedrals, moon landings, pyramids, and the like) can be modeled as an inverted U-shaped function of terror:
A smart entity would keep its component humans in the zone of maximum productivity, not demanding too much from them, nor allowing them to slack off (producing nothing for the glory and amusement of the egregore and anyway perhaps feeling bored and useless). Old entities often have a sabbath attached to them, a frequent, repeating time for rest. The sacralization of this time places a hard-to-erode boundary around the demands that can be placed on constituent humans. This may be a way of keeping “demands” (terror) within the productive zone, and not triggering the entity’s own demise by demanding too much. Holiness signaling by conspicuous loafing does not seem to be the kind of thing that would trigger a preference falsification spiral through the mechanism of sacredness, though it might be fun to try.
A smart entity would alter its demands to maximize productivity, just as a smart virus wouldn’t kill its host quickly. But many viruses do kill quickly (especially newly-evolved ones), and many governments do massacre their own people. Just as newly-mutated viruses are not “smart” in the sense of having virulence-moderating features, social entities have not all had time to evolve features to moderate their demands. Oldness is not a guarantee of friendliness, but it is a filter for certain kinds of unfriendliness.
Timur Kuran emphasizes how complex and inscrutable the workings of public policy are. Within a hierarchical bureaucracy, carrying out directives from above resembles a game of “telephone.” The day-to-day actions of its functionaries may bear scarcely any relation to its supposed mission. Consider also the fact that, as an ordinary and regular aspect of the legislative process, ambiguity is intentionally and knowingly written into legislation, because any of the more specific versions of the law would not be able to pass a vote. Legislators decide that a decision should be made, but fail to actually make the pointed-at decision, instead delegating it to an entity only very indirectly touched by public or private opinion, or by democratic representation. The concrete and cubicles of bureaucracy are the physical signs of a powerful entity at work in the human world, but not precisely directed by any coherent, recognizable sort of human agency. If we posit the missing agency as a kind of group-psychological entity, not acting through supernatural processes but wholly through natural ones, composed of but not a simple function of human minds, we might hope to understand some of these entities, our evolutionary partners, in a characteristically human way, one that we are especially good at thinking about.
Sometimes a sacred entity is personified, as with gods or demons or the centers of cults of personality. At other times, the sacred entity is composed only of abstract ideas, refusing to personify itself. Why might it benefit such an entity to hide its nature?
The Devil traditionally has to obey his contracts. In legend, demons tend to be crafty lawyers, because the words of contracts seem to exert some kind of power over them. In the late 18th century, a peculiar kind of demon was raised. Since it called itself “government,” or “democracy,” the people who raised it contracted with it based on the assumption that it was, in fact, a vehicle for the streamlined manifestation of the will of the people. But the demon was tricky; its functions were not well-characterized by its description, and it almost immediately started to encase its contract (exemplified by the United States Constitution) in a layer of itself, giving itself say over when the contract might be violated, and what the words meant.
The entity cloaked its contract in such powers as judicial review, appointing priests based on an arcane status competition to interpret its meaning. It noticed words that could increase its power, such as the Commerce Clause, and leaped out of its bottle wherever and however possible to exert more control over the processes of people and the world. No one can predict what it will do next.
Preference Falsification as Coordination Mechanism and Anti-Coordination Mechanism
To review, the preference falsification model posits that people have two sets of preferences, one public, and one private. There is a preference falsification equilibrium on the one hand, and an escape from that equilibrium on the other. Perhaps it seems that people’s natural state ought to be one free of such preference-falsification spirals. However, when a preference-falsification equilibrium collapses, it tends to be replaced (and may even be directly displaced) by a new preference-falsification equilibrium. Before the revolution, it is unwise to speak against the King; after the revolution, it is unwise to speak against the revolutionary party. And when the revolutionary party is displaced in turn, it becomes unwise to speak against the counter-revolution. People reel from the shifting social reality, but each time they settle into a new regime, and the next revolution usually comes as a surprise to everyone, even the revolutionaries themselves.
Preference falsification prevents cooperation: if people cannot communicate what they want, they cannot coordinate to achieve it. But consider how much coordination (cooperation) is done exactly on the dimension of preference falsification. Social politeness often takes the form of preference falsification; expressing enthusiasm, approbation, or concern when it is not sincerely felt is a basic social skill. The natural state of humans may be to exist in the thrall of many preference falsification equilibria of various ages and origins. It is likely that the thing most likely to cause the unseating of a previous egregoric entity, is another egregoric entity, rather than nothing.
To the extent that these entities evolve, some reproducing themselves (using humans and tapping into their resource streams, from money to labor to sacredness) and others dying, then we may expect surviving entities to have evolved defenses to the breakdown of preference-falsification equilibria. This may include features for moderating the demands placed on component humans (friendliness, mentioned above), or features for preventing the initial defections that might cause the equilibria to collapse, or features for making such initial defections less likely to trigger a collapse. These are immune responses, and may be either psychologically internalized or socially imposed, or both. They may fit themselves toward human flourishing, or adapt human preferences to fit themselves, and attempt to keep them from demanding alternative conditions.
We might model people as having not just two sets of preferences, but two natures. (Recall that dichotomies are particularly gratifying ways for humans to organize the world.) Expanding from Kuran’s public-private dichotomy, the two natures might be described as on the one hand social, long-term in focus, sacredness-respecting belonging-maintaining, and with high overlap of “public,” and on the other hand, short-term, moment-to-moment preference not connected to others or to cultural expectations, which must ordinarily be kept more “private.” We are Homo duplex, with one self spread out in time, connected to structures and institutions that may last even longer than a human lifetime, usually corresponding to the self viewed from the imagined perspective of others, the bee-self; and another self, perceived immediately from the privileged perspective of inside, the ape-self. (This is perhaps misleading, because the “ape self” is still quite social. Desiring belonging, or the sense of belonging, is near-term comfort, not merely long-term coordination. Solidarity or companionship of this type are often available in small units, as from prostitutes or Twitter.)
The relationship implied between the egregoric entity and its component people is mathematically complex, and difficult to represent in the two dimensions I have readily available to me. People do not necessarily prefer to exert the least amount of effort, to be on the far left side of the curve. Rather, they need to belong, to feel effective, and to feel useful to each other. They have a need for something that demands a great deal of effort from them, so that they may not feel bored or burdensome, but valuable and skillful.
Demandingness can be analogized to the negotiation between the short-term desires of the Nature Two self and the long-term, narrative, social desires of the Nature One self. The more one favors the long-term and social self over the short-term egotistical self, the more demanding the entity sacrificed for can be said to be. This demandingness is precisely the axis of coordination. In a sense what it demands is preference falsification, but it is more accurate to say that it demands preference husbandry, balancing the preferences of different sub-self (or perhaps extra-self) entities.
Moment to moment, adhering to the demands of egregoric entities often looks like preference falsification: the avoidance of “temptation,” which the short-term self, disconnected from narratives of identity and sacredness, might like to engage in: running from battle instead of fighting, engaging in adulterous sex, sitting around instead of doing something hard. But on the other side of the “temptation” is an entity whose preferences are satisfied by not engaging in the tempting activity. This is the self that believes in bravery and glory in battle, or in having an interesting life, or in becoming skillful at something, or in monogamous marriage – the long-term result of many short-term sacrifices.
But the magnitude of these sacrifices is the measure of their value, and by implication, costly signals of cooperation. Preference falsification may inhibit cooperation, but this form of preference falsification (or preference husbandry, in favor of a certain subset of preferences) is precisely the axis along which coordination occurs. Up to a point, what people want is to do what is difficult (and therefore usually awarded belonging and/or status). Preference “falsification” in this sense is therefore a necessary human survival skill, when survival is predicated on belonging to a social unit. Consider my revised version of Maslow’s pyramid:
But after a certain point, determined by many factors, the dynamic breaks down. Some yokes are too heavy, especially when they are not adequately rewarded. An entity may hitch humans to heavy, unrewarding yokes and keep them there with a true preference falsification dynamic, and therein lies the danger: the very attributes that allow us to cooperate for our mutual benefit, also allow us to “cooperate” for our mutual emiseration.
Positing egregoric entities as a layer in the structure of peopling is a suspicious move that tends to meet with opposition. While we regularly speak of these entified beings in specific cases – governments, religions, institutions – it meets with objections to speak of them as having their own natures and behaviors (and perhaps even agency) separately analyzable from that of their component humans. It may make about as much sense to attribute agency to a particular government as to a hurricane. But since the actions of a government are built on the substrate of human minds, unlike hurricanes, and since agency is a characteristic projected onto human minds (even if not actually detectable, strictly speaking), then it makes at least cognitive sense to attribute agency to one and not the other, if only provisionally. Another reason to project agency onto nonhuman egregoric entities like governments is that it gives them appropriately creepy connotations, so that we may look at them as alien things, so that we may have fresh eyes and be better anthropologists from Mars. The entity known as the Communist Party, centered on Mao, may not have intended to starve tens of thousands of people to death; egregores may not (always) eat people. But it is an interesting and very human-scale project, to ask what this kind of entity does eat, how it sustains itself, how it dies.
Things that are demanding of us (with or without their own attributed agency) help us coordinate together. But beings of our nature might coordinate on a more intimate level: to choose or create the egregoric entity on the Nature One level that gives us the most benefits on the Nature Two level – keeps us comfortable, belonging, well-fed and satisfied. We may wish to choose an entity that demands little, on the left half of the curve. But there is another possible axis of coordination: we might also coordinate to choose the egregoric entity that allows us to create the coolest stuff on the Nature One level – to go to the moon, to build cathedrals, to make atomic bombs and smallpox vaccines. We may wish to choose an entity toward the peak of the productivity distribution, even if it is more demanding. Of course all this posits that humans have a high degree of control over selecting the kind of egregoric entities they wish to serve. In reality, we may have no more control over that, than over what viruses infect us (which is to say, a non-zero but small amount).
Kuran’s preference falsification model necessarily simplifies matters, allowing people preferences between only two regimes. Often there are only two available choices; but what these choices happen to look like is determined by a complex set of forces, and the choices may not bear much resemblance to the desires of most of the population. All offered choices may be poor. Different people are differently susceptible to different kind of entity-choices; the kind of entity offered as a choice (either the status quo or the revolutionary vanguard) depends on the composition of the population, cultural and technological factors favoring entities with certain features, and pure chance and coincidence along an array of other factors. It is a huge and unwarranted assumption to expect that the actually-existing government, or its revolutionary or counter-revolutionary replacement, is a simple (perhaps additive or averaging) function of the desires of component individuals. It is also generally unwarranted to assume that any of these entities is friendly. However, “not belonging to one of these entities at all” is rarely an option.
Sacredness and Freedom of Speech
Free speech seems like a value designed to prevent preference falsification spirals from occurring, and perhaps even to ensure that cherished values are protected by making sure people are regularly exposed to common counterarguments or threats to values, so that an immunity can be developed. Kuran, p. 302, says:
It has been observed that our beliefs are strongest when they have been mildly attacked, for then we have become aware of their vulnerability and learned how to counter criticisms. Prior exposure to mild objections thus produces resistance to later persuasion, which then blocks sharp changes in private knowledge and preferences. By implication, beliefs whose counterarguments were unthought are easier to change than ones whose counterarguments, while treated as unthinkable, have enjoyed at least some public exposure. When a revolution challenges many established beliefs, the ones to succumb first may thus be those that had enjoyed the greatest protection from public challenges.
[Emphasis in original; citations omitted.]
Unfortunately, despite its many potential benefits, free speech must be implemented by those mysterious entities called governments, playing endless games of self-serving “telephone” with vague inputs from constituent humans. Free speech is also implemented socially, through norms and trust. Since free speech is a coordination mechanism and decreases preference falsification, it is a threat to any entity that benefits from preference falsification. And as we have seen, both revolutionary and counter-revolutionary governments benefit from preference falsification. Speech is the stuff of social information; few entities would not like to exert a great deal of control over its exchange. With threats from all sides, it seems like it would be difficult to maintain meaningful freedom of speech as an equilibrium.
Even where nominally free in the sense of the First Amendment, speech is the most important battleground of sacredness warfare by social mechanisms, such as shunning or disapproval. (This is especially the case if “speech” is interpreted broadly to include any attempts to communicate information symbolically, as in art or ritual.) As John Stuart Mill observed, the greatest threat to free speech – the greatest danger of a preference falsification equilibrium, to use Kuran’s term – comes from social and economic forces, not from the government. Everyone needs to make a living, and everyone has an innate need for social belonging; if these are held hostage in exchange for making the correct speech performances, those performances will be made (and others refrained from).
One strategy has been to attempt to sacralize free speech itself; but this has not proved very successful. Entities that wish to restrict speech simply define violations of their sacredness as “not speech.” And there seems to be no egregoric entity protecting the value of free speech for its own sake. Freedom of speech is a fragile and rather pathetic construct, unable to defend itself against virulent sacrednesses.
Escape from Preference Falsification?
Sacredness warfare may be conducted with words or guns, but relies in part on a sensory capacity for perceiving social information (and the presence, desires, promises, and demands of egregoric entities). Some people are entirely lacking in this sensory capacity – for various neurological or other reasons, they are blind to sacredness. For most people, the capacity varies, in accordance with the variance of individual susceptibility to sacredness in general, and the particular nature of the sacredness the individual is asked to accept, and the fit between them.
Only a small subset of sacredness-blind people may be needed to defeat a preference falsification equilibrium, provided that the private preferences of others are properly distributed such that their offer of coordination propagates. Under precisely the right circumstances, with the right lined-up dominoes of public and private preferences, “[o]ne truth-teller can break a preference-falsification equilibrium.” In other circumstances, even great outpourings of dissent are not enough to upset the equilibrium. And even if it is toppled, there is no guarantee that the new equilibrium will be any better; it may be much worse.
In sum, the bad news is that we don’t have much control; the good news is that we wouldn’t know how to use it if we did.