System Error: Where Big Tech Went Wrong and How We Can Reboot
Jeremy M. Weinstein, Mehran Sahami, and Rob Reich
HarperCollins, $27.99 (cloth)
Solving Public Problems: A Practical Guide to Fix Our Government and Change Our World
Beth Simone Noveck
Yale University Press, $30 (cloth)
Two new books about technology and the fate of democracy begin by describing the storming of the U.S. Capitol on January 6, 2021. They are right to see that fateful day as a turning point and a benchmark for debates about the course of U.S. society, and hint at important questions: Can democracy survive in its current form? What role did information technology play in encouraging a violent mob to tear through Congress? And what do we do now?
Both books see January 6 as the product of systematic misinformation and fraud, and argue that the solution is more participatory democracy. Though we agree for the most part, neither book offers a comprehensive theory of change or a particularly persuasive vision of an aspirational digital democracy. Solving Public Problems, by Beth Simone Noveck—director of Northeastern University’s Governance Lab and New Jersey’s inaugural Chief Innovation Officer—suggests that better government will produce a better public. System Error—by political theorist Rob Reich, computer scientist Mehran Sahami, and political scientist Jeremy M. Weinstein —explains how government might come to understand and perhaps constrain big tech more effectively. Each makes important contributions. System Error breaks new ground in explaining why Silicon Valley (SV) is wreaking havoc on U.S. politics and offers uniformly thoughtful reforms. Solving Public Problems, on the other hand, offers possibly the most detailed and serious treatment of how digital tools help enhance democratic governance around the world. Neither, however, answers the question implicitly posed by opening their books with a description of U.S. democracy’s failure: What happens now, after January 6?
System Error’s greatest contribution to public debate is to identify more precisely how SV went wrong. Books such as Shoshana Zuboff’s The Age of Surveillance Capitalism depict SV as a vast devouring Moloch, perfecting the means to manipulate human behavior. Others, such as Roger McNamee’s Zucked, focus on the business side. These books help correct an imbalance in public debate, which just a few years ago treated business leaders like Mark Zuckerberg as heroes, and took Facebook seriously when it claimed it was spreading freedom and building a new cosmopolitan world where borders didn’t matter and everyone was connected. But these books don’t get at the core problem, which is a product of the powerful mathematical techniques that drive SV’s business model.
System Error explains that SV’s ability to turn complicated situations into optimization problems accounts for both its successes and its most appalling failures. Optimization lies behind the ubiquitous use of machine learning and automated feedback, the relentless “solutionism” described by Evgeny Morozov, and SV CEOs’ obsession with metrics. It is a mathematical technique that allows engineers to formalize complex problems and make them tractable, abstracting away most of the messiness of the real world. F. A. Hayek wrote of the “religion of the engineers”—their modern heirs are animated by the faith that seemingly impossible problems can be solved through math, blazing a path to a brighter world.
The first step in optimization is identifying the quantity that will be either maximized or minimized. This allows for the creation of an “objective function,” which ranks the possible solutions from best to worst. The second step involves using available data to provisionally identify the resources that can be employed to reach that goal and the hard limiting constraints. The third step is identifying and implementing the best possible solution given those resources and constraints. The fourth step is to keep updating how those resources and constraints are understood as more information is gathered.
Optimization underlies what used to be exuberant and refreshing about SV, and very often still is. Engineers are impatient with intellectual analyses that aim to understand problems and debates rather than solve them. When engineers unleashed their energies on big social problems, such as bringing down the cost of rocket launches or making video conferencing at scale rapidly possible during a pandemic, it turned out that many things could and did get done.
Indeed, many of the great achievements of the modern age are the product of this kind of ingenuity. Finding relevant information for research used to require card catalogs, clippings files and vast amounts of human effort. Google search—a means for combing through a vast, distributed repository of the world’s information and getting useful results within a fraction of a second—would have seemed like a ludicrous impossibility only three decades ago. Google’s founders used a set of mathematical techniques to leverage the Internet’s latent information structures, ranking online resources in terms of their likely usefulness to searchers and unleashing a knowledge revolution.
The problem is that optimization cannot reconcile people’s conflicting world views. Though conflict has always been the meat of politics, political differences today mean that people not only disagree over solutions and precise settings of valuation parameters; they also clash over the fundamental terms in which problems are conceptualized. Is racism a characteristic of individual preferences or an arrangement of social forces? Is fairness a property of a whole society or of a particular algorithm? How well is human flourishing captured by economic output? What do “power” and its “decentralization” mean? For optimization theory, these are at best ill-posed problems.
SV bet that these political problems would evaporate under a benevolent technocracy. Reasonable people, once they got away from the artificial disagreement imposed by older and cruder ways of thinking, would surely cooperate and agree on the right solutions. Advances in measurement and computational capacity would finally build a Tower of Babel that reached the heavens.
Facebook’s corporate religion held that cooperation would blossom as its social network drew the world together. Meanwhile, Google’s founder Sergey Brin argued that the politicians who won national elections should “withdraw from [their] respective parties and govern as independents in name and in spirit.” System Error recounts how Reich was invited to a private dinner of SV leaders who wanted to figure out how to build the ideal society to maximize scientific and technological progress. When Reich asked whether this society would be democratic, he was scornfully told that democracy holds back progress. The participants struggled with how to attract people to move to or vote for such a society. Still, they assumed that as SV reshaped the world, democratic politics—with its messiness, factionalism, and hostility to innovation—would give way to cleaner, more functional systems that deliver what people really want. Of course, this did not work.
Reich and his co-authors (who all teach at Stanford and are refreshingly blunt about the University’s role in creating this mindset) explain how their undergraduates idolize entrepreneurs who move fast and break things. In contrast, as then-Stanford president John Hennessy once told Joshua Cohen, it would be ridiculous for Stanford students to want to go into government. What could they possibly change? SV was in the business of changing things, and its entrepreneurs saw themselves as more than just business leaders. They were a self-appointed Platonic aristocracy of guardians who understood what needed to be done and had the means to push it through.
The optimal arrangements for progress that they proposed bore an uncanny resemblance to the arrangements that would maximize SV’s profits. As System Error explains, optimization theory worked well in harness with its close cousin, the “Objectives and Key Results” (OKR) management philosophy, pioneered by Andy Grove at Intel, to align engineering insight with profit-making intent. For a little while, the mythology of optimization allowed entrepreneurs to convince themselves that they were doing good by virtue of doing well. When Facebook connected people, it believed it made everyone better off—including the advertisers who paid Facebook to access its users. Keeping users happy through algorithms that maximized “engagement” also kept their eyes focused on the ads that paid for the endless streams of user posts, tweets, and videos.
But politics kept creeping back in—and in increasingly unpleasant ways. It became clear that Facebook and other SV platforms were fostering profound division: enabling the persecution of the Rohingya minority in Myanmar, allowing India’s BJP party to foster ethnic hatred, and magnifying the influence of the U.S. far right. As the writer Anna Wiener explains, even seemingly innocuous services such as Github, which was supposed to help programmers cooperate on open-source coding projects, provided a space where the far right could organize.
As the chorus of objections grew, Facebook drowned it out by singing the corporate hymn ever more fervently. The company’s current Chief Technology Officer argued in a 2016 internal memo that Facebook’s power “to connect people” was a global mission of transformation, which justified the questionable privacy practices and occasional lives lost from bullying into suicide or terrorist attacks organized on the platform. Connecting people via Facebook was “de facto good”; it unified a world divided by borders and languages.
However, SV’s machineries of optimization—while not the sole causes of polarization—spread and deepened divisions. Social media algorithms used machine learning to maximize “engagement,” drawing consumers to content that would keep them on the service and viewing ads. The way that these algorithms personalized content with limited attention to diversity, social cohesion, or relationships across difference likely deepened division and failed to promote greater comity. Indeed, research suggests that this drew many users to shocking or surprising content that aided radicalization. Other work argues that social media just made it easier for angry people to find and share falsehoods that they already wanted to find.
While there is still much debate over the details, there is a growing consensus that these tools led citizens down rabbit-holes and into a land of dark wonder where facts became lies and logic was turned upside down. Instead of providing a rational alternative to divisive politics, SV’s products deepened social and political divisions, helping transform the United States into a balkanized country where outraged mobs could storm the Capitol and try to find and hang the vice president.
Reich and his coauthors, however, have more to say about the problem than they do about possible solutions. They argue correctly that we need government involvement in a system-wide solution, but their prescriptions mostly focus on better technology policy. System Error highlights the need for a new relationship between government and technology, in which governments revive institutions (such as Congress’s defunct Office for Technology Assessment) that help inform complex technical issues, and technologists incorporate ethics into how they build things. Its authors want an informed citizenry to boot out politicians who fail to protect their interest. But this is sketched out in light pencil.
Beth Noveck’s Solving Public Problems does the opposite; it centers solutions. Building on her earlier books (Wiki Government; Smarter Citizens, Smarter State), Noveck celebrates the potential of data and technology to solve problems by engaging citizens. This is her most comprehensive book yet, bursting with sage, practical advice for public sector officials and civil society actors who want to engage citizens and give them more power.
Noveck generally thinks that the problems of modern democracies are straightforward. People don’t trust government because it does not solve festering problems in people’s lives. Citizens blame government for their dire straits. She argues that these problems are fundamentally solvable, as citizens have much of the knowledge needed to tackle them. Government, then, must involve citizens so it can gain better outcomes. Noveck urges that public institutions should engage citizens through wide collaboration, taking advantage of new data and analytics to resolve the issues that citizens care about.
She describes many examples of how this can work. In San Pedro, Mexico, a city councilor launched an initiative to lower the amount of time that people spent driving their kids to school. In the U.S. Patent Office, Noveck herself helped develop a system of expert volunteers to help determine whether inventions should be patented. In Taiwan hundreds of thousands of citizens use online tools to figure out how best to understand and define political problems that can be addressed through legislation (we’ll return to this example a bit later). The Economic Development Authority in New Jersey (where Noveck works for the state government) sent out questionnaires to small- and medium-sized businesses during the COVID-19 pandemic to determine what support people needed.
Noveck cites philosopher John Dewey, whose understanding of the democratic challenge is close to her own. For Dewey politics begins in the problems that people experience in their lives. Many of these are the result of hidden interdependencies with other people. Uncovering these interdependencies is the first step toward problem solving—and citizens ought to be involved alongside experts. As Dewey puts it, even if the expert shoemaker knows how to make shoes, the customer knows where the shoe pinches. Governments miss out on valuable information when they ignore the perspectives of citizens or treat them as a nuisance.
Noveck believes that technology can play a crucial role in gathering this information and acting on it. Like Reich and his coauthors, she is skeptical of SV solutionism and argues that problem solving is a collective civic enterprise that works best when it can call on diverse perspectives and information to figure out what to do. Businesses have a hard time solving major social problems on their own, after all, “profit maximization and the financial interests of the firm will always come first.”
So, what can technology do? Online platforms help governments draw on a wider and more diverse range of citizen perspectives. For example, Noveck points to an online platform in Reykjavik that allows residents to suggest how money should be spent. More than half the city’s population has proposed solutions or voted on them. With help from the government, “design thinking” can be remade to create interfaces that are easy for individuals and communities to engage with. Indeed, data and algorithms can be used for the public good, so long as we understand their potential biases.
Noveck believes that technology can make government more participatory and effective. As she describes it, public institutions that have “responded to difficult problems by consulting the citizens directly affected by them and through the use of rapidly increasing quantities of data and predictive analytics . . . showed how successful government could be at improving people’s lives if such novel ways of working were the norm.” Moreover, rebuilding government could revive U.S. democracy, potentially solving “the crisis of trust in public institutions,” and building a “stronger but better government” in which managers work with the public to solve problems and build legitimacy.
This is an attractive goal, but the book doesn’t quite show how to get there. In fact, one must ask, if this all works so well, why haven’t we done it yet? The last decade has seen governments adopting ideas institutes, nudge labs, and other forms of experimental governance, in part thanks to the tireless work of Noveck and her collaborators. It is at best uncertain that they had any significant consequences for the crisis of trust that Noveck is rightly worried about. That isn’t Noveck’s fault: still, the book does not explain how this apparent mismatch between the scale of the solutions and the scale of the problem has affected her thinking about how to bring about change. Will trying harder next time really produce different results? Can we afford the experiment?
Like the optimizers, Noveck sees politics as a set of collective problems that we have a shared interest in solving. Her faith in using citizens’ diverse knowledge and perspectives to solve these problems, rather than employing optimized machine learning, is promising, but she understates its difficulty. Politics is often more struggle than collaboration. People don’t typically fight one another because they are disappointed in government; they fight because they fear what will happen if the other side gets a grip on the levers of power.
In the United States, politics is not only about filling potholes; the problems that divide citizens can’t be solved through better information and implementation. We must ask: how do we maintain social cohesion and solidarity while including people previously marginalized by everything from migration laws to restrictive zoning? Noveck highlights solutions that government officials have created to tackle COVID-19, but she doesn’t address citizens’ disputes about how the government should respond to the virus, or the severity of the virus’s threat.
On their own, participatory schemes to improve small aspects of peoples’ daily lives are not going to change their minds. Noveck’s vision of a world in which our disagreements center on the managerial challenges of a complex society requires us to first engage fundamental disagreements that shape U.S. politics, not to ignore or sideline them.
Something similar is true of the large-scale institutional fixes that Reich and his coauthors believe are necessary. Under current conditions these solutions are going to be practically impossible to implement, which may be why System Error focuses on improving the processes of government and the knowledge available to democratically elected legislators rather than the deeper problems of U.S. democracy that it identifies at the outset.
These two books elide the difficulty of the current political situation in the United States for a good reason: it is difficult to know where to begin. We ourselves are far from a complete answer, but the starting point might be the invasion of a different country’s capitol.
On March 18, 2014, hundreds of student protestors under the banner of the “Sunflower Movement” stormed Taiwan’s national legislature. They opposed the governing party’s trade deal, which would have tied the island’s tech stack closer to that of the mainland. The protestors weren’t violent, but they occupied the legislature’s main chamber for twenty-four days. On March 30, half a million people rallied in support of the protestors. According to one poll, a plurality—and nearly a majority—of Taiwan’s population supported the invasion, while nearly 70 percent agreed with the protestors’ broad demands. The protestors forced the government to back down and won substantial concessions, including better oversight of future trade deals.
Both System Error and Solving Public Problems praise programmer-turned politician Audrey Tang, who played a key role in the protests. She helped design the platform that protestors used to communicate with each other, build consensus around their demands, and broadcast online videos of the occupation of Parliament. After this process’s success, the government appointed Tang and her compatriots as “reverse mentors” to the government’s ministers. When an independence-oriented party swept the government out of power on the back of the protests, Tang became minister without portfolio. Later, she became digital minister, tasked with bringing an agenda like Noveck’s to life on a national scale.
And, to a large extent, Tang did precisely what the two books would have wanted. Taiwan’s government is now arguably the most sophisticated government when it comes to technology. It has built a system of “digital competency” education that is emulated around the world. Even the school districts around Microsoft’s headquarters have sent delegations to learn from them. It has also created online platforms for civil involvement in policymaking, based on Pol.Is, designed to minimize trolling and make it easier for the public to reach consensus on divisive issues. While the platforms have only been used for a small range of issues, and their results are not binding on legislators, they are widely accessible. Nearly half of Taiwan’s citizens have signed up as active users. Moreover, they have addressed controversial topics—such as labor rights in the gig economy and gay marriage—and seen them through with often ingenious solutions ratified by the legislature.
Taiwan has also built a rapid response system to disinformation attacks, which occur more frequently there than they do in any other country in the world (according to some observers, because of its proximity and importance to China). Some experts believe that this has helped dampen the deep polarization along ethno-political lines related to time of migration to the island and feelings toward the mainland. Taiwan also managed arguably the best COVID-19 response on the planet, balancing the strongest economic growth in Asia in 2020 with the world’s lowest per capita death rate among countries with reliable data.
These efforts to build and protect democratic consensus and public goods stemmed directly from democratic confrontation. Fearful that techno-authoritarianism would creep into their society, the Sunflower movement protestors peacefully and decisively changed the debate over Taiwanese democracy by highlighting its incompatibility with a non-democratic neighbor and driving a wedge into the ruling Kuomintang party. The g0v movement that Tang founded had been a bit player in the country until that larger mobilization.
There is much to learn from this experience. Trying to detach technology policy from political conflict on the ground is probably a losing strategy. After all, citizens are devoting a great deal of time and energy to engaging with civic life, even if much of it comes in the form of shit-posting. The question is not how to dissipate that energy, but how to harness it.
Here, Taiwan also offers a clue. Two subjects of concern that Americans share with the Sunflower protesters are the threat of the Chinese Communist Party (CCP) and the concentration of power in those controlling technology. These both rank consistently in polls of the top ten threats perceived by Americans.
Could a great struggle for digital democracy against the Chinese surveillance state and Silicon Valley surveillance capitalism really form the foundation for a social movement supporting national renewal? There are certainly many reasons to be skeptical. Such an agenda could easily degenerate into anti-Asian racism and militarism. And yet, Taiwan is not the only example of a digital democracy that bootstrapped itself from the shadow of an authoritarian threat. Estonia, facing a similarly precarious situation, has built arguably the second most impressive example of digital democracy.
A remarkable property of both examples is their deep appeal across the standard U.S. political spectrum. Tang is perhaps the most prominent transgender political leader in the world and a self-proclaimed anarchist; she also has great potential with the U.S. right as one of the most popular leaders on the front lines in the defense of liberal democracy against the CCP. Estonia is a trailblazer in participatory democracy and one of the only post-Soviet countries that has developed a strong welfare state. At the same time, it is a poster child for libertarians.
The coupling of a struggle for democratic ideals and the adjustment of economies and technologies to match is not new in U.S. history. The Free Soil movement that helped birth the Republican Party and the abolition of slavery was as much a movement to protest competition from slave labor and the rise of industrialization as it was a moral crusade. Roosevelt tied confronting the Great Depression and dictatorship together to stimulate public infrastructure and scientific investment. Sputnik provoked a generational investment in technology that birthed the internet. And we’ve seen explicit echoes of these moments recently in, for example, the Endless Frontier Act, which draws on the legacy of the Cold War science surge to revitalize public investment in response to Chinese competition.
Matching these past achievements will require mastery of the kind of politics that we often forget went into making them. To realize the bold visions Reich and his coauthors urge as necessary and to which Noveck aspires, we will have to speak to the hearts—not just the heads—of those we often disagree with. This will require empathy, inspiration, vision, compromise, and a willingness to acknowledge those whose beliefs we abhor. In short, it will require politics.
Politics is about coalition building. And that, in turn, is about identifying who you do not want to win and figuring out how to stop them. The great struggles of the twenty-first century will pit democracy against authoritarianism, freedom against surveillance, public control against one party rule, and popular interests against elite domination. Our chances of avoiding another event like January 6 depend on whether democratically-oriented politicians and thinkers succeed in connecting these grand battles to the messy and sordid business of everyday politics in ways that scramble existing coalitions and forge new ones.
