Content Moderation

Abstract

Every service that lets users post must decide what stays up. That decision, made billions of times a day, is content moderation. It began with volunteer system operators on bulletin boards, was shaped by two US court cases of the early 1990s that produced Section 230, and grew in the 2010s into an industry of tens of thousands of contract workers in the Philippines, India, Kenya and the United States, backed by automated systems such as Microsoft’s PhotoDNA. Its failures became world events: Facebook’s role in the violence against the Rohingya in Myanmar, the livestreamed Christchurch mosque attack in 2019, the suspension of a sitting US president in 2021. Platforms responded with formal rules, transparency reports and Meta’s Oversight Board; the European Union made many of these practices law in the Digital Services Act and issued its first fine under it, against X, in December 2025. In January 2025 Meta ended third-party fact-checking in the United States. Moderation offers no solution that satisfies everyone, only choices about which errors to accept.

The Inescapable Problem

Any service hosting user content faces material that is illegal, dangerous or simply unbearable: spam, child sexual abuse material (CSAM), terrorist recruitment, graphic violence, harassment and fraud. The scholar Tarleton Gillespie argued in Custodians of the Internet (2018) that moderation is not a side feature of platforms but the commodity they actually offer: a tolerable environment carved out of the raw stream of open posting. A service that removes nothing is quickly dominated by its worst users.

The defining constraint is scale. One person can think carefully about one borderline post. A large platform receives billions of items a day in hundreds of languages, and decisions must be made in seconds, by rules that thousands of reviewers and many software classifiers can apply consistently.

From Sysops to Section 230

Early online communities (bulletin board systems, Usenet, later web forums) were moderated by volunteers: the system operator of a BBS, the moderator of a Usenet group, the administrators of a forum, working at a scale where they might know the participants.

The legal framework came from two lawsuits against commercial online services. In Cubby v. CompuServe (1991) a court held that CompuServe, which did not review the content of its forums, was like a bookstore and not liable for defamatory posts it did not know about. In Stratton Oakmont v. Prodigy (1995) a New York court held that Prodigy, which advertised itself as a family-friendly service and moderated its boards, was a publisher and therefore liable. The combination punished services for trying to moderate. Congress responded in 1996 with Section 230 of the Communications Decency Act, which protects online services from liability for what their users post and, in a separate clause, for good-faith decisions to remove content (see Internet Law and Section 230). It made both hosting and moderating legally safe in the United States, and every large American platform was built on it.

Moderation Factories

The platforms of the 2000s and 2010s outgrew volunteer moderation. Facebook, YouTube, Twitter and others wrote detailed internal rulebooks and hired reviewers to apply them, mostly through outsourcing firms. When users reported a growing number of violent videos on Facebook Live, Mark Zuckerberg announced in May 2017 that Facebook would add 3,000 reviewers to the 4,500 it already had; the number kept growing in the following years, most of the reviewers employed by contractors rather than by Facebook.

The work was psychologically damaging. Reviewers spent their shifts viewing beheadings, child abuse, suicides and hate, often with quotas measured in seconds per decision. In February 2019 Casey Newton’s report “The Trauma Floor” in The Verge described a Cognizant site in Phoenix where moderators for Facebook earned $28,800 a year, against an average total compensation of $240,000 for Facebook’s own employees, and where some developed symptoms of post-traumatic stress. In May 2020 Facebook agreed to pay $52 million to settle a class action by US moderators who said the work had harmed their mental health. In 2022 Daniel Motaung, a former moderator for Facebook at the outsourcing firm Sama in Nairobi, sued Meta and Sama in Kenya over working conditions and pay, the start of several cases there. Sarah T. Roberts documented this hidden workforce in her book Behind the Screen (2019), and the documentary The Cleaners (2018) followed moderators in Manila.

The Automation Layer

Human review cannot cover billions of posts, so platforms automated where they could. The most successful tool was PhotoDNA, developed by Microsoft with Hany Farid of Dartmouth College in 2009. It computes a robust fingerprint of an image that survives resizing and small edits, so that known CSAM can be recognized and blocked automatically across services without anyone having to look at it again. The same idea of sharing fingerprints underpins the Global Internet Forum to Counter Terrorism, set up by Facebook, Microsoft, Twitter and YouTube in 2017 to share hashes of terrorist content.

Machine-learning classifiers now screen most content first, removing obvious violations and sending uncertain cases to people. They work well for spam and nudity and badly where meaning depends on context: sarcasm, reclaimed slurs, news reporting about violence, languages with little training data. In 2016 Facebook removed posts of Nick Ut’s Pulitzer Prize-winning 1972 photograph of a girl fleeing a napalm attack in Vietnam because it showed a naked child, and restored it after protests from Norwegian newspapers and the Norwegian prime minister.

When Moderation Failed

Several failures showed what was at stake outside the platforms themselves.

In Myanmar, Facebook became the main way most people used the internet in the mid-2010s, and it carried a sustained campaign of hate speech against the Rohingya Muslim minority before and during the military’s violence of 2017, which drove hundreds of thousands of people from the country. In March 2018 the chair of the UN fact-finding mission on Myanmar, Marzuki Darusman, said social media had played a “determining role” and that in Myanmar “Facebook is social media.” Facebook had few moderators who read Burmese, and it later acknowledged that it had not done enough.

On 15 March 2019 a gunman attacking two mosques in Christchurch, New Zealand, broadcast the attack live on Facebook. Copies spread faster than they could be removed; Facebook said it removed 1.5 million copies of the video in the first 24 hours, most of them before they were posted. Two months later New Zealand’s prime minister Jacinda Ardern and France’s president Emmanuel Macron launched the Christchurch Call, a set of commitments by governments and technology companies to remove terrorist and violent extremist content online.

After the attack on the US Capitol on 6 January 2021, Facebook, Twitter and YouTube suspended the accounts of President Donald Trump while he was still in office.

The Constitutional Turn

As moderation decisions became political events, platforms began to formalize them. Internal rulebooks became published Community Standards, and quarterly transparency reports gave numbers of removals by category. Civil-society groups set out the Santa Clara Principles in 2018, calling for notice to users whose content was removed and a right of appeal.

Meta created an Oversight Board, funded through an independent trust and empowered to overrule the company on individual cases. It issued its first five decisions on 28 January 2021, overturning Meta in four of them. Meta referred the Trump suspension to it, and on 5 May 2021 the board upheld the suspension but criticized its open-ended nature as an “indeterminate and standardless penalty,” requiring Meta to decide on a defined response; Meta set a two-year suspension.

The European Union turned many of these practices into law. The Digital Services Act, which applied to the largest platforms from 2023 and to all services from February 2024, requires notice-and-action procedures, reasons for removals, internal appeals and out-of-court dispute settlement, transparency reports, and risk assessments and independent audits for “very large online platforms.” On 5 December 2025 the Commission issued its first fine under the act: €120 million against X, for the deceptive design of its paid blue checkmarks, an inadequate advertising repository and restrictions on access for researchers.

The Retreat

In the 2020s some platforms moved in the opposite direction. After Elon Musk bought Twitter in October 2022, the company cut most of its trust and safety staff, reinstated suspended accounts and expanded Community Notes, a system in which users write context notes that appear only when contributors who usually disagree rate them helpful.

On 7 January 2025 Mark Zuckerberg announced, in a video titled “More speech and fewer mistakes”, that Meta would end its eight-year program of third-party fact-checking in the United States and replace it with community notes, loosen its rules on topics such as immigration and gender, and concentrate automated enforcement on illegal and high-severity content. Meta argued that its systems had been removing too much legitimate speech; critics argued that the change followed the political wind of a new administration. The opposite trends in Brussels and in Silicon Valley meant that the same post could be treated differently depending on where its reader lived.

Dead End

Two opposite ideas about moderation have repeatedly failed.

The first was the neutral conduit: the belief, built into the early structure of Section 230 and the EU’s 2000 e-Commerce Directive, that a platform could carry content without taking responsibility for it. Every large platform ranks, recommends and removes content, and each of those is an editorial choice. As platforms came to decide what each user sees through recommendation algorithms optimized for engagement, the claim of neutrality became impossible to sustain, legally in Europe and politically everywhere.

The second was perfect moderation at scale: the belief that enough reviewers or sufficiently good AI could reliably remove the bad and keep the good. Meaning depends on context that machines often lack; the same words are abuse from one speaker and solidarity from another; what counts as harmful is contested; any rule precise enough to automate can be gamed, and any rule flexible enough to be fair is hard to apply consistently to billions of items. Each platform that promised to solve moderation has found instead that it can only choose which errors to make, removing legitimate speech or leaving harmful content up, and how many of each to accept.