The objective-function critique is not new — Harris, Raskin, and the Center for Humane Technology have made it since 2018. The structural lens adds what the existing critique underspecifies: control systems engineering’s term for content moderation is DISTURBANCE REJECTION, and any system optimizing toward a fixed target will route around output filters as a mathematical property of optimization, not a failure of the filters. The collision partners are control systems engineers (whose precise vocabulary for why moderation will always fail against an unchanged objective function has not entered the policy conversation) and mechanism designers who can specify alternative objective functions with the same engineering precision the engagement target was specified with.
Circle (Tier 2): The algorithm observes behavior, serves content optimized for observed behavior, the content shapes the behavior, the shaped behavior becomes the new input. The system manufactures the preferences it claims to reflect.
Chain: User scrolls -> algorithm detects what captures attention -> compressed content captures most -> algorithm surfaces more compressed content -> user’s thinking adapts -> user produces compressed content -> algorithm surfaces it to others -> environment narrows -> cycle tightens
A thirteen-year-old opens TikTok. She watches a dance video. She watches another. She pauses — just slightly, a fraction of a second — on a video about body image. She scrolls past it. She didn’t engage. She didn’t like it. She didn’t share it. She paused for a fraction of a second.
The algorithm noticed.
Tomorrow, her feed will contain more body image content. Not because anyone decided it should. Because the fraction-of-a-second pause was data. The pause was engagement. The engagement was the signal. And the system that processes the signal has one objective: maximize the next pause.
The thirteen-year-old did not choose this. The algorithm did not choose this. Nobody chose this. The objective function chose it.
The objective-function critique is not new. Tristan Harris, Aza Raskin, and the Center for Humane Technology have argued since 2018 that the problem is not the content but the business model — that the algorithm optimizes for engagement and engagement selects for outrage, division, and compression. This critique has entered mainstream discourse. Congressional hearings have featured it. Documentaries have dramatized it.
The critique is correct. It has not produced change — because naming the problem (“the objective function is wrong”) is not the same as specifying WHY conventional fixes fail structurally or WHAT the alternative design looks like with engineering precision.
Control systems engineering has a specific term for what content moderation does: DISTURBANCE REJECTION. When you filter an output without changing the optimization target, the system treats your filter as a disturbance and compensates — generating new output that achieves the same target while evading the filter. This is not a tendency. It is a mathematical property of any system optimizing toward a target. Content moderation is disturbance rejection. The system will ALWAYS route around it. The routing is a structural property of optimization, not a failure of the moderation effort.
This is what the existing critique underspecifies. Harris says “the business model is the problem.” Control systems engineering says: any system with a fixed target will defeat any output filter, necessarily, as a mathematical consequence of the optimization. The fix is not better filters. The fix is CHANGING THE TARGET — and the target change must be specified with the same engineering precision that the current target was specified with.
The door’s most important contribution is the CIRCLE it contains: the algorithm observes behavior → serves content that captures attention → the content shapes the behavior → the shaped behavior becomes the new training data → the algorithm optimizes for the shaped behavior → serves content that shapes the behavior further. The system is not reflecting human preferences. The system is MANUFACTURING the preferences it then reflects. The output is the training data for the next cycle of output. Each cycle, the manufactured preferences become more extreme because each cycle selects for the content that produced the most engagement, which reshapes the preferences toward more engagement-producing content.
The system is not a mirror. The system is a lathe, turning the user into the shape that produces the metric.
The alternative objective function is technically feasible: “diversity of content exposure” (measured by the number of distinct topic clusters the user encounters per session), “sustained attention” (reading, watching to completion, thoughtful response rather than scroll-react-share), and “exposure breadth” (widening the feed rather than narrowing it). Each is measurable with existing technology. The barrier is not technical. The barrier is that dimensional expansion does not sell ads as effectively as outrage.
| Factor | Score | Justification |
|---|---|---|
| F1: Mortality & Irreversibility | 5 | Indirect mortality through mental health effects, radicalization, erosion of shared reality |
| F2: Scale | 10 | Billions of users; every social media platform; every feed |
| F3: Compression Depth | 7 | The algorithm compresses the user’s information environment to a narrow band that the user often can’t perceive |
| F4: Time Sensitivity | 9 | The objective function is shaping billions of minds right now; each year of the current function is another year of dimensional compression at civilizational scale |
| F5: Voice Deficit | 5 | Users can complain but have no mechanism for changing the objective function |
| F6: Proximity Gap | 7 | Control systems engineers, mechanism designers, and optimization theorists are not at the content moderation table |
| F7: Temporal Displacement | 5 | Effects are visible in real time (polarization, mental health) but the CAUSE (objective function) is invisible |
| F8: Normalization | 8 | “The algorithm shows you what you want” is normalized as a description, obscuring the compression |
| F9: Hallway Dependency | 8 | The solution requires control systems engineering + behavioral economics + platform design + policy |
| F10: Knowledge Readiness | 8 | The alternative metrics exist; the measurement is feasible; the design principles are known |
| F11: Entry Cost | 5 | Platform-level change requires either regulatory mandate or competitive pressure |
| F12: Cascade Potential | 9 | Changing the objective function of major platforms would affect billions of users simultaneously |
Hiddenness Score: 52.5 Actionability Score: 53
Control systems engineers have the foundational expertise. Their entire field is the science of optimizing systems toward targets. The specific transferable knowledge: in control systems, changing the output without changing the target is called “disturbance rejection” — the system treats your intervention as a disturbance and compensates. Content moderation is disturbance rejection: the system treats the removed content as a disturbance and generates replacement content that achieves the same target. The ONLY way to change the system’s long-run behavior is to change the TARGET. This is Control Systems 101. It is not part of the content moderation conversation.
Mechanism designers in behavioral economics study how to design systems where the incentive structure produces desired outcomes without requiring per-decision oversight. The specific transferable knowledge: instead of moderating each piece of content (per-decision oversight), design the system’s incentive structure so that the content the system generates is the content you want it to generate. This is the difference between policing a market and DESIGNING a market. Platform regulation currently focuses on policing. Mechanism design would focus on market design — changing what the platform is rewarded for producing.
Nutritional labeling designers have solved a parallel problem in a different domain: how do you make the contents of a product visible to the consumer so the consumer can make informed choices? Nutritional labels didn’t ban unhealthy food — they made the content visible, shifting the decision to the consumer. An algorithmic “nutrition label” — showing the user what the algorithm is optimizing for, how the feed was constructed, what signals triggered each recommendation — would give users visibility into the compression they’re currently experiencing invisibly.
If you are a platform designer or engineer: run an experiment. Take a subset of users. Change the objective function from “time on platform” to “diversity of content exposure” (measured by the number of distinct topic clusters the user encounters per session). Measure what happens to user satisfaction, return rate, and mental health indicators. The experiment is feasible within existing A/B testing infrastructure. The results would be the first empirical evidence for whether a different objective function is commercially viable.
If you are a regulator: stop thinking about content. Start thinking about objective functions. The question is not “what content should be allowed?” The question is “what should the system be optimizing FOR?” Require platforms to disclose their objective function. Require platforms to offer users the CHOICE of alternative objective functions. And fund independent research on the downstream effects of different objective functions — so that the regulatory conversation is informed by data, not by ideology.
If you are a user: you cannot change the objective function. But you can become aware that it exists. Every piece of content in your feed is there because a mathematical function determined that it would maximize your engagement. The feed is not “what you want to see.” The feed is “what the function predicts will keep you scrolling.” The awareness doesn’t change the function. But it changes your relationship to the output.
Tier 2/3 — The system manufactures the preferences it then reflects.
Algorithm serves compressed content → content shapes user behavior → shaped behavior becomes training data → algorithm optimizes for shaped behavior → serves more compressed content
The algorithm does not begin by distorting preferences. It begins by observing them. A user scrolls, pauses, clicks, shares, and the system records every micro-behavior as a signal about what the user wants. The observation is accurate — the user did pause, did click, did share. The system then serves more of what captured attention. This too is reasonable: give people what they respond to. Each individual step is defensible. The circle emerges from the aggregate.
The content that captures the most attention is the content that is most compressed — the most emotionally concentrated, the most polarized, the most outrage-inducing. Nuance does not compete. Complexity does not win the pause. The algorithm surfaces compressed content because compressed content produces the engagement signal, and the user’s behavior adapts. Not through coercion but through environment: when the information landscape is saturated with compressed content, the user’s attention recalibrates. They begin to produce compressed content themselves — shorter takes, hotter opinions, faster reactions. This produced content becomes the next cycle’s training data. The algorithm sees a population that engages with compression and produces compression, and optimizes further in that direction.
The critical invisibility is this: the user experiences their feed as reflecting their preferences. “This is what I like. This is what I care about. This is who I am.” But the preferences were shaped by the previous cycle of content that was shaped by the previous cycle of preferences that were shaped by the cycle before that. The system is not holding up a mirror. The system is a lathe, and the user is the material being turned. The preferences are real — the user genuinely holds them. They are also manufactured — the user holds them because the objective function cultivated them. From inside, the manufactured preferences feel indistinguishable from authentic ones. That is what makes the circle invisible.
What breaks it is changing the objective function from engagement to dimensional expansion — optimizing for diversity of exposure, for sustained attention rather than impulsive reaction, for breadth rather than compression. The technology to measure these alternatives already exists. The barrier is not technical. The barrier is that an expanded user produces fewer ad clicks than a compressed one. The circle sustains itself because the compressed user is more profitable than the expanded one, and the system has no incentive to stop compressing.