Every major privacy framework built in the last two decades shares a foundational assumption: that people, when given adequate notice and a meaningful choice, can protect themselves. Read the disclosure. Click the button. Manage your preferences. The model is tidy, legally defensible, and built around a vision of the individual as an autonomous agent capable of weighing risks and making informed decisions about their own data.
That model breaks down badly when the individual is eight years old and the system doing the influencing is not serving them content so much as building the architecture of who they are going to become.
This is the argument that privacy governance professionals, developmental psychologists, and a growing number of legal scholars have begun to advance with urgency: that consent-based AI governance, however rigorously designed, arrives fundamentally too late to protect children. Not because the consent is fake or the disclosures are deceptive, though those problems are real and documented. But because identity formation, the process by which children develop values, self-perception, social expectations, and a working model of reality, happens continuously, passively, and long before any child possesses the cognitive framework to understand that an algorithm is shaping it.
If AI governance does not move upstream, to the conditions under which AI influences children rather than only to the outputs and incidents that follow, then we are building an elaborate legal architecture that protects children after the formative damage has already been done.
How AI Became a Formative Environment, Not Just a Tool
There is a meaningful distinction between a tool and an environment. A hammer is a tool. The neighborhood a child grows up in is an environment. Tools extend capability. Environments shape identity.
For most of the early internet era, digital systems functioned more like tools. Search engines returned results. Email delivered messages. Online games had levels. The interactions were transactional, bounded, and relatively legible even to children with limited digital literacy.
Modern AI-driven systems operate on a categorically different logic. Recommendation engines do not simply respond to stated preferences. They model behavior, predict engagement, and surface content designed to extend the interaction. Personalization algorithms do not just customize an experience. They gradually narrow the range of experiences a user is exposed to, reinforcing certain patterns while making others progressively less visible. Engagement-optimization systems do not just keep users on a platform. They learn which emotional states, social anxieties, and identity questions generate the most durable attention and serve content calibrated to those pressure points.
For adults, these systems influence preferences and consumer behavior. They can reinforce political views, push purchasing decisions, and shape cultural consumption. These are not trivial effects, and the governance challenges they raise for adult users are substantial in their own right.
For children, the stakes are structurally different. An adult who uses a recommendation engine already has a relatively stable identity, a set of values, a social context, and a worldview developed over decades of embodied experience. Algorithmic influence operates on that foundation. For a child, no such foundation yet fully exists. The algorithm is not influencing a formed identity. In developmental terms, it is participating in forming one.
Repetition shapes children. Reinforcement shapes children. The things that are normalized through consistent exposure become part of what feels real and possible. When AI systems decide what is visible, what receives positive feedback, what is surfaced repeatedly and what disappears, they are not just delivering content. They are acting as a kind of invisible curriculum for how the world works and who a child is within it.
The Consent Mechanism Cannot Reach What It Cannot See
Privacy law has long recognized that consent is an imperfect tool, particularly for vulnerable populations. The EU General Data Protection Regulation treats children’s data with heightened protections and imposes strict requirements on the legal bases available for processing it. The Children’s Online Privacy Protection Act in the United States requires verifiable parental consent before collecting personal information from children under 13. The UK Children’s Code, which took effect in 2021, introduced age-appropriate design requirements that go beyond consent to require that digital services accessed by children be configured with their best interests as a primary design consideration.
These frameworks represent genuine progress. They acknowledge that children are not simply small adults and that the standard consent-and-notice architecture is inadequate when applied to them without modification. But even the most sophisticated child-protective privacy frameworks share a structural limitation: they are primarily oriented toward data collection and processing. They ask whether personal information is being taken, whether parental consent exists, whether data is being shared inappropriately.
They are much less equipped to address influence that operates not through data collection but through environmental design. A recommendation algorithm does not need to collect a child’s name, address, or social security number to shape their developing self-concept. It needs to present certain kinds of content repeatedly, withhold other kinds, reward specific emotional responses, and do so over a sustained period of time. None of that necessarily triggers a COPPA review or a GDPR Article 8 analysis. The influence accumulates in the gap between what privacy law regulates and what AI systems actually do.
There is no moment in the flow of algorithmic experience at which a child, or their parent, encounters a clearly bounded decision point. A notification does not appear saying: “This system is about to normalize a particular definition of social success. Do you consent?” The influence is ambient. It is the water the child swims in, not a door the child chooses to open.
By the time any governance mechanism activates, whether parental permissions, transparency disclosures, enforcement actions, or class-action lawsuits, the formative exposure has already occurred. Governance that arrives after the fact is, by design, a response to damage rather than a prevention of it.
Pre-Consent Harm: A Legal and Developmental Concept That Has No Home Yet
There is a name for what is missing from current frameworks, even if no statute or regulation has fully codified it yet. Privacy and AI governance scholars have begun describing it as pre-consent harm, a category of injury that occurs before an individual is developmentally capable of understanding or resisting the forces shaping them.
The concept is more radical than it might initially appear, because it severs the traditional link between harm and awareness. In most legal frameworks, harm requires some combination of unlawful conduct, a discrete injurious act, and a plaintiff who can articulate what happened to them and why it constitutes a cognizable injury. Pre-consent harm, by its nature, occurs in the absence of awareness. It can arise without any unlawful data processing. It can occur even when a platform is technically COPPA-compliant, GDPR-compliant, and operating entirely within the bounds of its disclosed privacy policy.
A platform that repeatedly surfaces content portraying a narrow definition of physical attractiveness to a 10-year-old girl is not necessarily collecting her data unlawfully. But the developmental effects of that exposure, absorbed over months or years of daily engagement, may be profound and lasting. A platform that rewards adolescent boys for increasingly extreme expressions of dominance and contempt is not necessarily violating anyone’s terms of service. But it is participating in the formation of social attitudes that those young men will carry into adulthood.
Pre-consent harm sits in this space. It is real, it is developmental, it is shaped by algorithmic design choices, and existing law has almost no language for it. The absence of legal language for a harm does not mean the harm does not exist. It means governance has not yet caught up to the reality it is supposed to govern.
The Diagnostic Misframe: When Individual Pathology Masks Systemic Influence
One of the most consequential governance failures around children and AI is how the downstream effects of formative algorithmic influence are typically framed when they become visible. A child who develops severe anxiety, disordered eating, a distorted sense of social reality, or significant behavioral dysregulation is typically routed into clinical or educational systems. Diagnosis, therapy, medication, individualized education plans. The frame is individual pathology requiring individual treatment.
That frame is not wrong. Children who are struggling need support, and medical and therapeutic responses are often genuinely necessary. But the individualized frame systematically obscures the systemic dimension of what is happening. When the same patterns appear across millions of children, across different demographics, different school systems, different family structures, the most parsimonious explanation is not that millions of individual children are experiencing separate and unrelated failures of mental health. It is that the shared environments those children inhabit are producing shared effects.
The peer-reviewed literature on adolescent anxiety, depression, and social comparison has grown substantially over the past decade, and a meaningful portion of it points toward the role of algorithmically curated social media exposure in shaping the mental health outcomes of young people. Researchers at institutions including Harvard, Stanford, and University College London have produced studies linking heavy algorithmic platform use among adolescents with increased rates of social anxiety, body image disturbance, and depressive symptoms, with effect sizes that are modest but consistent across large samples.
The governance implication is significant. If millions of children are being shaped in systematically harmful ways by the design of AI-driven platforms, and if the response is exclusively clinical treatment of individual children, then the companies whose design choices produced those effects will never be held accountable for them. The cost of systemic influence gets externalized onto families, schools, and healthcare systems. The profit from that influence stays with the platforms.
Upstream governance asks a different question than clinical diagnosis. Not: what is wrong with this child? But: what conditions are producing this pattern, and who is responsible for designing those conditions?
What Upstream Governance Actually Looks Like
Moving AI governance upstream, from its current focus on data processing events and after-the-fact enforcement to the design conditions under which AI shapes children’s development, requires changes at multiple levels.
At the regulatory design level, it means shifting from a purely transactional model of privacy protection to a developmental protection model. Rather than asking only whether personal data was collected with appropriate consent, regulators must also ask whether AI systems deployed in environments where children are present are designed to optimize for outcomes that are compatible with healthy child development. The UK Children’s Code represents a partial step in this direction, requiring that platforms likely to be accessed by children default to privacy-protective settings and consider children’s best interests as a design priority. But best-interest standards for data are not the same as developmental impact assessments for algorithmic design.
At the technical design level, it means requiring that AI systems deployed in child-accessible contexts be evaluated not just for data security and privacy compliance but for developmental effect. What does repeated exposure to this recommendation system do to a child’s developing self-image? What social norms does this engagement-optimization algorithm reinforce? These are not questions that current privacy impact assessments are designed to answer, and answering them requires interdisciplinary collaboration between AI engineers, developmental psychologists, and child welfare researchers that the industry has largely not prioritized.
At the liability level, it means developing legal theories that can reach pre-consent harm. The existing toolkit, negligence, deceptive practices, products liability, statutory privacy violations, is not well-suited to developmental harm that accumulates gradually, that is diffuse across millions of young users, and that produces no single discrete injurious event. Some legal scholars have proposed that AI systems deployed in child-accessible contexts should be subject to a duty of developmental care, analogous to the duties of care imposed on other professionals who work with children, including teachers, physicians, and childcare providers. Whether that theory gains traction in courts or legislatures remains to be seen, but the argument is gaining momentum in academic literature.
The State-Level Landscape: Where Upstream Thinking Is Emerging
Several states have begun moving in directions that reflect, at least partially, the upstream governance logic described here. California’s Age-Appropriate Design Code, modeled closely on the UK Children’s Code and signed into law in 2022, requires businesses that provide online services likely to be accessed by children under 18 to conduct data protection impact assessments that account for the risks to children from engagement-promoting features, from the default visibility of children’s posts, and from systems that estimate users’ ages. The law was challenged on First Amendment grounds, and its status in the courts has been contested, but its underlying philosophy, that design choices affecting children require prospective impact assessment rather than just reactive enforcement, represents exactly the upstream orientation that current AI governance frameworks lack.
Texas, Florida, and several other states have passed or are actively considering legislation requiring parental consent for minors’ social media accounts, restricting algorithmic recommendation features for minors, and imposing data minimization requirements on platforms likely to be used by children. These laws have faced serious First Amendment challenges, and courts have produced inconsistent rulings. But the legislative impulse they reflect is significant: an acknowledgment that downstream enforcement alone is insufficient and that the design conditions under which AI influences children require affirmative regulatory intervention before the harm occurs.
At the federal level, the Children and Teens’ Online Privacy Protection Act, known as COPPA 2.0, would extend the age threshold for heightened privacy protections from 13 to 16, ban targeted advertising to minors, and require data minimization for child users. The Kids Online Safety Act, which has passed the Senate with bipartisan support, would impose a duty of care on platforms with respect to child users and require risk assessments for features that could harm minors. Neither has yet become law, but the legislative pressure is building, and it is building precisely because parents, educators, and policymakers have observed the downstream effects of several years of largely unregulated algorithmic influence on a generation of children.
Why “No Malicious Intent” Is Not an Adequate Defense
A common response from the technology industry to concerns about AI’s effects on children is that no harm was intended. Platforms were not designed to damage children’s mental health, distort their self-image, or narrow their developing sense of what is possible in the world. The algorithms were optimized for engagement. The personalization was meant to improve user experience. The recommendation systems were designed to keep people on the platform because that is the business model, not because anyone wanted to hurt children.
This argument deserves to be taken seriously, and then set aside. Intent is relevant to criminal liability and to some forms of civil fraud. It is largely irrelevant to the question of whether a system is producing harmful effects, whether those effects are foreseeable given what is known about child development and algorithmic design, and whether the companies operating those systems have an obligation to mitigate them.
A pharmaceutical company does not escape liability for a drug’s side effects by demonstrating that it intended only to help patients. A building contractor does not escape liability for structural failures by demonstrating that they wanted to build a safe building. Intent establishes motive. It does not establish safety. When a foreseeable consequence of a design choice is harm to children, the fact that harm was not the goal does not dissolve the responsibility to prevent it.
The governance question is not whether AI companies intended to harm children. It is whether the design choices they made, the optimization targets they selected, the features they built, and the safeguards they declined to implement were appropriate given what can reasonably be known about the developmental vulnerability of the children their systems reach. That is a question about standards of care, not about malicious intent.
What This Means for Parents Today
- Understand that privacy settings and parental controls address data collection, not developmental influence. Turning off location sharing on your child’s apps does not limit the algorithmic reinforcement those apps are designed to deliver. The most privacy-compliant platform in the world can still be designed to optimize for engagement in ways that affect a child’s developing self-image and social expectations.
- Duration matters more than content in isolation. Research consistently suggests that cumulative exposure time, rather than exposure to any single piece of content, is the more reliable predictor of developmental effect. The question worth asking about any AI-driven platform your child uses is not just what it shows them but how much of their attention and how many hours of their developmental years it is designed to capture.
- School-mandated platforms deserve the same scrutiny as consumer apps. As the PowerSchool and Naviance cases demonstrate, platforms assigned by schools are not inherently safer than consumer-facing products. Mandatory educational technology may collect sensitive behavioral data, deploy third-party analytics, and optimize for engagement metrics that are not aligned with educational outcomes. Parents have FERPA rights to inspect their child’s educational records and to ask their school district what data vendors are collecting and how.
- The regulatory landscape is moving, but slowly. Multiple federal and state bills address aspects of child AI safety, but most have not yet become law. Parents should not assume that what their children’s platforms are legally permitted to do today reflects what is developmentally safe.
- Advocacy at the school district level is concrete and actionable. Demanding that your school board require algorithmic impact assessments before adopting edtech platforms, and vendor certification requirements that prohibit third-party behavioral data monetization, are specific governance changes that parents can push for through existing institutional channels without waiting for federal legislation.
The Deeper Question Governance Has Not Yet Asked
There is a question that sits beneath all of the legal and regulatory analysis described in this article, one that governance frameworks have been slow to ask directly: what kind of people do we want AI systems to help children become?
That question sounds philosophical, and it is. But it is also a practical design and governance question, because AI systems deployed in environments where children spend significant portions of their developmental years are already answering it, whether anyone asked them to or not. Every optimization target encodes a value. Every engagement metric reflects an implicit theory of what human attention and behavior are for. Systems designed to maximize time-on-platform are not neutral about human flourishing. They are making a bet that human attention is the primary resource to be captured, and designing accordingly.
Governance that is adequate to the challenge of AI and child development must eventually get to this question. Not just: was the data collected lawfully? Not just: did the parent consent? But: are the conditions this system creates compatible with the development of children who are curious, resilient, socially connected, capable of self-reflection, and equipped to participate in a democratic society?
Children cannot opt out of the environments adults build for them. They absorb those environments. They become, in significant part, what those environments make it easy and rewarding to become. When the architects of those environments are AI systems optimized for engagement, and when governance arrives only after identity is already formed, the law is not protecting children. It is processing their paperwork.
Meaningful protection has to start earlier. It has to start upstream, where the formative influence first takes hold, before the child is old enough to consent, before the damage is done, and before the only options left are diagnosis and litigation.
