In March of this year, Carnegie Mellon University’s Block Center for Technology and Society convened a group of interdisciplinary experts to grapple with a pressing question: how should privacy regulation evolve as artificial intelligence reshapes the data landscape? The discussions, summarized in a recent center publication, paint a sobering picture of a regulatory system ill-equipped for today’s realities — and warn that incremental tweaks won’t cut it in the face of rapidly advancing AI.
The event brought together voices from law, technology, policy, and ethics. What emerged was a consensus that the traditional approach to privacy — centered on individual notices and consents — is fundamentally broken. Instead, experts advocated for a framework that puts real responsibility on the institutions collecting and using data, establishes hard limits that can’t be waived, and gets ahead of AI’s transformative risks before they become entrenched problems.
The Current System Is Failing Us
At the heart of the conversations were two core realizations. First, today’s privacy rules don’t meaningfully protect people in a data-driven economy. Second, AI systems are supercharging existing problems, making half-measures not just inadequate but potentially dangerous.
“Privacy is a policy choice, not a technological inevitability,” the summary notes. The current state of online tracking, profiling, and inference stems from deliberate business decisions and design choices. Alternative approaches exist, but without stronger legal and economic incentives, they struggle to gain traction.
The much-criticized notice-and-consent model came under particularly heavy fire. Most people never read privacy policies, and those who do often find them impenetrable. Interface designs nudge users toward “agree” buttons, turning consent into what experts called a legal fiction rather than genuine authorization. This leaves individuals shouldering an impossible burden in systems engineered to overwhelm them.
Privacy harms, meanwhile, are real but hard to quantify in ways that influence policy. While the economic costs of regulation get plenty of attention, subtler damages — loss of autonomy, behavioral manipulation, discriminatory inferences drawn from seemingly innocuous data — are systematically undercounted. The result is policymaking that undervalues the human impact of unchecked data practices.
AI Is Accelerating the Risks
The timing of the discussion couldn’t be more relevant. As AI agents and large language models grow more capable of autonomous action, the volume, velocity, and sensitivity of data processing are exploding. Systems that can act on behalf of users, infer deeply personal attributes, and make decisions at scale present challenges that current rules simply weren’t built to handle.
Experts highlighted how agentic AI will expand data collection and inference exponentially. Existing frameworks struggle even with today’s relatively static data practices; layering on autonomous, multi-step AI processes risks entrenching today’s failures on a much larger scale. Without proactive intervention, we could lock in problematic norms before society fully understands the consequences.
This isn’t abstract academic debate. Recent developments — from national initiatives on AI digital identities to enforcement actions against airlines mishandling passenger health data — show both the promise and pitfalls of trying to govern AI-driven data flows. The Block Center discussions push further, arguing we need systemic change rather than sector-specific patches.
Five Key Findings That Should Guide Reform
The event distilled several actionable insights:
- Privacy is a policy choice, not a technological inevitability. We can design systems differently, but we need the incentives to do so.
- The notice-and-consent framework is fundamentally broken. It places too much weight on individuals who lack meaningful power or information.
- Privacy harms are real but systematically undercounted. Better metrics are needed to capture non-economic impacts like manipulation and loss of dignity.
- Responsibility has been misplaced onto individuals. Consumers can’t reasonably manage privacy in complex, opaque ecosystems designed against them.
- AI intensifies existing privacy risks. Agentic systems demand new thinking about autonomous data use and decision-making.
Shifts for Policymakers
Rather than more procedural hurdles, the experts called for substantive, institution-focused reforms:
Shift responsibility to institutions. Move away from user consent as the main legal basis for processing. Treat major platforms and AI developers as “information fiduciaries” with clear duties to prioritize user interests, backed by enforceable standards that apply regardless of purported consent.
Emphasize substantive protections. Implement strict data minimization rules, limits on secondary uses, and prohibitions or tight controls on sensitive inferences. Certain practices should be off-limits, not merely disclosed.
Prevent harm proactively. Broaden the legal understanding of harm to include psychological, autonomy, and dignity impacts. Ban manipulative design patterns outright. Develop standardized metrics so regulators and companies can consistently assess and address these risks.
Build AI-specific guardrails now. Require impact assessments for high-risk AI systems involving personal data. Limit fully autonomous data processing without human oversight. Mandate transparency around how AI systems collect, infer, and act on personal information.
The overarching message is urgency. “Action must be taken quickly before existing failures are further entrenched,” the summary stresses. Waiting for perfect data or more evidence risks locking in a status quo that benefits powerful data holders at the expense of individual rights and societal well-being.
Privacy Community & AI Governance
For those working in compliance, policy, or AI governance, the Block Center’s takeaways offer a roadmap that moves beyond compliance checkboxes. At Captain Compliance, we’ve long argued that effective privacy isn’t just about avoiding fines — it’s about building systems people can actually trust.
This discussion reinforces the need for organizations to audit not only their legal bases but their entire approach to data responsibility. Companies experimenting with AI agents should pay particular attention: the autonomy that makes these tools powerful also amplifies privacy obligations.
As governments worldwide — from Estonia’s AI identity experiments to ongoing EU and U.S. debates — wrestle with these issues, academic forums like the Block Center play a vital role in shaping smarter policy. The challenge now is translating these insights into concrete legislative and industry action.
The experts gathered in Pittsburgh made one thing clear: the age of AI demands more than updates to old playbooks. It requires rethinking who holds power over personal data and ensuring that power comes with real accountability.
