There is a recording device in your therapist’s office. You may have consented to it. You almost certainly do not know what happens to the recording afterward.
This is the quiet reality unfolding across major health systems as AI-powered ambient scribes — tools that listen to and transcribe clinical encounters in real time — are deployed at scale into one of medicine’s most privacy-sensitive environments: mental health care. The technology is framed as a clinician productivity solution. Its privacy implications are being treated as a disclosure problem, if they are being treated at all.
They are not a disclosure problem. They are a HIPAA problem, a consent architecture problem, and increasingly, a litigation problem. This piece breaks down what is actually happening, what the law requires, and where the exposure lies for health systems and their AI vendor partners.
What AI Scribes Actually Do
Ambient AI scribes are software tools that continuously record clinical conversations and use machine learning to generate clinical documentation — visit notes, treatment summaries, follow-up instructions — with minimal clinician input. The pitch to health systems is straightforward: reduce documentation burden, accelerate throughput, reduce burnout.
The largest deployment in this space is Abridge, which Kaiser Permanente began rolling out in 2024 across hospitals and medical offices in eight states and Washington D.C. — ultimately operating across 40 hospitals and more than 600 medical offices. Abridge operates in more than 14 languages and describes itself as a “Business Associate” to providers under HIPAA.
The tool’s foundational mechanism is continuous ambient recording of clinical encounters. In mental health settings, that means therapy sessions, psychiatric evaluations, and crisis interventions — the most sensitive category of protected health information that exists. Kaiser has stated that recordings are stored for no longer than 14 days, that data processing meets all HIPAA requirements, and that patient data is not used to train AI models. It has also confirmed that patient consent is required before the tool is activated.
What Kaiser has not publicly disclosed in granular detail — and what multiple providers report being unable to obtain from their own institution — is the full contractual framework governing how Abridge handles patient data, what downstream uses are permitted, who has access, and what happens to data when the contract ends.
The Consent Problem Is Structural, Not Incidental
Under HIPAA, the legal floor for using an AI scribe in a clinical encounter is a Business Associate Agreement between the health system and the vendor, combined with patient authorization or a determination that the use falls within the scope of treatment operations. Kaiser maintains that it obtains patient consent before activating Abridge.
But legally valid consent and meaningfully informed consent are not the same thing — and the gap between them is where the real exposure lives.
Providers at Kaiser have reported that the consent process, as implemented, does not consistently explain to patients how their recording is stored, how long it is retained, who has access beyond their direct care team, whether the audio or transcript could be used for AI model training, or what happens to the data if a patient later requests deletion. That is not a procedural gap. That is the functional definition of inadequate informed consent under most state health privacy frameworks and, increasingly, under HIPAA enforcement guidance.
Crucially, providers themselves report being unable to obtain this information from health system leadership. Licensed clinical social workers and psychiatric social workers working within Kaiser’s system have described asking direct questions about patient privacy protections and data governance — and receiving what one characterized as “empty assurances”: assertions of HIPAA compliance without substantive disclosure of the underlying framework.
One provider described the institutional response this way: “We are compliant. That’s it. That’s all you need to know. We vet the technology, therapist. Don’t worry. That’s not your job. We have tech experts. That’s their job.”
Another provider reported that a colleague who raised concerns was told it was “unprofessional” to “provide your personal beliefs on AI in our work setting.”
This is a governance failure with legal consequences. Clinicians who have a duty of confidentiality to their patients — a duty that exists independently of and in addition to HIPAA — cannot fulfill that duty if their institution will not disclose the data handling framework they are operating within. Health systems that deploy AI tools while simultaneously preventing clinical staff from understanding those tools’ data practices are creating institutional liability, not managing it.
Coercive Consent: When the Patient Feels Guilted Into Agreeing
The consent problem has a second dimension that does not map cleanly onto existing legal frameworks but carries significant legal risk nonetheless: the coercive consent dynamic.
Providers within the Kaiser system describe being trained to present Abridge to patients in a framing that places the patient’s comfort in tension with the clinician’s well-being. Patients are told the tool will help their doctor with documentation, reduce burnout, and allow them to spend more time with their families. The implicit message is that refusing the tool creates a burden for the provider.
This framing is not accidental. It is an implementation choice. And it carries meaningful legal risk under state consumer protection statutes, health care patient rights frameworks, and the FTC’s evolving guidance on dark patterns in consent design — all of which treat consent obtained through social pressure or manipulative framing as legally suspect even when the technical act of consent is voluntary.
One provider who experienced this dynamic as a patient described it directly: her physician did not ask for consent before activating the tool, but informed her it would be used. She declined. The session continued, but she perceived a visible change in the physician’s demeanor. She subsequently changed doctors.
A patient interviewed for the underlying reporting was unambiguous about the stakes: “I fear that this kind of information that’s being recorded now can get into the wrong hands. I may not want my employers, I may not want my family members, I might not want people to know some of these very kind of intimate conversations and deep conversations I have with my doctors [and] with my mental health provider.”
That fear is legally grounded. Mental health records occupy a special category of sensitivity under federal and state law precisely because of the downstream harms their disclosure enables.
Why Mental Health Data Is a Different Category of Risk
Not all HIPAA-protected health information carries equivalent risk upon exposure. A blood pressure reading disclosed without authorization is a privacy violation. A therapy session transcript disclosed without authorization is a potential vector for employment discrimination, child custody litigation, immigration consequences, security clearance revocation, and social stigma that follows an individual for years.
Mental health data can be weaponized in ways that most other health data cannot. A record of someone’s lowest moments — a crisis episode, a trauma disclosure, a psychiatric diagnosis — is qualitatively different from a record of a chronic physical condition. It carries stigma. It invites discrimination. And in the wrong hands, it is leverage.
This is why state mental health privacy laws often impose requirements stricter than HIPAA’s baseline. California’s Lanterman-Petris-Short Act, for example, imposes specific confidentiality obligations on information generated in psychiatric treatment contexts. Illinois, New York, and Texas impose comparable heightened protections. AI scribes deployed in mental health settings are not merely subject to HIPAA — they are subject to the full stack of applicable state law, which in most jurisdictions is more restrictive.
The vendor contracting framework matters enormously here. Business Associate Agreements between health systems and AI scribe vendors must address: whether audio recordings may be used to train AI models; the de-identification standard applied to any retained data; data retention schedules and deletion protocols; downstream sharing with subcontractors and third parties; what happens to patient data upon contract termination; and breach notification obligations specific to mental health data. If those terms are not explicitly negotiated and documented, the health system — not the vendor — typically bears the primary regulatory exposure.
The Provider Pressure Problem: Coercion on Both Sides of the Consent Form
The consent problem in AI scribe deployments runs in both directions. Patients are being coerced into consent by the framing of the tool’s presentation. Providers are being coerced into using the tools by institutional productivity pressures.
Providers within Kaiser’s system describe a documentation environment in which patient volume has increased significantly, creating sustained pressure to reduce time per encounter and accelerate clinical note completion. In that environment, AI scribes are presented not merely as a convenience but as a performance management tool — one whose rejection may be characterized as a failure to meet job expectations, subject to disciplinary action.
One provider described the dynamic explicitly: “I consider that to be coercive because you’re putting someone in a position to either lose their job or use the software. That’s another choice that’s under duress.”
This matters legally because coerced provider use has downstream effects on patient consent quality. A clinician who does not understand or trust the tool they are deploying — and who is using it under institutional pressure rather than informed professional judgment — is not positioned to obtain genuinely informed patient consent. They cannot explain what they do not know, and they cannot advocate for what the institution has decided is not their concern.
The structural result is a consent process that is nominally compliant and substantively hollow: patients are asked, patients say yes, and neither the patient nor the clinician has a meaningful understanding of what they have agreed to.
What the BAA Should Actually Say
For health systems and their legal and compliance teams, the AI scribe deployment questions are increasingly not whether to deploy these tools, but how to deploy them with defensible governance. The Business Associate Agreement is the foundational document, and most BAAs currently in circulation for AI scribe products do not address the full scope of relevant risk.
A defensible BAA for an AI ambient scribe deployed in mental health settings should address, at minimum:
- Audio retention and deletion: Specific retention periods for audio recordings (as distinct from transcripts and generated notes), with automated deletion protocols and audit trail documentation.
- Model training prohibition: An explicit, unambiguous prohibition on use of patient audio, transcripts, or derived data for AI model training, fine-tuning, or benchmarking — with no carve-outs for de-identified data unless the de-identification standard meets or exceeds HIPAA‘s Expert Determination method.
- Subcontractor controls: Identification of all subprocessors with access to patient data, with equivalent BAA obligations flowing down to each.
- Access controls: Specificity about which personnel at the vendor have access to audio recordings, under what circumstances, and subject to what authorization requirements.
- Termination data handling: Explicit protocol for destruction or return of all patient data upon contract termination, with certification.
- State law compliance: Explicit representations covering applicable state mental health privacy law, not merely HIPAA.
- Breach notification: Mental health data-specific notification timelines and procedures, given the heightened harm profile of exposure in this data category.
Health systems that cannot obtain these terms from their AI scribe vendor should treat that refusal as a material risk signal — not a negotiating inconvenience.
AI Governance & Patient Rights
For patients navigating this landscape, the legal rights exist. The access to exercise them meaningfully often does not — which is itself a disclosure problem that regulators are beginning to scrutinize.
- You have the right to refuse recording. In all currently known AI scribe deployments, patient consent is required and refusal is legally protected. Exercise that right without apology. If a provider’s demeanor changes after a refusal, document it and consider filing a patient complaint.
- Read intake forms before signing. Consent to AI scribe use may be embedded in intake paperwork rather than presented as a distinct decision. Slow down at any language referencing “ambient,” “documentation technology,” “AI-assisted notes,” or “recording.”
- Ask specific questions. Request written answers to: How long is the recording stored? Who has access beyond my care team? Is my data used to train AI models? What happens to my data if I change providers or if the health system ends its contract with the vendor?
- Request access to your records. Under HIPAA’s right of access, you can request copies of clinical notes generated with AI assistance. Reviewing those notes for accuracy is both your right and a practical quality check on the technology’s output.
- File a complaint if your rights are not honored. The HHS Office for Civil Rights accepts HIPAA complaints and has signaled increased focus on health technology compliance. State health departments and attorneys general offices are additional complaint venues in most jurisdictions.
The Litigation Horizon
AI scribe deployments in mental health settings represent an emerging but clearly developing litigation risk category. The legal theories that plaintiff-side firms are already developing in adjacent contexts — wiretapping claims under state electronic surveillance statutes, breach of fiduciary duty claims grounded in therapist-patient confidentiality, and consumer protection claims based on dark-pattern consent design — translate directly to this factual pattern.
California’s Invasion of Privacy Act (CIPA), which has already generated significant class action litigation in the healthcare context around pixel tracking and chat widget technology, applies with full force to the recording of telephonic and electronic communications without adequate consent. Mental health providers offering telehealth sessions via recorded AI scribe platforms in California should treat CIPA exposure as an active rather than theoretical risk.
Illinois’s Biometric Information Privacy Act (BIPA) creates parallel exposure where AI scribe tools collect voiceprint data — a biometric identifier under Illinois law — without a compliant written release and retention schedule. Given the volume of mental health telehealth encounters involving Illinois residents, BIPA exposure for AI scribe vendors and their health system partners is a question of when, not whether.
The factual record being built in reporting on Kaiser’s Abridge deployment — provider accounts of coercive implementation, institutional refusal to disclose data governance details, patient reports of inadequate consent processes — is precisely the kind of record that supports class certification arguments in healthcare privacy litigation.
The Deeper Obligation
The therapeutic relationship is built on something more than legal compliance. It is built on the patient’s willingness to be vulnerable in a controlled, confidential space — a willingness that is itself clinically necessary for treatment to work. One provider put it plainly: “Therapy is most effective in privacy and when trust is achieved through two human beings. Healing occurs when human empathy is offered sincerely as part of any sort of mental health treatment relationship. I believe recording a therapy session changes human behavior. It changes the patient’s demeanor.”
AI scribe tools that reduce documentation burden and free clinician attention are not inherently incompatible with that therapeutic relationship. But their deployment without transparent data governance, genuine informed consent, and meaningful provider autonomy is incompatible with it. The technology can be deployed responsibly. The question is whether health systems are choosing to do so.
The compliance infrastructure exists. The legal frameworks exist. The patient rights exist. What is missing, in too many deployments, is the institutional will to treat those frameworks as genuine obligations rather than boxes to check.
How Captain Compliance Can Help
Health systems, mental health practices, and AI health technology vendors navigating HIPAA compliance, Business Associate Agreement review, state mental health privacy law obligations, and AI governance frameworks need more than general compliance advice. Captain Compliance provides the specialized privacy software and AI Governance tools from our team of privacy experts to build defensible, audit-ready programs that hold up under regulatory scrutiny and litigation pressure.
Book a demo below to assess your AI scribe deployment’s compliance posture and build the governance framework your patients and your institution require.
