Abstract

Mello, Char, and Xu propose in JAMA a two-factor framework for deciding when healthcare organizations should notify patients about AI tools or seek their consent. The framework asks organizations to assess risk of harm and patient agency, then sort AI tools into three bins, namely consent, notification, or neither. I argue that the framework rests on three premises that undermine its own architecture. First, it treats “human in the loop” oversight as a reliable error-interception mechanism while simultaneously cataloging the reasons it is not. Second, it quantizes patient agency into a binary when agency exists on a gradient, assigning patients the role of quality-control agents while arguing elsewhere that patients cannot absorb more information. Third, it presumes that healthcare organizations possess the evaluative infrastructure to perform the risk assessments the framework demands. These are not independent weaknesses. They are surface expressions of a deeper structural problem. The Mello framework is a consent architecture, and consent architectures fail when they assume human capacities that do not exist at scale. Privacy law scholarship has already diagnosed this failure and begun developing alternatives. Healthcare AI governance should reckon with the same insight. I acknowledge that diagnosing the consent failure is easier than building the alternative, and that disclosure sometimes imposes real costs on patients. But these complications do not rescue a framework whose premises do not hold.

I. The Source and Its Stakes

Michelle Mello, Danton Char, and Sonnet Xu published “Ethical Obligations to Inform Patients About Use of AI Tools” in JAMA in September 2025. AI tools are proliferating across healthcare settings, and practitioners need guidance on what to tell patients about them. The authors propose a two-factor framework organized around (1) the risk that using the tool could cause physical harm and (2) the extent to which patients can exercise agency in response to a disclosure. Tools that score high on both factors warrant consent. Tools that score high on one warrant notification. Tools that score low on both warrant neither.

The framework contains structural weaknesses that deserve direct treatment. I will focus on three, then explain what connects them.

II. The Circular Loop

The authors build their framework on the premise that human oversight reduces residual risk to patients. They write that “most tools in use today entail a human reviewing and acting on model output,” and that “the key question therefore is whether the residual risk to patients is low once one considers the likelihood that the human in the loop will successfully detect errors.”

This is the load-bearing wall of the entire architecture. It determines which tools fall into the “neither consent nor notification” category. If human oversight reliably catches errors, residual risk is low, and disclosure becomes unnecessary.

And then the authors undermine it themselves. In the very next section, they acknowledge that “capacity constraints, automation bias, and users’ unfamiliarity with a tool’s weaknesses may undercut efficacy.” Automation bias is one of the most well-documented phenomena in human-computer interaction research. Clinicians who know an AI has flagged or not flagged a condition routinely defer to the machine’s judgment. The premise that human oversight will “successfully detect errors” is precisely the premise that automation bias scholarship has spent two decades dismantling.

The framework says, in effect, that disclosure is unnecessary when a human in the loop will catch errors, and then concedes that humans in the loop frequently do not catch errors. If the human-in-the-loop assumption holds only sometimes, the entire “neither” category is unstable. Some of the tools the authors place in that category belong there. Some do not. And the framework provides no mechanism for distinguishing which is which.

III. The Agency Paradox

The second factor in the framework asks whether patients have a “meaningful opportunity to exercise agency.” The authors identify two forms. Patients might opt out of the AI tool, or patients might alter their behavior in response to knowing AI is involved.

The second form does important work. It is the basis for recommending notification for AI-drafted patient emails and AI-generated clinical summaries. The authors argue that a patient who knows an email was AI-drafted “may be more likely to question something that seems odd,” and that a daughter who knows a nursing summary note relating to her mother was AI-generated “may be more likely to log on to the electronic health record, check the note for errors and omissions, and alert the incoming nurse.”

Earlier in the article, however, the authors argue against broad disclosure partly because “patients who want to be kept informed about their care in principle may struggle with the information overload that a hospital admission entails.”

These two positions are in tension. You cannot simultaneously argue that patients are too overwhelmed to process more information about AI tools and that informed patients will serve as effective quality-control agents for AI-generated communications. The daughter expected to check an AI-generated nursing note for errors needs the medical literacy to recognize what constitutes an error, the time and inclination to log into the electronic health record, and an understanding of what the note should contain. The framework assigns her a role it has given no reason to believe she can perform.

The deeper problem is that the framework treats agency as binary. Either patients can opt out or they cannot. Either they can alter their behavior meaningfully or they cannot. But agency exists on a gradient. A patient told about an AI tool but lacking the expertise to evaluate its output has more agency than one told nothing, but less than the framework assumes. A more honest treatment would acknowledge that the “notification” category rests on aspirational rather than demonstrated patient capacity.

IV. The Missing Institutional Competence Question

The framework instructs healthcare organizations to assess, for each AI tool, “(1) the risk that the tool poses, (2) the likelihood that errors will reach patients without being intercepted, and (3) the severity of the harm that could result.”

Who performs this assessment? With what data? And using what methodology?

Most healthcare organizations are still struggling with basic AI governance, the setting of AI governance committees notwithstanding (that’s window dressing). They lack the technical personnel to audit algorithmic performance across patient subgroups, the data pipelines to monitor error rates in production, and the institutional processes to translate risk assessments into disclosure policies applied consistently. These are not speculative deficiencies. A 2025 CHIME Foundation survey found that only 8% of healthcare organizations described themselves as ‘very confident’ in their ability to identify emerging AI risks, and a little more than half had a formal process requiring approval before AI implementation. There is a certain irony in a framework that worries about “perfunctory and legalistic” consent being implemented by institutions that may apply the framework itself in a perfunctory and legalistic manner. An organization that lacks the infrastructure to assess algorithmic risk will default to the path of least resistance. And the path of least resistance in this framework is the “neither” category, because it requires no action. The framework’s own structure creates an incentive to underassess risk.

V. The Disclosure-Harm Tradeoff

Before connecting these weaknesses, I want to engage the strongest argument the Mello framework has in its favor, because it is genuinely difficult.

The authors argue that disclosure can paradoxically harm patients. They cite evidence that patients perceive AI-drafted messages as more empathetic than physician-drafted ones, but rate those same messages significantly lower once told AI was involved. When AI-augmented care outperforms the alternative, a consent regime may result in clinicians “having to deliver suboptimal treatment.” This is a real cost. It is not hypothetical. And any critique of the framework must reckon with it.

I think the argument is correct on its own terms but insufficient as a foundation for the framework’s permissive categories. The disclosure-harm evidence establishes that some patients, told about some AI tools, will make choices that leave them worse off. That is a genuine tradeoff. But the framework uses this tradeoff to justify a “neither” category that sweeps in tools where the calculus is far less clear. The evidence that disclosure harms patients comes from specific contexts (patient messaging, mammography interpretation) and cannot be generalized to every tool the framework places in the “neither” bin without further empirical work. The framework treats a finding about particular tools as a license for institutional silence across categories.

Moreover, the cost of disclosure-induced suboptimal choices must be weighed against the cost of systematic under-disclosure by organizations that default to the category requiring no action. The former cost is visible and measurable. The latter is diffuse and delayed, which makes it harder to track but not less real.

VI. The Consent Substrate

The three weaknesses in Sections II through IV are not independent. They share a common root. The Mello framework is, essentially, a consent architecture. It assumes that if organizations assess risk and disclose appropriately, patients will exercise informed agency. The three failures are symptoms of a condition that privacy law scholarship has already diagnosed.

The notice-and-consent paradigm in privacy law fails at four sequential stages. Reading is impossible given the volume of policies. Comprehension is unattainable given their complexity. Evaluation is foreclosed because users lack the technical expertise to assess risk. And action is impossible because users face take-it-or-leave-it terms. Even ambitious regulatory frameworks like the CCPA have failed to remedy these defects, because rights are rendered worthless when they cannot be exercised. The evidence base for these claims is substantial. Carnegie Mellon researchers calculated in 2008 that reading the privacy policies an average American encounters would require roughly 30 working days per year. Literacy surveys show nearly half of American adults lack the reading level these policies demand. CCPA exercise rates remain remarkably low years after implementation.

Empirical work on patient attitudes toward AI disclosure exists and is growing, but there remains almost no empirical measurement of the structural variables the analogy requires, such as how many AI tools a typical admission implicates, whether patients in practice comprehend AI disclosure forms, or how often patients exercise opt-out rights when offered, and I want to be precise about that gap. But the structural parallels are strong enough to warrant concern. Reading is impossible when a patient’s care implicates dozens of AI tools. Comprehension is unattainable when evaluating algorithmic risk requires expertise patients do not possess. Evaluation is foreclosed because patients cannot assess whether a human in the loop is actually catching errors. And action is impossible for operational AI tools that patients cannot opt out of. Empirical work specifically measuring patient comprehension of AI disclosure forms, and patient exercise rates when opt-outs are offered, would test whether these parallels hold quantitatively. But the structural logic does not depend on the numbers being identical. It depends on the same mismatch between assumed and actual human capacity.

Privacy law scholarship has begun moving past this impasse. The emerging recognition is that when consent is structurally impossible, effective protection requires delegation to intermediaries capable of acting at scale on behalf of individuals. Whether that intermediary takes the form of an AI agent with limited legal capacity, a fiduciary with enforceable duties, or some other institutional design, the underlying insight is the same. The evaluative function that consent regimes assign to individuals must be relocated to systems designed to perform it.

I am aware that identifying the consent failure is easier than building the alternative. An intermediation model for healthcare AI governance raises questions about cost, access, liability, and regulatory design that this commentary cannot resolve. Healthcare is a domain where regulatory complexity, liability exposure, and patient vulnerability are all higher than in consumer data protection. The intermediaries that privacy law scholarship envisions do not yet exist in healthcare, and constructing them is an enormously difficult institutional project.

But the first step is recognizing that the current framework rests on assumptions that do not hold. The Mello framework asks patients to do what privacy law has demonstrated individuals cannot do, namely evaluate institutional disclosures about complex algorithmic systems and exercise meaningful agency in response. A healthcare AI governance framework that takes the consent literature seriously would ask not “what should patients be told” but “what institutional structures ensure that someone with the competence to evaluate algorithmic risk is actually doing so on the patient’s behalf.”

VII. What the Framework Does Not Say

The Mello framework assumes human oversight that automation bias research calls into question. It assumes patient agency that information overload scholarship undermines. It assumes institutional capacity that healthcare governance surveys consistently find lacking. And it inherits these assumptions from a consent paradigm whose structural failure is no longer a matter of speculation but of accumulated evidence.

A disclosure framework is only as strong as the institutions that implement it, and a consent architecture is only as strong as the humans it expects to exercise consent. Without intermediation, the elegant two-factor matrix becomes a mechanism for sorting AI tools into the category that requires the least organizational effort. That is not a governance framework. That is a permission structure for opacity.