Professional AI’s Dual Trust Problem

Balkinization
an unanticipated consequence of
Jack M. Balkin

Balkinization Symposiums: A Continuing List

E-mail:
Jack Balkin:
jackbalkin at yahoo.com
Bruce Ackerman
bruce.ackerman at yale.edu
Ian Ayres
ian.ayres at yale.edu
Corey Brettschneider
corey_brettschneider at brown.edu
Mary Dudziak
mary.l.dudziak at emory.edu

Joey Fishkin
joey.fishkin at gmail.com
Heather Gerken heather.gerken at yale.edu
Abbe Gluck abbe.gluck at yale.edu
Mark Graber
mgraber at law.umaryland.edu
Stephen Griffin
sgriffin at tulane.edu
Jonathan Hafetz
jonathan.hafetz at shu.edu

Jeremy Kessler
jkessler at law.columbia.edu
Andrew Koppelman
akoppelman at law.northwestern.edu
Marty Lederman
msl46 at law.georgetown.edu
Sanford Levinson
slevinson at law.utexas.edu
David Luban
david.luban at gmail.com
Gerard Magliocca
gmaglioc at iupui.edu
Jason Mazzone
mazzonej at illinois.edu
Linda McClain
lmcclain at bu.edu
John Mikhail
mikhail at law.georgetown.edu

Frank Pasquale
pasquale.frank at gmail.com
Nate Persily
npersily at gmail.com
Michael Stokes Paulsen
michaelstokespaulsen at gmail.com
Deborah Pearlstein
dpearlst at yu.edu
Rick Pildes
rick.pildes at nyu.edu
David Pozen
dpozen at law.columbia.edu
Richard Primus
raprimus at umich.edu
K. Sabeel Rahman
sabeel.rahman at brooklaw.edu
Alice Ristroph
alice.ristroph at shu.edu
Neil Siegel
siegel at law.duke.edu

David Super
david.super at law.georgetown.edu
Brian Tamanaha
btamanaha at wulaw.wustl.edu
Nelson Tebbe
nelson.tebbe at brooklaw.edu
Mark Tushnet
mtushnet at law.harvard.edu
Adam Winkler
winkler at ucla.edu

Compendium of posts on Hobby Lobby and related cases

The Anti-Torture Memos: Balkinization Posts on Torture, Interrogation, Detention, War Powers, and OLC

The Anti-Torture Memos (arranged by topic)

The First Problem: AI’s Professional Advice Isn’t Trustworthy

The professional relationship with a doctor, lawyer, financial advisor, accountant, pharmacist, therapist, or another advice-giving professional is a specific social interaction. The professional possesses knowledge the client lacks; this results in a knowledge asymmetry that creates vulnerability. The law responds with a set of safeguards: licensing requirements, fiduciary duties, malpractice liability, informed consent. These mechanisms protect the conditions under which a client or patient can reasonably place confidence in a professional’s expertise.

Public-facing AI eliminates the human professional. What remains looks like professional advice: conversational, personalized, authoritative in tone. But the legal and ethical framework that ensures professional advice is trustworthy is absent.

The accuracy problem is real and documented: as of 2024, no commercially available AI app met professional standards for skin cancer detection. Earlier studies on general health queries found frequent errors, and more recent work reinforced those findings. A study published in Nature Medicine found that participants using AI chatbots to navigate common medical scenarios performed no better than a control group relying on ordinary home resources such as internet searches—and users describing the same symptoms sometimes received conflicting advice depending on how they phrased their questions. A separate Mount Sinai study found that ChatGPT Health under-triaged more than half of medical emergencies in structured clinical testing, potentially directing patients with serious conditions toward routine follow-up rather than urgent care.

But accuracy is not even the core issue. Trust is an attitude; trustworthiness is a property. And as Ignacio Cofone argues in a companion piece to this symposium, as well as in more detail in a forthcoming article,[1] trustworthiness is a property of institutions, not of AI systems. The professional relationship, not the chatbot, carries that institutional trustworthiness.

The Second Problem: Untrustworthy AI Undermines Trust in Human Professionals

When a patient consults a public-facing AI and then sees a physician whose advice diverges, the patient faces a question they are not equipped to answer: who is right? And behind that question lurks a more unsettling one: where does expertise actually live?

The proliferation of AI that mimics professional judgment creates epistemic uncertainty about institutional expertise itself. Beyond harming individual users, the AI systematically undermines confidence in the professionals it displaces or contradicts. The problem is structural, rooted in the same inequities of access that drive people to seek AI as a substitute for healthcare in the first place.

Many people turn to AI health tools precisely because they lack access to affordable human care. A March 2026 KFF tracking poll found that about one in five adults who use AI for health advice cite inability to afford a provider as a major reason, a figure that rises to nearly three in ten among users ages 18 to 29. Uninsured adults are more than twice as likely as insured adults to rely on AI for mental health guidance. And the pattern tracks race: Black and Hispanic adults turn to AI for mental health advice at substantially higher rates than White adults. Viewed this way, AI health tools are an attempted patch for a broken system.

The trust being displaced was already fragile, and unevenly distributed across race, income, and geography. Worse, the AI health tools with the most personalized features—those enabling direct integration with medical records—are increasingly behind paywalls, potentially placing them out of reach for those who are already struggling to afford care. What consumer-facing health AI offers is not a substitute for the human professional relationship.

As the law and political economy literature would recognize, the roots of this problem predate AI. First Amendment doctrine, as I have argued elsewhere, has long assumed the availability of professional advice without reckoning with its unequal distribution.[2] This assumption places a heavier burden on those who can least afford expert counsel and who are most dependent on publicly available information (however unreliable) as a substitute. Consumer-facing health AI does not solve this problem; it exploits it, offering a widely available facsimile of expert advice. And the data suggests it’s relied on by users for whom the absence of access to professional advice was already most consequential.

Trust in Institutions, Not AI

The stakes extend beyond individual harm to institutional erosion. As Woodrow Hartzog and Jessica Silbey argue, AI has the capacity to destroy the civic and professional institutions on which public life depends.[3] It may do so by steadily undermining the trust that sustains them. The professions are no exception. Professional expertise generates trust because it is grounded in training, accountable to standards, and answerable to the people it serves. Deploying AI that mimics expertise without embodying any of those properties creates bad individual outcomes and casts doubt on where expertise lives.

The question, then, is not simply whether we should trust AI. It is whether deploying untrustworthy AI erodes the very institutions whose trustworthiness we depend on, and what regulatory frameworks built around human professional relationships can do about it.

Claudia E. Haupt is Professor of Law and Political Science, Northeastern University. You can reach her by e-mail at c.haupt@northeastern.edu.

[1] Ignacio Cofone, Institutional Accountability and Legitimate Inference in Algorithmic Adjudication: Beyond Trustworthy AI, Cambridge Forum on AI Law and Governance (forthcoming 2026), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6516459

[2] Claudia E. Haupt, Assuming Access to Professional Advice, 49 J. Law, Med. & Ethics 531 (2021).

[3] Woodrow Hartzog & Jessica Silbey, How AI Destroys Institutions, UC L. REV. (forthcoming 2026) https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5870623.

Books by Balkinization Bloggers