What Should Healthcare Organizations Look for When Evaluating AI Coding Vendors?

The RFP framework that worked for evaluating traditional revenue cycle vendors does not work for evaluating AI medical coding vendors. The questions are different, and the answers require different kinds of evidence. Buyers using a traditional RFP miss the substantive distinctions.

What questions should an AI coding RFP include?

A rigorous RFP for autonomous medical coding has to surface five categories of evidence that standard software RFPs were never designed to surface. The first is auditable accuracy methodology. The right question is not "what is your accuracy" but "how is your accuracy measured, by whom, against what comparison standard, on what chart sample, and at what specialty stratification." A single accuracy number with no methodology behind it is not comparable to another vendor's single number with no methodology behind it. Both are marketing.

The second is specialty footprint versus production claims. The KLAS 2025 Autonomous Coding report documents that current production deployments are concentrated in radiology and emergency department coding, where documentation is structured and code distributions are narrow. A vendor that quotes a high automation rate without disclosing which specialties that rate covers, and in what proportion of the buyer's actual encounter mix, is presenting a number that does not transfer to the buyer's operation. Ask for accuracy and automation by specialty, in the proportions the buyer actually produces. Not the vendor's most-tested slice.

The third is architectural transparency. Does the system train on patient data, and from which patients? Models trained against historical human-coded examples inherit the structural ceiling of human inter-coder reliability, established in Peng et al., 2018 at 82.2% agreement between credentialed coders at four-digit ICD-10 specificity. A system trained that way cannot reliably exceed it. Other architectural questions: where does inference compute run; how are ICD-10-CM, CPT, and HCPCS specification updates handled; how are AHA Coding Clinic and NCCI updates incorporated; how is multi-tenant data isolation implemented; does customer data flow into model training, even indirectly.

The fourth is operational depth. Production medical coding is a regulatory and operational discipline, not a software problem. Who at the vendor holds CCS, CPC, CCS-P, or equivalent credentials, and what authority do they hold over coding decisions and audit findings? How is the post-go-live audit cadence structured, and who runs it? How does the vendor handle denial follow-up, payer-specific rules, and ICD/CPT specification updates between releases? Vendors that grew out of pure software backgrounds frequently lack the operational depth that determines whether automation works in production. The fifth, in the wake of the operational depth question, is compliance posture: BAA execution, U.S. data residency, audit log accessibility, SOC 2 Type II reporting, HIPAA compliance, and the documented controls a covered entity will be audited against itself.

What does "auditable accuracy" look like in practice?

Auditable accuracy is the property of an accuracy claim that an outside party can verify. The methodology that supports it is the credentialed-panel audit: a representative sample of charts is independently coded by two or more credentialed coders, the AI system codes the same charts in parallel, and disagreements are adjudicated against the source clinical documentation and current coding guidance. The published outputs are the initial agreement rate (the share of charts where the AI matched the human coder before adjudication) and the post-adjudication consensus rate (the share where the AI matched the panel-adjudicated ground truth after disagreements were investigated). Both numbers matter. The post-adjudication rate is the system's actual error rate once human coder errors are removed from the comparison.

A vendor-produced audit establishes baseline credibility. A buyer-conducted pilot audit, on a representative sample of the buyer's own charts, is a stronger evaluation signal. The two are complementary. The vendor audit demonstrates the system's performance against a population the vendor has worked with previously; the buyer pilot demonstrates the system's performance against the buyer's documentation patterns, payer mix, and specialty distribution. Buyers should expect to do both, and should not treat reference-customer accuracy claims as a substitute for either. Reference customers typically deploy a narrow slice of their encounter mix at first; ask for accuracy data on the full encounter mix the buyer would deploy, not the slice the reference customer ran the pilot on.

A defensible audit publishes its methodology in enough detail that an outside auditor could replicate it. That means sample size and how it was drawn; specialty stratification of the sample; credentialing level of the human reviewers; the adjudication process and who held final authority over disagreements; and the time horizon. A 500-chart single-snapshot audit is informative. An 18-month longitudinal audit across tens of thousands of encounters by two independent organizations, on the highest-volume and the highest-complexity codes in the field, is a different category of evidence. Both can be done, but they cannot be presented as equivalent.

The AccuCode audit posture, as a concrete example of what published methodology looks like, is independent audit by MedAxiom (the American College of Cardiology's cardiovascular collaborative) on cardiovascular surgery and evaluation-and-management codes, plus independent audit by Professional Consulting Services (PCS) on E/M and broader specialty coverage. Both audits were led personally by the most senior credentialed coders at each organization. Initial agreement of 95.6% with the credentialed coding panel on cardiovascular surgery (the floor of the specialty audit, intentionally chosen because it is among the most complex specialties in the codebook) rises to 99.1% after panel adjudication. On the higher-volume E/M codes, initial agreement is 98.4% and post-adjudication consensus is 99.6%. The full methodology is published on the medical coding product page; what makes those numbers auditable is the methodology, not the numbers themselves.

How do you distinguish rigorous claims from marketing theater?

The patterns that signal marketing theater are recurring and recognizable. The most common is the automation-rate equivocation. A vendor reports "90% automation" without specifying what the 90% is a percentage of. The honest reading frequently turns out to be: 90% of charts within certain workflows that the vendor is configured to attempt, where those workflows represent perhaps 20% of the buyer's actual encounter volume. The net automation rate of the buyer's total volume is then not 90% but 18%. The buyer who deploys at the 90% expectation and discovers the 18% reality has bought a different product than they were sold.

A related pattern is accuracy without methodology. A single accuracy number with no audit methodology behind it is not comparable to any other number. The remedy is to insist on the methodology before accepting the number: who conducted the audit, how was the sample drawn, what was the credentialing level of the human reviewers, how were disagreements adjudicated, and what comparison standard was used. A vendor unwilling or unable to supply that methodology has not, in any operational sense, measured what they are reporting. A vendor whose accuracy is measured "against human coders" without panel adjudication is structurally capped at the human inter-coder reliability ceiling of roughly 82% (Peng et al., 2018), which means the published number is comparing the system to an 82%-reliable standard rather than to source documentation.

Two further patterns deserve scrutiny. The specialty-coverage gap: a vendor claims to handle "all specialties" but the audit data covers radiology, ED, and perhaps one or two other narrow specialties at production accuracy. Per the KLAS 2025 segment report, this is the actual current state of most production deployments. The remedy is to ask for audited accuracy by specialty in the proportions the buyer actually produces, and to treat anything else as untested. The vendor-funded audit: an audit conducted by the vendor, by a consulting firm the vendor has paid, or by a customer with whom the vendor has an active commercial relationship is not equivalent to an audit conducted by an independent third party with no financial relationship. Both can be informative. They are not the same evidence.

What rigor looks like, in contrast, is straightforward to describe. Independent audit by named credentialed reviewers with their full credentials disclosed. Published methodology that an outside auditor could replicate. Specialty stratification of accuracy results across the full encounter mix. Longitudinal accuracy reporting over a meaningful time horizon, not a single snapshot. Compliance documentation that a covered entity will be audited against itself. And operational leadership at the vendor with credentialed coding authority over the system's behavior in production. Buyers who insist on those properties end up with a smaller short list and a higher hit rate on deployments that survive go-live.

About the author

Allen Fienberg, PhD

Chief Strategy Officer, AccuCode AI

Dr. Fienberg co-founded Intra-Cellular Therapies in 2002, leading business development and investor relations as the company grew from startup to a multi-billion-dollar NASDAQ-listed enterprise (ITCI) and developed CAPLYTA, the FDA-approved treatment for schizophrenia and bipolar depression. He holds a Ph.D. in Human Genetics from Yale and conducted postdoctoral research at The Rockefeller University under Nobel laureate Paul Greengard. At AccuCode he leads strategy and healthcare partnerships, drawing on two decades of experience evaluating evidence under the standards that determine whether a clinical or operational claim survives external scrutiny.

View full bio →