Almost everything written about implementing agentic AI in healthcare is a checklist or a list: “eight use case categories here”, “twelve demo examples there”, a readiness framework to score yourself against. Useful for orientation, and useless the day you actually have to ship something. What I’ve found is missing is a real deployment you can learn from: what got built first, what the architecture actually was, what it scored, and what the team would change. This article is that walkthrough, and the lessons transfer whether the system you need touches records, revenue cycle, patient messaging, or clinical workflows.
This is the story of implementing agentic AI in healthcare the way it actually worked for us: one platform, three systems, honestly labeled. The foundation is a medical document processing and summarization system we built for a top healthcare startup, running at 96%+ extraction accuracy on the top 10 medical fields and 90%+ across all fields. That specific piece is deliberately not agentic, and we will explain why that matters. On top of it sit two systems that are: a Patient Records Q&A system answering clinical questions with correct document-and-visit attribution at over 91% accuracy, and a Medical Chronology Generation system that turns 10,000+ documents per personal injury case into a trial-usable PDF with page-and-line-level citations. Our healthcare startup partner raised its Series A on the foundation alone, two healthcare companies have raised rounds on variants of this work, and the platform runs across both medical and legal industries. For the general, vendor-agnostic build guide, read our agentic AI in healthcare pillar; this article is the deep dive on one deployed platform.
What this walkthrough covers:
Written for the technical decision-maker who is past the readiness checklist and evaluating a real build.
Every implementation guide on this topic starts with use cases. Ours starts with a failure mode and an understanding of the foundational systems required. When an agentic clinical system focused on patient documents and/or EHR retrieval gives a wrong answer, the postmortem almost always lands in the same place: the agent reasoned correctly over the wrong material. It pulled the wrong visit, missed half the patient's records, or worked from a field a generic extractor got wrong. The agent takes the blame; the foundation earns it.
Real clinical document data is messy in ways I’m sure you already know. Patient records arrive in wildly varied layouts with no template consistency. A large share are scans that only exist through OCR. The vocabulary is unforgiving: ICD codes, medication names, and dosages are exact strings where close enough is wrong. The same patient's records disagree about the patient: name spellings, addresses, and visit information vary across sources, so cross-document identity resolution is a real problem, not an edge case (full guide on patient records). And all of it is PHI that, for many deployments, cannot leave the client's infrastructure.
We evaluated the off-the-shelf routes (Amazon Comprehend Medical, Google Cloud Healthcare NLP, generic OCR plus a strong LLM) and they fall short on exactly those axes: accuracy on varied real-world layouts, exact-string field reliability, cross-document patient matching, and on-prem deployment. Our patient record summarization work had already shown us what that gap looks like in practice: stacks that look fine in a demo lose precisely those fields, ICD codes and dosages, on real scans, and miss the layout cues that tie a value to the right field. That is why the first thing we built was not an agent. It was a foundation that could hit a provable accuracy floor on the client’s actual documents.
The foundation we’ve built for multiple healthcare startups is two standalone systems. The first is patient record/document processing: custom document processing that turns varied patient record layouts, including OCR-processed scans, into structured, validated medical fields with no template assumption. The second is summarization: a framework that consumes those structured fields and produces per-document summaries plus a unified patient timeline. The dependency runs one direction. Record processing stands entirely on its own, and some deployments stop there, feeding structured fields straight into downstream systems. Summarization only exists on top of record processing. The same architecture is deployed in production across the medical and legal industries, and the pipeline is customizable at each step per client. The full architecture breakdown is public for readers who want the implementation detail.
Record processing runs in two stages. Ingestion normalizes whatever arrives: scanned PDFs go through a custom OCR and layout-understanding pipeline, born-digital documents are parsed directly, and both land in a common representation before extraction sees them. The layout-understanding side is the part off-the-shelf OCR skips: table detection, table structure recognition, and functional analysis, meaning the system finds each table, maps its rows and columns, then works out which cells are headers and which carry values. The same pipeline is tuned for the messiness shown earlier: handwriting, checkboxes, circled items, and low-quality scans.
Extraction runs fine-tuned models against that representation with a structured output schema. Two techniques from our document processing work carry most of the accuracy. Hierarchical relationship extraction uses program synthesis to learn key-to-value attribution rules on the most common layouts: how far a value tends to sit from its label, how entities map into boxes and tables, even character-size relationships, so a dosage attaches to the right medication rather than the nearest string. This is real research published by one of our engineers!
Layout-schema-driven prompt tuning adjusts the extraction rules per page based on what the layout model recognized: a page classified as a medication list gets medication-focused rules, and pages with nothing worth extracting are skipped rather than run through the full ruleset, which is one of the quieter ways hallucination stays out of extraction output. Per-field validation closes the loop: outputs are checked against the schema, scored by a fine-tuned hallucination-detection model, and retried through a rebuild ladder when a field fails, so a malformed ICD code or an impossible date is caught at the field level rather than discovered downstream. The output is a deliverable on its own: clean, validated fields for every document, whether or not summarization ever runs.
Summarization is the second system, and it reads only what record processing produced; it never touches a raw page. Per-document summaries come first: abstractive summaries grounded in the extracted fields, with conflicts resolved and repeated information deduplicated across records. The framework follows our plan-then-execute approach to long-document summarization, planning what each summary needs before generating it, which is what keeps abstractive quality up at lengths where single-pass summarization degrades.
Timeline assembly is the hardest stage: merging per-document extractions into one patient chronology means reconciling conflicting dates, collapsing duplicate visits, and resolving cross-document references to the same encounter. Keeping the two systems separate is also what makes failures debuggable: when a summary is wrong, the error almost always traces to a processing miss underneath, and evaluating each layer on its own is how you find it.
Deployment is a choice, not a constraint, and it applies to the whole foundation: the same pipeline ships as a fine-tuned model served via Ollama that runs entirely on-prem or in the client’s private cloud, so no PHI leaves the boundary, or as a GPT-backed option for organizations comfortable with managed-model terms.
One boundary worth drawing explicitly, because write-ups in this space blur it constantly: retrieval is not part of actual document processing. Extraction does not search for anything; it processes every page it is handed. Retrieval exists for the agentic systems later in this article, which need to find the right evidence across a patient’s processed records before they reason over it. That layer is a custom medical embedding model, built by improving open-source MedEmbed, that reached state-of-the-art MRR@5 on TRECCOVID and 68% MRR@5 on NFCorpus, where the next-best published model sits at 52%. A 16-point retrieval delta is not a rounding error; it is the difference between ai agents reasoning over the right evidence and the wrong evidence. The pillar covers the embedding work in more depth.
The numbers that matter: 96%+ accuracy on the top 10 extracted fields, and 90%+ accuracy across every extracted field. The figures come from a held-out evaluation set built from the client’s real document mix, layouts, scan quality, and all, with human-verified ground truth per field, not from clean vendor demo documents. That distinction is the whole game in medical document processing: systems evaluated on studio-quality documents routinely fall apart on real records. We built the benchmark before we trusted the pipeline, and the accuracy floor became a gate: no downstream system shipped until the foundation held it.
It also became the business story. Our healthcare startup client raised its Series A with this system as core infrastructure, and two other healthcare companies have since raised rounds using variants of the same platform. Investors did not fund a demo; they funded a measured accuracy floor on real documents.
No. Neither system in the foundation reasons toward a goal, plans steps, or chooses tools. Record processing structures; summarization condenses and assembles what processing produced. Fine-tuned models, structured extraction, deterministic assembly. We label it that way on purpose, because a market full of overclaim is exactly where honest labels earn trust, and because the label carries the implementation lesson: the non-agentic layers are what make the agentic layers possible. If you want the general definition of what does count as agentic, the pillar covers it. The two agentic systems below clear that bar.
The first agentic system answers clinical questions against a patient's records with the answer attributed to the correct document and the correct visit, not the first text match on a page. Ask who the referring physician was, which home-health provider was involved, or more complex queries involving multiple admission dates, and the difference between a right and wrong answer is usually attribution: the phrase appears in six places across three visits, and only one of them is the answer.
The system runs a multi-step reasoning loop rather than a single retrieval pass: question analysis, page-type identification, visit-context classification, cross-document matching, answer construction, and attribution, each as a distinct step with its own validation. It calls tools (retrieval over the foundation's output, document parsing, record matching) and iterates when a first-pass answer misses its confidence threshold, switching retrieval strategies rather than returning a weak answer. Confidence is tiered by design: 50 preset clinical questions carry an accuracy guarantee, while open-domain questions return best-effort answers explicitly flagged at lower confidence, so the interface can treat the two differently.
The machinery is our planning with files agentic framework, run against the foundation’s processed records, so the architecture reads as a generated task plan, deployed goal specific agents leverage the ReAct framework, and a tool registry. A plan-builder classifies each incoming question’s complexity and emits a dependency-ordered plan: what evidence would answer it, which search steps run in what order, which tools each step is allowed to use, and what a valid answer must contain. Simple questions take a fast path (one agent, one round of tool use) instead of paying the multi-agent tax; complex ones fan out into ordered steps.
Execution is a ReAct loop (reason, act, observe, validate) with a budgeted iteration count and fresh query generation on each recursion, and a loop that hits its budget returns a flagged partial answer rather than crashing. The tools that loop calls are the document understanding stack: retrieval through the medical embedding layer; page-type and visit-context classification via LayoutLMv3 plus program synthesis, which turns 'what was the dosage change?' into a lookup against the right field on the right visit instead of a keyword match; and the record-matching architecture that widens search to every document belonging to the patient even when surface identifiers disagree, without which half the patient’s records silently go missing from every answer. Every call runs through a single registered tool interface, where the plan’s per-step tool list is enforced and a failed call comes back as an observation the agent can react to rather than an exception that kills the run. The LayoutLMv3 + program synthesis breakdown documents the document-understanding side of the implementation.
In production evaluation, the system answers the 50 preset clinical questions at over 91% accuracy, classifies document types at 97.8% accuracy, and hits 95.6% top-k=5 retrieval accuracy, with open-domain handling beyond the preset set. Each number maps to a layer: retrieval accuracy tells you the right pages reach the reasoning step, classification accuracy tells you questions route to the right page types, and end-to-end preset accuracy tells you what the clinician actually experiences.
Since some of the questions are non-deterministic we used real healthcare providers to evaluate the answers and provide feedback. Each of these professionals had already reviewed the exact patient medical histories so they understood the real life patient outcomes.
The second agentic system takes the foundation to its logical extreme: given 10,000+ medical documents related to a personal injury case, plus additional sources like police reports and deposition transcripts, it generates a high-end medical chronology as a real PDF. Every visit entry carries its relevant documentation, quotes, and page-and-line-level citations, plus case-specific callouts: medications received, injury focus, and provider callouts tuned to what the case is actually about. The output is a draft an attorney verifies, which is human-in-the-loop with an unusually demanding reviewer.
This is long-horizon agentic work. The system plans the entire chronology task at ingest: what sources exist, how each normalizes into the foundation representation, and what sequence of extraction and assembly the case requires. It applies case-context-aware filtering, using the case's injury focus and relevant medication categories to decide what earns a place in the chronology rather than summarizing everything indiscriminately. And it operates under adversarial scrutiny: every citation in the output could be checked by opposing counsel, so citation provenance is validated at every step rather than trusted.
The goal of this system being agentic instead of a single pass is the ability to use recursions for deeper and more complete search of the documents. We can do this through a few key ideas that this provides:
Scale is the defining constraint where I see these systems fail. 10k documents does not fit any context window, so the work decomposes into parallel sub-agents, each with its own fresh context, handling discrete jobs (extract the visits from this record set, resolve this provider's appearances) and returning short structured results to the coordinator. Multi-source ingestion normalizes police reports and deposition transcripts through the same foundation extraction layer as the medical records, so the chronology reasons over one representation. Visit-level assembly identifies, deduplicates, and enriches each encounter with the case-relevant callouts. The citation architecture tracks provenance from every output sentence back to source document, page, and line, and the PDF pipeline produces the formatted, validated deliverable a legal team actually files. It is the same foundation-first pattern as the Q&A system, pushed to a scale where the architecture either holds or visibly fails.
A case study is only useful if it shows the forks in the road. The calls that mattered:
The platform's outcomes are unusually legible for this space. Our main healthcare client raised its Series A with this build as core product infrastructure. Two separate healthcare startups have since funded rounds using variants of the platform, which says the architecture is portable, not a one-client artifact. The same foundation serves production deployments in both medical and legal industries with different agentic layers on top. And the retrieval layer's benchmark results are publicly verifiable, which is a different tier of claim than the consulting-style outcome percentages this SERP is full of. One platform, multiple funded companies, two industries: that is the durability argument.
The pillar covers the general HIPAA-compliant architecture pattern; here is what this specific build required. Deployment mode was the first decision: for the regulated deployment path, the fine-tuned model serves via Ollama inside the client's infrastructure, and embedding inference runs on-prem as well, so retrieval and generation both happen without PHI crossing the boundary. PHI handling was field-aware: identifying fields stay intact inside the boundary where the product needs them, and de-identification applies where inference does not require identity. And the audit requirement shaped the pipeline itself: extraction and assembly steps log at the action level, so the client's compliance program can trace any output field back to its source document and processing step. None of this is exotic; all of it had to be designed in rather than bolted on.
Several patterns we now treat as standard came out of or were validated by this platform: the agent-factory structure (a library of capability agents, an orchestrator, per-deployment configuration), the plan-then-execute pattern with per-step validation, sub-agents as fresh context windows for document-scale work, and merging multiple retrieval methods with reciprocal rank fusion. The pillar describes each as a general pattern; this platform is where several of them earned the description. Since 2022 we have carried the same machinery across 50+ organizations' agentic systems, and the healthcare stack above is the most complete single expression of it.
The next natural application is the one related literature keeps gesturing at: revenue cycle management. Prior authorization and claims work is, at its core, medical record extraction plus policy retrieval: assemble the clinical justification from the record, match it to payer requirements, and produce a defensible document. Per the AMA, physicians average around 43 prior authorizations per week, consuming roughly 13 hours of physician and staff time. The foundation described in this article is exactly the layer that work depends on; the agentic layer on top changes from Q&A to authorization assembly, and the implementation pattern stays identical. If your problem lives in RCM, read this case study as a head start, not an adjacent story.
This platform's pattern fits when your problem is document-heavy, accuracy-gated, and regulated: large volumes of varied clinical or legal documents, downstream reasoning that inherits every extraction error, an accuracy bar that must be demonstrated rather than assumed, and deployment constraints where PHI cannot leave. If your need is broad, non-specialty automation on standard workflows with acceptable managed-model terms, an off-the-shelf platform is honestly the faster answer, and the pillar's off-the-shelf vs custom decision framework will help you tell the difference for your specific case.
Three key lessons from this helped us refine and take this build further. First, build the evaluation set even earlier than feels reasonable. The held-out benchmark with human-verified ground truth is what made every subsequent claim provable, and every week it did not exist, the pipeline was improving against intuition instead of measurement. Second, treat the accuracy-floor gate as non-negotiable from day one. The discipline of refusing to ship agentic layers on an unproven foundation looks slow early and pays for itself the first time an agent's error traces cleanly to a measurable layer instead of a mystery. Third, invest in per-step customizability sooner. The configuration seams that later allowed one architecture to cover both medical and legal deployments were retrofit work that would have been cheaper as an original design assumption. None of these are glamorous, and all three are now defaults in how we build.
If you are still at the readiness stage, start with the vendor-agnostic build guide. If you are evaluating a specific build, a document-heavy pipeline, a records Q&A system, chronology or reporting generation, or the RCM workflows the same foundation powers, that is the conversation we have with healthcare teams every week. We will scope it honestly, including telling you when off-the-shelf is the better call.
The platform in this case study started as one well-scoped foundation with a hard accuracy bar. Yours can too. Book a call and we will walk through your documents, your constraints, and what the build actually looks like. Let's chat.
Build the foundation first: document extraction and summarization evaluated to a hard accuracy floor on your real documents, deployed in a HIPAA-appropriate mode. Then add agentic layers that plan, call tools, validate, and iterate on top of that foundation, with humans reviewing at defined checkpoints. In our production platform, the foundation runs at 96%+ accuracy on key medical fields, and the two agentic systems above it inherit that reliability. Build evaluation frameworks on real patient data and use real healthcare professionals to review the complex outputs.
Generative ai systems like patient record Q&A that answers clinical questions at over 91% accuracy with correct document-and-visit attribution, and a Medical Chronology Generation system that assembles 10,000+ case documents and electronic health records, plus police reports and depositions, into a chronology PDF filed at trial, every entry cited to page and line. Beyond it, common examples include claims management automation, personalized treatment plan generation, patient engagement analytics, and clinical decision support.
Document-heavy, high-volume, lower-clinical-stakes workflows: record summarization, records Q&A, prior authorization and claims assembly, and portal message triage. They deliver measurable value fast, their errors are catchable in review, and the extraction foundation they require is reusable for everything you build next for healthcare organizations. Past that we see a ton of ROI in virtual health assistants for onboarding and patient monitoring.
The honest variable is the foundation. Building and evaluating the extraction layer against your real document mix is the long pole; the agentic layers on top move faster once the foundation holds its accuracy floor. Scoping is measured in weeks; a production foundation with provable accuracy is typically a few months; agentic layers follow incrementally on top of it.
With fine-tuned models evaluated on real document mixes, our production foundation reaches 96%+ accuracy on the top 10 extracted medical fields and 90%+ across all fields, with the Q&A layer at 97.8% document-type classification and 95.6% top-k=5 retrieval. Generic tools evaluated on clean documents claim similar numbers but rarely hold them on real records; the evaluation methodology matters as much as the figure. While metrics like accuracy scores are valuable during development, real world metrics like improved patient outcomes and reduced administrative burden need to be the foundation.
Choose the deployment mode first: where PHI cannot leave the boundary, serve fine-tuned models on-prem (we use Ollama) and run embedding inference locally too. Add field-aware PHI handling, action-level audit logging through the pipeline, and per-agent access control. Compliance in a clinical setting is the deploying organization's responsibility; the architecture's job is to make it achievable.
Off-the-shelf wins for broad, non-specialty automation where managed-model terms are acceptable. Custom wins when accuracy must be provably high on your specific documents, when PHI requires on-prem deployment, or when the reasoning is specialty-specific. This platform shipped both a fine-tuned on-prem model and a GPT-backed option on the same pipeline, which is the decision made into architecture. Make sure your approach takes ethical considerations into account, and has a seamless integration into your stack.
Most ai healthcare systems are assembled from a few common building blocks rather than a single product. At the foundation sit LLMs, either commercial APIs or open models self-hosted for data residency and cost control, paired with domain-tuned embedding models that understand clinical vocabulary well enough to retrieve the right chart, patient intake form, or prior authorization rule. Around that core, document-intelligence and computer-vision models handle the messy reality of clinical data: layout-aware extraction for clinical trial results, scanned records, and forms, and specialized vision models for medical images such as radiology, pathology, and dermatology studies. Traditional machine learning still does much of the heavy lifting for predictive analytics — risk stratification, readmission and no-show forecasting, patient satisfaction score prediction — while RAG generation grounds outputs in source documents to limit hallucination. Tying it together is an orchestration layer that decomposes work into discrete agent steps, validates each one, and integrates with the systems clinicians already use through interoperability standards like HL7 FHIR, SMART on FHIR, and EHR APIs. Finally, every serious deployment wraps this in a governance layer: audit logging, evaluation harnesses, HIPAA-aligned infrastructure, and explicit human oversight, so that AI accelerates the work of clinical and administrative teams while accountability for patient care decisions stays with the people responsible for it.
