Many health care organizations and health systems still use paper-based medical records & bills that are scanned into PDF or image formats to turn them into a type of electronic medical records (EMRs). Converting them into proper EMRs by extracting all their information and turning that data into structured formats is a time-consuming but necessary task. It leads to better decision-making by all health care providers over the patient's lifetime in addition to satisfying regulators and insurance companies.
In this article, we take a look at our SOTA pipeline used for processing these long and complex medical records into a data format that can be used for EMRs. We’ve built this out based on countless builds in this domain and have mastered the challenges faced with these specific documents.
The EMRs and electronic health records (EHRs) we'll talk about in this article are in PDF or image formats. PDF EHRs can be one of two types:
Some sample EMRs are analyzed in the sections below to help you understand the data quality challenges we face in EMR information extraction.
Although it looks like a simple layout, such EMRs can trip basic OCR implementations or even cloud services like Amazon Textract. A common idea you see in this article is the idea of correlating information and grouping it based on readability. These documents are full of titles, section titles, key value pairs, and other values that require mapping to other information to fully understand what they mean. Even in the relatively simple document above you can see how this data relates. The column headers help with understanding the key value pairs, and the titles help with understanding the column headers.
A major challenge of extracting patient data from EMRs involves processing tables of text. This is because we have to correlate fields to the column titles with positional information and natural language understanding of the titles. As columns get spread out, it becomes harder to correlate fields and values without misidentifying the fields with values outside the table. Tables are by far the most challenging part of this process. Due to the high variation of formats that providers use, its challenging to build pipelines that fully correlate this information.
This kind of EMR poses interesting challenges to convert into a structured format. This health record data contains highly nested key-value pairs that have multiple layers to them. Headers come in different boldness levels and sizes, which makes them even more difficult to recognize and extract.
Extracting the text while retaining location information is critical for downstream use cases. If the text is extracted in a way that doesn’t help the models understand correlations between different elements, the extracted data will be inaccurate at worst or unusable at best.
In the example below, standard OCR has extracted the text left to right with line breaks where necessary.
We can see that extracting tables left to right doesn’t work and puts the read order of information out of order. If we were using a large language model (LLM) downstream to extract information from this document, it could produce inaccurate results.
AWS Layout Extraction is supposed to be the solution for this. It extracts the text in a format that fits the read order of the document and labels based on the nature of the text (like text, header, or table).
However, you can see that it missed a ton of text and has much worse OCR than the raw text extraction above.
But a bigger problem is that the read schema isn’t entirely correct. The text isn’t correlated to any specific headers, and sub-headers are not correlated to headers. These documents read more like nested structures where the keys above (headers) are relevant to the smaller and more precise values. The key-value pairs and headers to sub-headers do not exist in a vacuum. We need the data to read like this for any downstream processing and understanding. “Date” fields mean nothing to us unless we understand the context of where they are being used.
So, the format that AWS Layout Extraction identifies isn’t really useful. It should recognize that "Admission Information" is not the same level as "Visit Information" and that “Admission Information” is actually a subsection of "Visit Information." This once again causes many issues for downstream models to understand how the data correlate.
This is how the data should look with a nested schema that clearly defines what is a parent of what, what fields are children, and what text is a key and value. This is an extract from a similar document run through our medical document processing pipeline. You can see each section gets its own key-value set under each header.
This EMR has an unusual two-column layout. Such documents can be challenging as they don't represent a large percentage of the data variance we see, which means that document processing systems don’t have much training data to learn the layout, or systems just don’t handle this mapping well. For us, it’s the exact same use case as our resume parser which achieved over 98% on two-column resume formats.
The scanned report above has an additional block of handwritten entries at the bottom. We’ll talk more about it further down.
The technical challenges of this task are the same ones that most real-world document digitization and information extraction projects run into:
A model would extract the allergies section and return it as is based on our use case. But if we have later patient visit documents that list other allergy medications or the removal of this specific one then we wouldn’t want to include this in the results. This is also true when considering the page type that the medication exists on. That means that the entire document schema and understanding of the other fields is valuable when returning information.
Our clinical data extraction pipeline combines vision, natural language processing, and multimodal models to address all these challenges in EMR data extraction. The data extraction process is shown below.
The pipeline consists of these subsystems:
We explain our methodology for health information extraction in detail in the sections that follow.
Some EMR PDFs can run into hundreds or even thousands of pages. Since the nature of information, layout, and medical specialization can be different on every page, we process each EMR page by page while doing internal page operations to extract the data in a better format.
- The format and schema of the document can change quite a bit page by page. If these are medical records that encompass pages from different facilities you might have a ridiculous amount of variation from page to page.
- Many of the downstream operations we do with this extracted data care about specific meta-data found at a page level. Lots of use cases surrounding chronological summaries or Q&A are reliant on date fields and location fields. If we split up these pages into smaller chunks we might lose that context if only a single date or address is provided.
- Chunks that are larger than a single page may incorporate too much information and include different visits, locations etc. While this isn’t the worst case for chronological ordering, splitting relevant context from a visit means that the context is also missing from the next chunk, which can lead to the information not showing up at all in the summary. This is the same idea I talked about in this dialogue summarization article. That being said, our most recent page title based approach reduces the issues that you can run into with the above approach.
At an internal level we use our table extraction, positional OCR, and row matching algorithms to better define the schema inside the document.
Our text extraction subsystem uses a fine-tuned multimodal vision-language model that simultaneously extracts text and does OCR. This reduces spelling mistakes and other misidentification errors of OCR-only approaches. Ensuring high-quality input images to this subsystem greatly improves its accuracy. For that, we apply a set of preprocessing steps to the EMRs to improve their image quality.
Let's look at these preprocessing techniques.
We apply these image processing operations to scanned EMRs to improve the accuracy of character recognition:
Some pages may contain medically relevant handwritten sections or minor additions, strikethroughs, and markings.
Processing handwritten words or short phrases inserted somewhere above or below a printed line is challenging. Handwritten text can change the meaning of the printed text in medically consequential ways. So, they must be merged with the main text accurately based on their locations.
For all this, we use a fine-tuned region-based convolutional neural network (R-CNN) object detector model to identify the positions of handwritten text areas.
After identifying the handwritten regions, more image preprocessing techniques are applied to those regions of interest to improve the accuracy of handwriting recognition. We use techniques based on continuous offline handwriting recognition using deep learning models which include:
The results of handwriting recognition with an off-the-shelf text recognition framework are shown below:
Notice that its character recognition is not accurate.
In contrast, our fine-tuned, multimodal, language-image model produces far better results shown below.
Text extraction consists of the following steps:
Character recognition based only on pixels can go wrong if the scanned image contains missing pixels, noisy text, blurring, or poor handwriting (a common problem in health care).
To avoid such problems, state-of-the-art text extraction prefers multi-modal language-vision models. Rather than base their character recognition on just the pixels, they use additional context like:
This additional context greatly improves the accuracy of the text extraction from both printed and handwritten sections.
Based on the extracted text, we can classify each page according to the type of data into a category like:
For classification, we use an LLM that's aware of health-care concepts. Depending on each client's data security and budget requirements, we either fine-tune a self-hosted open-source LLM or use an LLM service like OpenAI's GPT-4. The page classification helps us understand the overall schema of the medical document and break down information into buckets based on this schema. The goal is to add understanding to specific fields that change their value based on the page they’re on. An example would be a data field. These can be an admission date, discharge date, medical issue date, or others based on the page they’re on. We need this information to properly extract.
The layout understanding is done by our vision-language model which is trained for document layout recognition based on OCR images.
It works as follows:
A sample run of our model on an EMR is shown below:
We can observe the following:
In addition to regular or handwritten text, the text extraction subsystem must also understand special marks that may be medically relevant.
In this first example, the health care provider has circled "Y" for Yes.
Recognizing information like this is critical to the use case. Boxes being checked, conditionals being circled, and specific pages being signed by a physician can completely change the information required for downstream systems. In the above example, if you’re building a summarization system you need to understand which medications are circled yes and which ones are no to ensure the correct information is provided in the summary. If you ever plan on automating the process of extracting relevant information from these medical records, handling this text is a requirement.
In the next example, a printed form contains many check marks that must be accurately identified as checked or unchecked.
Such unusual data outside the purview of regular text extraction is why we fine-tune image-language models for document understanding. We fine-tune our image-to-text model for special marker recognition to produce a single multiple-capability model.
So far, we identified the text, carved out layout elements, and extracted the text of each element. However, these extracted text fragments may not be directly usable for information extraction and other downstream use cases. For example, text like page numbers and hospital logos are just noise to be discarded.
For that, we use the approach described in the language model pre-training over clinical notes for our text extraction model. Basically, we adapted the paper's sequential, hierarchical, and pre-training approaches to teach our model to recognize the text patterns and data sources unique to health records and ignore the rest. We fine-tuned our layout understanding model with additional embeddings for medically relevant information like section labels and ICD codes.
Within each layout element, there may be useful named entities like:
To identify such named entities, we can use the same medical-aware LLM to do NER. This is one of the key benefits of a custom pipeline approach over using something like Textract or Azure document processing services. We can customize the training to recognize specific values instead of the system extracting them as just normal key/value pairs. This helps us downstream as we’ve already identified specific data we care about.
The rather unstructured data in the form of sections, field-value pairs, and named entities identified so far undergo validation and structuring into JSON data elements like this:
Some clients require the structured information to comply with EMR data standards like the HL7 C-CDA which requires the information to follow well-defined extended markup language (XML) schemas. For such clients, we transform the structured information from JSON to the required XML schema.
Similarly, clinical research also follows its own data standards to record clinical trials and clinical information of patients. Our pipeline can handle that use case too.
An approach we're exploring is turning the layout understanding model into a seq2seq model to directly output C-CDA XML, just like the document understanding transformer (Donut) model.
In this article, we explored the challenges we face in EHR data extraction and explained the pipeline of deep learning models we use to accurately extract the data.
If you're a medical startup, health care organization, or service provider assisting with EMR processing, we can help you implement reliable, scalable, and accurate EMR data extraction.
Contact us for consultation and insights into any challenges you're facing in your EMR data extraction initiatives.
