What is intelligent document processing? Why IDP matters in the enterprise

Paperwork is the lifeblood of many organizations. According to one source, 15% of a company’s revenue is spent creating, managing and distributing paper documents. But documents aren’t just costly — they’re time-wasting and error-prone. More than nine in 10 employees responding to a 2021 ABBY survey said that they waste up to eight hours each week looking through documents to find data, and using traditional method to create a new document takes on average three hours and incurs six errors in punctuation, spellings, omissions or printing.

Intelligent document processing (IDP) is touted as a solution to the problem of file management and orchestration. IDP combines technologies like computer vision, optical character recognition (OCR), machine learning and natural language processing to digitize paper and electronic documents and extract data from then — as well as analyze them. For example, IDP can validate information in files like invoices by cross-referencing them with databases, lexicons and other digital data sources. The technology can also sort documents into different storage buckets to keep them up to date and better organized.

Because of IDP’s potential to reduce costs and free up employees for more meaningful work, interest in it is on the rise. According to KBV research, the market for IDP solutions could reach $4.1 billion by 2027, rising at a compound annual growth rate of 29.2% from 2021.

Processing documents with AI

Paper documents abound in every industry and every company, no matter how fervently the industry or company has embraced digitization. Whether because of compliance, governance, or organizational reasons, enterprises use files for things like order tracking, records, purchase orders, statements, maintenance logs, employee onboarding, claims, proof of delivery and more.

A 2016 Wakefield research study shows that 73% of the “owners and decision-makers” at companies with fewer than 500 employees print at least four times a day. As Randy Dazo, group director at InfoTrends, explained to CIO in a recent piece, employees use printing and scanning both for ad hoc businesses processes (for example, because it’s more “in the moment” to scan a receipt) and for “transactional” processes (such as part of a daily workflow in human resources, accounting and legal departments).

Adopting digitization alone can’t solve every processing bottleneck. In a 2021 study published by PandaDoc, over 90% of companies using digital files still found business proposals and HR documents difficult to create.

The answer — or at least part of the answer — lies in IDP. IDP automates processing data contained in documents, which entails understanding what the document is about and the information it contains, extracting that information and sending it to the right place.

IDP platforms begin with capturing data, often from several document types. The next step is recognition and classification of elements like fields in forms, the names of customers and businesses, phone numbers and signatures. Lastly, IDP platform validates and verifies the data — either through rules, humans in the loop or both — before integrating it into a target system, such as customer relationship management or enterprise resource planning software.

Two ways IDP recognize data in documents are OCR and handwritten-text recognition. Technologies that have been around for decades, OCR and handwritten text recognition attempt to capture major features in text, glyphs and images, like global features that describe the text as a whole and local features that describe individual parts of the text (like symmetry in the letters).

When it comes to recognizing images or the content within images, computer vision comes into play. Computer vision algorithms are “trained” to recognize patterns by “looking” at collections of data and learning, over time, the relationships between pieces of data. For example, a basic computer vision algorithm can learn to distinguish cats from dogs by ingesting large databases of cat and dog pictures captioned as “cat” and dog,” respectively.

OCR, handwritten text recognition, and computer vision aren’t flawless. In particular, computer vision is susceptible to biases that can affect its accuracy. But the relative predictability of documents (e.g., invoices and barcodes follow a certain format) enables them to perform well in IDP.

Other algorithms handle post-processing steps like brightening and removing artifacts such as ink blots and stains from files. As for text understanding, it typically falls under the purview of natural language processing (NLP). Like computer vision systems, NLP systems grow in their understanding of text by looking at many examples. Examples come in the form of documents within training datasets, which contain terabytes to petabytes of data scraped from social media, Wikipedia, books, software hosting platforms like GitHub and other sources on the public web.

NLP-driven document processing can let employees search for key text within documents, or highlight trends and changes in documents over time. Depending on how the technology is implemented, an IDP platform might cluster onboarding forms together in a folder or automatically paste salary information into relevant tax PDFs.

The final stages of IDP can involve robotic process automation (RPA), a technology that automates tasks traditionally done by a human using software robots that interact with enterprise systems. These AI-powered robots can handle a vast number of tasks, from moving files database-to-database to copying text from a document, pasting it into an email and sending the message.

With RPA, a company could, for example, automate report creation by having a software robot pull from different processed documents. Or they could eliminate duplicate entries in spreadsheets across various file formats and programs.

Growing IDP platforms

Lured by the enormous addressable market, an expanding number of vendors are offering IDP solutions. While not all take the same approach, they share the goal of abstracting away filing that’d otherwise be performed by a human.

For example, Rossum provides an IDP platform that extracts data while making corrections through what it calls “spatial OCR (optical character recognition).” The platform essentially learns to recognize different structures and patterns of different documents, such as the fact that an invoice number might be on the top left-hand side in one invoice but somewhere else in another.

Another IDP vendor, Zuva, focuses on contract and document review, offering trained models out of the box that can extract data points and present them in question-answer form. M-Files applies algorithms to the metadata of documents to create a structure, unifying categories and keywords used within a company. Meanwhile, Indico ingests documents and performs post-processing with models that can classify and compare text as well as detect sentiment and phrases.

Among the tech giants, Microsoft is using IDP to extract knowledge from paying organizations’ emails, messages and documents into a knowledge base. Amazon Web Services’ Textract service can recognize scans, PDFs, and photos and feed any extracted data into other systems. For its part, Google hosts DocAI, a collection of AI-powered document parsers and tools available via an API.

How IDP Makes a difference

Forty-two percent of knowledge workers say that paper-based workflows make their daily tasks less efficient, costlier, and less productive, according to IDC. And Foxit Software reports that more than two-thirds of companies admit that their need for paperless office processes increased during the pandemic.

The benefits of IDP can’t be overstated. But implementing it isn’t always easy. As KPMG analysts point out in a report, companies run the risk of not defining a clear strategy or actionable business goal, failing to keep humans in the loop and misjudging the technological possibilities of IDP. Enterprises that operate in highly regulated industries might also have to take additional security steps or precautions when using IDP platforms.

Still, the technology promises to transform the way companies do business — importantly while saving money in the process. “Semistructured and unstructured documents can now be automated faster and with greater precision, leading to more satisfied customers,” Deloitte’s Lewis Walker writes. “As business leaders scale to gain competitive advantage in an automation-first era, they’ll need to unlock higher value opportunities by processing documents more efficiently, and turning that information into deeper insights faster than ever.”