The demand for IDP (Intelligent Document Processing) software is surging. While estimates vary, most agree that the market is growing at ~50% p.a. and will soon exceed $4bn in size. But IDP is not new, so why is demand now so strong? And what should we expect next from the IDP industry?
Why Is IDP Growing So Quickly?
Many vendors have been providing document processing software for decades. Kofax and ABBYY have been long-time leaders in this sector and have deeply capable and mature offerings. But a surge of interest has led to many new market entrants. This ranges from big tech vendors Microsoft, Google, and Amazon to well-funded start-up’s such as Hyperscience and Rossum, and other niche vendors focused by industry, geography, or even document type.
Two drivers are behind IDP’s accelerated market growth, one related to the speed of modern business and the other reflecting recent technology innovations.
Speed of business requires faster document flow
Organizations are under pressure to continually increase their pace of operation. Speed matters to banks as they open new customer accounts, to insurers as they adjudicate claims, and to retailers who need to squeeze latency from supply chains. None can allow slow document flow and processing to delay critical business processes. Therefore, the velocity of information flowing into and through their organization must increase. As a result, IDP is of premium importance to those pushing the frontier of automation in their business.
Cloud-based AI disrupts the economics of document automation
New ways of applying AI in cloud-based systems to understand documents have changed the game in document automation. Prior methods of document processing required large investments of time and effort to buy, install, configure, and maintain software.
Cloud-based AI radically changes a purchaser’s “buy, install, configure and maintain” investment. Buying is done differently, as new usage-based pricing closely aligns usage with cost, unlike older commercial models that required large upfront investments. Installing and maintaining is usually not required. This is because the vendor creates, manages, and maintains the IDP product in a cloud infrastructure for which they take responsibility. As AI is maturing and streamlining its setup approach, configuring the IDP software has become increasingly efficient also.
In short, buyers now have greater need for the information velocity that IDP can help deliver, with cloud innovation making this an attractive market for new vendors who take a cloud-first approach.
What’s Next in IDP?
Innovation continues to be a central focus among IDP vendors, with intense competition for new customers and a greater share of wallet. Future product development will focus on several areas, such as:
Cloud expansion: Traditional market leader ABBYY continues an aggressive push to join the cloud-only innovators with their Vantage product. ABBYY wrap useful prebuilt connectors to RPA and other tech in their marketplace, while Kofax’s TotalAgility Cloud and SmartHub marketplace moves that traditional leader into the cloud too.
Machine Learning (ML) growth: The race is on for vendors to make ML the primary method of automatic document understanding. The “magic” of machine learning requires a lot of work and huge volumes of samples. The large tech vendors have an advantage simply on account of their scale. But smaller vendors who specialize in particular business processes (e.g. invoicing and AP is a popular target for many) are winning by investing R&D in more targeted solutions.
Enterprise-scale IDP: Smaller vendors are investing heavily in order to catch up with the requirements of enterprise-scale customers. This sees R&D budgets being spent now on adding multi-language capability, handwriting recognition, signature detection, and other features large enterprises require.
Expansion of IDP: Documents are only part of an organization’s automation challenge. A document is part of a process, so RPA, BPM/process orchestration, process mining and conversational AI, are increasingly deployed alongside IDP in an end-to-end solution. In a melding of markets, IDP vendors will extend their ecosystem connectivity, allowing RPA and other tech to easily treat their IDP engine as an on-demand service. Meanwhile, vendors with roots elsewhere, such as Automation Anywhere in RPA, will continue to build their own IDP as part of a competitive strategy to provide a single intelligent automation platform.
IDP Widens Its Horizons
In parallel to these developments, the more visionary vendors are investing to win a bigger long-term prize. The spark igniting this growth is lit by a reinterpretation of what constitutes a “document”.
IDP traditionally processes documents of three types.
- Structured documents are fixed form documents, often in printed or fixed PDF formats, such as passport applications. The same data fields are in the same place on every form.
- Semi-structured documents always contain the same essential data but not always in the same place. An invoice will always have an invoice number, but it won’t be in the same place on every invoice.
- Unstructured documents contain information but not predictably or reliably. A letter of complaint will be a free-form text document with entirely unpredictable contents.
While IDP products have traditionally focused on these three types of documents, the aperture is widening and IDP now eyes an opportunity to process many types of unstructured data, of which a document is only one.
Think of unstructured data as information that is not already in a digital format that can be automatically processed. Currently IDP transforms documents such as paper invoices into digital formats that an AP or ERP system can automatically process. IDP’s continued high-speed growth will be boosted by its ability to process other types of unstructured data currently trapped in other non-document formats.
Innovators are building software that will read (or see) and understand unstructured data trapped in:
- Text streams (e.g., social media or web chat sessions)
- Static images (e.g., photographs or branch scans)
- Videos (e.g., social posts or CCTV film)
- Voice interactions (e.g., live talk or recorded messages)
The Technology Behind It All
Natural Language Processing (NLP) technology is already used to process unstructured documents, the third type of document in the focus of traditional IDP vendors. In a letter of complaint, NLP can automatically understand meaning from context, mimicking how a human reader or listener understands the complaint even though it’s not expressed in a highly structured way.
As NLP usage expands, it will be widely applied to social media or webchat text streams and to voice interactions that are converted to text stream formats. Specialist NLP software vendors are the new kids on the IDP block, well positioned to prosper as non-document-based text streams constitute a growing percentage of important but unstructured business data.
Computer Vision technology goes that extra step, escaping the confines of the written word and using visual inspection to understand content. Facial recognition is the most widely understood application to date, to static and moving video images. The big tech vendors invested early in Computer Vision technology. Google Vision, Amazon Rekognition, and Microsoft Azure’s Computer Vision provide on-demand analysis of static and moving images, packaged in increasingly easy-to-use interfaces.
IDP’s Future Impacts
What impact will these innovations within IDP have on our day-to-day world?
A poster-child example, often discussed and now increasingly feasible, is the car crash insurance process that decides a claim in seconds, not days or weeks, by exploiting a new generation of IDP to quickly gather from multiple sources and analyze the documents, photos, videos, and chat associated with the event. As IDP evolves, it brings more automation to bear in this and other business processes where understanding and processing unstructured data is crucial.
The bottom line? Technological innovation is making it increasingly feasible for businesses to unlock the business value of unstructured data. Whether this opportunity is acted upon, depends on the ambition of individual businesses.
