Document AI: How to Build It

How to use this page

This page is different from the rest of this section. The modules explain how things work. The build sheets list what to buy. This page walks you through building one product, start to finish.

It is written for someone who has not built this before. Every step says what goes in, what comes out, and how to plug it into the next step.

Nothing here is secret. All of it is doable by two engineers in a few weeks. The hard part is not any single step. It is knowing which eight steps there are, and what to connect to what.

What Document AI actually is

Document AI takes a picture of a document and gives you back fields you can put in a database.

Picture of a PAN card goes in. Out comes: name, PAN number, date of birth, father's name. Each one with a score saying how sure the system is.

People also call it ICR, which stands for Intelligent Character Recognition. Older OCR read printed text. ICR is meant to also read handwriting and understand layout. In practice the words get used loosely and nobody minds.

What it is not:

• It is not a decision. It reads a document. It does not tell you whether to open the account.
• It is not verification. Reading "Rajesh Kumar" off a card does not prove Rajesh Kumar exists or that this is his card. That is a different product.
• It is not one model. It is six or seven things in a row. This is the single biggest surprise for teams who thought they were buying an API.

Who buys it: banks, lenders, insurers, and anyone who currently has people typing things off scanned paper. The business case is almost always we have forty people doing data entry.

The whole journey, in one table

Here is the whole thing. Eight steps. Read this table once and the rest of the page is just detail.

Steps 3 and 4 are where the money goes. Steps 2, 6, 7 and 8 are where the product lives.

Steps 1 and 2 — getting a usable file

Step 1 — Get the file

What it does: the document gets from the customer to your server.

What comes out: a file on disk or in object storage, plus an id.

Connect it to step 2: do not pass the file around. Save it once, give it an id, and pass the id. Every later step reads from storage using that id. If you pass image bytes between services you will run out of memory on the day someone uploads a 40 MB scan.

Step 2 — Check it is usable

What it does: looks at the image and decides whether it is worth reading. Blurry, dark, cropped, upside down, or a photo of a screen.

import cv2

# This is the whole thing. It is not clever. It works.
def usable(path):
    img = cv2.imread(path)
    if img is None:
        return {"ok": False, "why": "not_an_image"}

    grey = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    h, w = grey.shape

    # Blur. Laplacian variance: sharp images have high variance, blurry ones low.
    blur = cv2.Laplacian(grey, cv2.CV_64F).var()

    # Brightness. A photo taken in the dark, or washed out by flash.
    bright = grey.mean()

    checks = {
        "too_small":  w < 800 or h < 500,
        "too_blurry": blur < 100,       # tune this on YOUR documents
        "too_dark":   bright < 50,
        "too_bright": bright > 205,
    }
    failed = [k for k, v in checks.items() if v]
    return {"ok": not failed, "why": failed,
            "blur": round(blur, 1), "brightness": round(bright, 1)}

# WHAT TO CHECK
# [ ] tune the thresholds on 200 of YOUR real documents, not on the numbers
#     above. A PAN card photographed indoors and a scanned bank statement have
#     completely different normal ranges
# [ ] tell the user WHICH check failed. "Please try again" makes people retry
#     the same bad photo. "Too dark - move to a window" gets you a good one
# [ ] let the user override after two failures and send it to human review.
#     A gate with no escape hatch becomes a customer who cannot open an account
# [ ] log the scores even when the image passes. When accuracy drops next month
#     you will want to know whether the images got worse

What comes out: a yes/no, and if no, which check failed.

Connect it to step 3: only files that pass go forward. Files that fail go back to the user with the specific reason. This is the cheapest step on the page and it removes more OCR cost than anything else you will do.

Steps 3 and 4 — what is it, and reading it

Step 3 — Work out what the document is

What it does: decides whether this is a PAN card, an Aadhaar, a passport, a salary slip or a bank statement.

You need this because step 4 and step 5 are different for each type. There is no general "read any document" that gives you clean fields.

What comes out: a document type, and a confidence score if you used B or C.

Connect it to step 4: the document type chooses which reader you call. PAN and Aadhaar go to an identity-document API. A bank statement goes to a table extractor. Sending a bank statement to an ID reader gets you nothing useful and still bills you.

Step 4 — Read the text

What it does: turns the picture into words, with a position on the page and a confidence for each one.

What comes out: a list of text pieces. Each has the text, a box saying where it sat on the page, and a confidence between 0 and 1.

Connect it to step 5: this is the most important handoff on the page, so it has its own section below.

The wiring between step 4 and step 5

Most guides stop after "call the OCR API". That is where the actual work starts, so this is the part we are going to be precise about.

Step 4 gives you words. Step 5 needs to produce fields. The contract between them is where teams lose a fortnight.

# STEP 4 GIVES YOU THIS. Every OCR engine returns roughly this shape, with
# different key names. Normalise to one shape IMMEDIATELY, in one small adapter
# per engine. Then nothing downstream knows or cares which engine you used --
# and swapping engines becomes a one-file change instead of a rewrite.

ocr_output = {
    "doc_id": "doc_8814",
    "engine": "textract-v1",          # keep this. You WILL need to know later.
    "pages": [{
        "page": 1,
        "words": [
            # text,            box (x, y, width, height as 0-1 fractions), score
            {"text": "INCOME",   "box": [0.10, 0.05, 0.08, 0.03], "score": 0.99},
            {"text": "TAX",      "box": [0.19, 0.05, 0.04, 0.03], "score": 0.99},
            {"text": "ABCDE1234F","box": [0.10, 0.44, 0.22, 0.04], "score": 0.97},
            {"text": "RAJESH",   "box": [0.10, 0.52, 0.12, 0.03], "score": 0.94},
            {"text": "KUMAR",    "box": [0.23, 0.52, 0.11, 0.03], "score": 0.95},
        ],
    }],
}

# STEP 5 MUST GIVE YOU THIS. Note what is carried through: the score, and WHERE
# on the page it came from. Without those two, steps 6, 7 and 8 cannot work.

fields_output = {
    "doc_id": "doc_8814",
    "doc_type": "pan_card",
    "fields": {
        "pan":  {"value": "ABCDE1234F", "score": 0.97,
                 "source": {"page": 1, "box": [0.10, 0.44, 0.22, 0.04]}},
        "name": {"value": "RAJESH KUMAR", "score": 0.94,   # LOWEST of its words
                 "source": {"page": 1, "box": [0.10, 0.52, 0.24, 0.03]}},
    },
    "missing": ["father_name", "dob"],      # ALWAYS list what you did not find
}

# WHAT TO CHECK
# [ ] boxes as fractions of the page (0-1), never pixels. Pixels break the
#     moment someone uploads the same document at a different resolution
# [ ] a joined field takes the LOWEST score of its parts, not the average.
#     "RAJESH" at 0.99 and "KUMAR" at 0.60 is a 0.60 name, not a 0.80 one.
#     Averaging confidence is how bad reads get through
# [ ] "missing" is a real list, not an absence of keys. "We could not find the
#     date of birth" and "we never looked for it" are different facts
# [ ] keep the box. When a human reviews this in step 7, you want to highlight
#     the exact spot on the image. Reviewers go three to five times faster when
#     they can see where the value came from
# [ ] keep the engine name and its version. When accuracy shifts next quarter,
#     the first question is whether the vendor changed the model
# [ ] normalise every engine into this ONE shape in an adapter. Do not let
#     Textract's key names leak into your database

The rule underneath all of that: carry the confidence and the position all the way through. They feel like debugging information at step 5. By step 7 they are the product, and by step 8 they are your evidence.

Steps 5 and 6 — fields, and checking them

Step 5 — Turn words into fields

What it does: decides which of those words is the name, which is the PAN number, which is the date of birth.

Connect it to step 6: pass the whole field object, with scores and boxes. Do not flatten it to plain strings here. Step 6 needs the scores and step 7 needs the boxes.

Step 6 — Check the fields are real

What it does: catches wrong values that the OCR was confident about. This is the step that separates a demo from a product.

A confidence score tells you how clearly the system saw the characters. It does not tell you whether the answer is right. An OCR engine can read a smudged 8 as a 3 with 0.98 confidence.

Connect it to step 7: each field now carries both a confidence and a validation result. Step 7 needs both, and they disagree more often than you would expect. A field can be read clearly and still be wrong; a field can be read poorly and still be correct.

Steps 7 and 8 — deciding, and keeping proof

Step 7 — Accept, review or reject

What it does: turns scores and checks into an actual decision about this document.

# Three outcomes. Never two. A system with only accept and reject will either
# reject good customers or accept bad data, and usually both.

def decide(fields, doc_type, rules):
    hard_fail, weak = [], []

    for name, f in fields.items():
        if name in rules["required"][doc_type]:
            if f["value"] is None:
                hard_fail.append((name, "missing"))
                continue
            if not f["valid"]:                    # from step 6
                hard_fail.append((name, f["invalid_reason"]))
                continue
            if f["score"] < rules["min_score"].get(name, 0.90):
                weak.append((name, f["score"]))

    # A failed CHECKSUM is different from a LOW SCORE. The first means the value
    # is wrong. The second means we are unsure. Do not merge them into one
    # "confidence" number -- you lose the ability to tell the user which it was.
    if hard_fail:
        return {"outcome": "review", "reason": "failed_validation",
                "detail": hard_fail, "show_boxes": True}
    if weak:
        return {"outcome": "review", "reason": "low_confidence",
                "detail": weak, "show_boxes": True}
    return {"outcome": "accept"}

# WHAT TO CHECK
# [ ] "reject" is almost never the right automatic outcome for a document that
#     a real customer uploaded. Route to REVIEW. Let a person reject
# [ ] the review screen shows the image with the box drawn on it, next to the
#     extracted value, in one editable field. Reviewers go 3-5x faster
# [ ] thresholds live in config, are versioned, and are approved. Changing a
#     threshold changes who gets accepted -- treat it as a control change
# [ ] measure the review queue every week. If it is growing faster than volume,
#     something upstream broke and nobody noticed
# [ ] measure the OVERRIDE RATE. If humans accept 95% of what you send them,
#     your thresholds are too tight and you are paying people to click yes
# [ ] every correction a reviewer makes is training data and a bug report.
#     Store the before and after. This is the highest-value data you will
#     generate and most teams throw it away

Connect it to step 8: the decision, the reason and the reviewer's edits all go into the record. Especially the edits.

Step 8 — Keep the evidence

What it does: stores what happened, so that in two years you can answer "why did you accept this document?"

This is the step that gets left until last and then never gets built. It is also the one that regulators and auditors ask about first.

What it costs

These figures were verified in May 2026 for Build Sheet 01 and are carried here unchanged. Cloud pricing moves. Check the vendor's calculator before you commit to a number in a business case.

Three versions you could build

The weekend version

Build: upload form → OpenCV quality check → ask the user what the document is → Textract AnalyzeID → format and checksum validation → anything uncertain goes to a spreadsheet a person looks at → store everything in one Postgres table.

You get: a working product. Genuinely. Two engineers, one weekend, and it handles real documents.

It breaks when: volume passes a few hundred a day, or the review spreadsheet gets more than one person using it.

The proper version — start here if you are serious

Build: everything above, plus — a real review interface that draws the box on the image → one adapter per OCR engine so you can switch → thresholds in versioned config → an append-only evidence table → corrections captured as training data → weekly numbers on queue size and override rate.

You get: something you can run a business on and show an auditor.

Cost shape: the API bill is small. The reviewers are the cost.

It breaks when: nothing, for a long time. This is the right build for almost everyone.

The enterprise version

Build: everything above, plus — self-hosted OCR for data that cannot leave your building → custom models per document type → a routing layer that picks the cheapest engine that will work → full audit and model governance.

Use when: millions of documents, or a data-residency rule that rules out the cloud APIs.

It breaks when: you build it too early. Most teams that start here spend six months and end up with the proper version anyway, having paid for the detour.

What goes wrong

Ranked by how often we see them, not by how serious they are.

Where to go next

This page covered one product. The same eight-step shape appears in others, and once you have built this you will recognise it.

Sources

Every figure, rule and date on this page, and where to check it. Entries are typed so you can see which numbers are primary-sourced and which are industry reporting — they are not equivalent, and treating them as if they were is how a confident wrong number gets repeated.