Document annotation

Document OCR Annotation for Tables, Forms and Layout Parsing

Labels that turn scanned statements, contracts and forms into structured data: every cell, field and line placed in the right order and linked to its meaning.

Start a Pilot Explore Services

Compatible workflows

Tools we can work with

Roboflow
SuperAnnotate
Darkhorse
Roboflow
SuperAnnotate
Darkhorse

TL;DR

Complex financial and legal tables are labelled cell by cell, including merged cells, multi-line headers and tables that run across pages.
Key-value parsing links each label to its value, even when they sit far apart, in different columns or on different lines.
Reading order is labelled explicitly for multi-column pages, footnotes, sidebars and stamps, so text is read the way a person reads it.
Multilingual and mixed-script documents get transcription rules per script, with right-to-left text and handwriting handled separately.

Layout

Layout regions and document structure

Before any text is read, a page has to be divided into what it is made of. The layout taxonomy is fixed per document type before production starts.

Region classes

Pages are divided into regions such as title, heading, paragraph, list, table, figure, caption, header, footer, page number, signature, stamp and handwritten note. Each region gets a box or polygon that follows the visible content, not the page margin. Rules state how to handle regions that touch, such as a caption sitting directly under a table, and how nested regions, like a list inside a table cell, are labelled.

Scan quality and artefacts

Skew, faint print, fax noise, punch holes and coffee stains are common in real archives. The guideline states when text is too degraded to transcribe and how it is marked, so the model is never trained on guessed characters presented as certain.

Box and polygon tightness rules are explained on our image annotation page.

Tables

Complex financial and legal tables

Tables are where most document models fail: balance sheets, transaction lists, fee schedules and legal schedules rarely follow a clean grid.

Cells, spans and headers

Each table is labelled as a structure: rows, columns and cells, with row and column spans for merged cells. Header cells are marked as headers, and multi-level headers keep their hierarchy so a value can be traced to every header above it. Empty cells are labelled as empty rather than skipped, because a missing cell shifts every value after it.

Financial tables bring their own rules: negative numbers in brackets, currency symbols in a header instead of the cell, subtotal and total rows, and footnote markers attached to figures. Each of these is written into the guideline with an example, so numbers are captured exactly as printed.

Tables across pages and without lines

Tables that continue on the next page are linked into one logical table, with repeated headers marked so they are not read as data. Borderless tables are labelled from alignment and spacing, with rules for wrapped text that spills into the next visual row.

Forms

Key-value pairs and form field parsing

Invoices, applications, contracts and statements carry most of their meaning in pairs: a label and its value.

Linking keys to values

Annotators mark the key text, the value text and the link between them. Keys can sit above, beside or below their values, and a single key can have several values, such as a list of account numbers. Checkboxes and radio buttons are labelled with their state, and values written by hand are flagged as handwriting.

For contracts and legal documents, parties, dates, amounts, governing law and clause references can be linked to a field schema you define, so extraction maps straight onto your data model.

Entity labelling and field schemas for the extracted text are covered on our text annotation page.

Reading order

Reading order for multi-column and mixed layouts

OCR that reads every line left to right across two columns produces nonsense. Reading order is labelled, not assumed.

Order rules

Each region and text line gets a position in the reading sequence. The guideline states how to order columns, sidebars, footnotes, captions, pull quotes and text inside tables, and which elements, such as page numbers and running headers, are left out of the main flow. Footnote markers are linked to their footnotes.

Languages

Multilingual OCR transcription and QA

Many document sets mix languages and scripts on one page, such as an English contract with local-language addresses or bilingual forms.

Script, direction and handwriting

Text lines are tagged with language and script. Right-to-left scripts are transcribed in logical order, and numbers embedded in them follow a written rule. Transcription keeps diacritics, ligatures and punctuation exactly as printed, and the guideline says how to handle characters that look alike across scripts. Handwriting is labelled as its own class and transcribed only where it is legible.

QA follows our two-stage process: annotators self-check, then an independent reviewer audits cell structure, key-value links, reading order and character-level transcription. Delivery can be JSON with regions, cells and links, COCO JSON for layout boxes, or the format your OCR pipeline expects.

Inspect Sample Deliverable & QA Rubric

Illustrative sample

// Illustrative sample, not client data
{
  "images": [{ "id": 7, "file_name": "statement_p2.png", "width": 1700, "height": 2200 }],
  "categories": [{ "id": 1, "name": "table" }, { "id": 2, "name": "table_cell" }, { "id": 3, "name": "key" }, { "id": 4, "name": "value" }],
  "annotations": [
    { "id": 70, "image_id": 7, "category_id": 1, "bbox": [120, 640, 1460, 820] },
    { "id": 71, "image_id": 7, "category_id": 2, "bbox": [120, 640, 480, 60],
      "attributes": { "row": 0, "col": 0, "row_span": 1, "col_span": 2, "header": true, "text": "Description" } },
    { "id": 80, "image_id": 7, "category_id": 3, "bbox": [120, 300, 210, 34], "attributes": { "text": "Account no.", "reading_order": 4 } },
    { "id": 81, "image_id": 7, "category_id": 4, "bbox": [360, 300, 300, 34], "attributes": { "text": "XX-0000-00", "key_id": 80, "reading_order": 5 } }
  ]
}

Stage 1 — Annotator self-check

Every object in the label set is tagged
Boxes/segments are tight to visible edges
Class names match the guideline exactly
Occlusion and truncation flags set

Stage 2 — Independent reviewer audit

Independent reviewer re-checks a sampled batch
Errors logged against the rubric by type
Batch returned if it misses the accuracy threshold
Edge cases added back to the guideline

Document OCR annotation FAQ

How do you label merged cells and multi-level headers?

Each cell carries its row and column position with row and column spans, and header cells keep their hierarchy. That lets every value be traced to all the headers above and beside it.

How is reading order labelled on multi-column pages?

Every region and line gets a position in the reading sequence under written rules for columns, sidebars, footnotes and captions. Page numbers and running headers can be excluded from the main flow.

Can you link keys to values that are far apart?

Yes. The key, the value and the link between them are labelled separately, so a value can sit below, beside or on another line from its key. One key can link to several values.

Do you handle multilingual and right-to-left documents?

Yes. Lines are tagged with language and script, right-to-left text is transcribed in logical order, and characters are kept exactly as printed. Handwriting is its own class and transcribed only where legible.

Send us the table your OCR gets wrong.

Start with a Free Calibration Pilot