Batch PDF OCR Detection: korporativnaya baza znanij

The most common budget sink in enterprise knowledge base projects is not the vector database—it is blind OCR on every PDF. In a batch of three thousand legal contracts, half may be native PDFs exported from Word with intact text layers ready for RAG chunking; the other half are scans that truly need Batch OCR. Skipping PDF OCR Detection and OCR-ing everything wastes GPU cycles and injects garbage chunks that hurt Enterprise Knowledge Base recall.

This guide is for platform engineers and tech leads building or operating enterprise knowledge bases. It presents a practical three-layer detection pipeline: metadata and text-layer fast filters, image-coverage and character-density scoring, then sampled OCR validation for gray-zone files. Combined with Batch OCR queue design and governance, it answers one critical question: does this PDF actually need OCR?

Bottom line: detect → route → OCR-on-demand typically saves 40%–70% versus full-corpus OCR while keeping retrieval quality stable.

1. Why full OCR is the most expensive mistake

Many teams equate "PDF ingest" with "OCR everything." That made sense in 2018 when scans dominated; in 2026 native PDFs often exceed half of an Enterprise Knowledge Base: ERP exports, Confluence prints, code-review PDFs, e-signed contracts. Re-OCR-ing them wastes money, adds noise, and blocks launch SLAs when million-page jobs run for days.

Dual-layer PDFs are worse: a poor OCR text layer sits above the real scan. A naive page.get_text() returns gibberish that looks like text. Systematic PDF OCR Detection beats a single "empty text → OCR" rule.

If you already run vector search plus Agent memory (see our TencentDB Agent Memory: dolgovremennaya pamyat AI za 10 minut), dirty OCR pollutes long-term memory—wrong characters in L1 atomic facts are harder to debug than missing documents. Detection is the first quality gate.

2. Three-layer PDF OCR Detection model

Each PDF lands in SKIP, OCR_REQUIRED, or REVIEW. Layers run fast to slow.

2.1 Layer 1: metadata fast filter (< 10 ms/file)

Use Apache Tika or pdfinfo for page count, encryption, Producer/Creator strings, and XMP markers. Encrypted files → REVIEW; zero pages → quarantine; scanner Creator without font embedding → lean OCR_REQUIRED.

2.2 Layer 2: text-layer quality (< 200 ms/page)

With PyMuPDF (fitz) or pdfplumber, compute chars_per_page, word_ratio, image_coverage, and font_count.

2.3 Layer 3: gray-zone sample OCR

Sample 1–2 pages from REVIEW, run fast OCR, compare to Layer 2 text via Levenshtein/Jaccard. Similarity < 0.3 → OCR_REQUIRED; > 0.8 → SKIP. Disable blind trust of embedded OCR layers in Tika PDFParserConfig.

3. Batch OCR pipeline architecture

Decouple detection and OCR:

1. Ingest: object storage + queue (Kafka/SQS);
2. Detector workers: output {file_id, tier, scores, hash};
3. Router: text_extract, ocr_queue, human_review;
4. OCR workers: OCRmyPDF or cloud Document AI;
5. Indexer: chunk, embed, push to vector + full-text search.

Batch OCR essentials: idempotency on content_hash + ocr_engine_version; priority queues; DLQ for timeouts; monitor REVIEW ratio—if > 25%, tune Layer 2 thresholds.

4. Hands-on: Python batch detection

#!/usr/bin/env python3
import csv, sys
from pathlib import Path
import fitz

def score_page(page):
    text = page.get_text("text") or ""
    chars = sum(1 for c in text if c.isprintable())
    blocks = page.get_text("dict")["blocks"]
    img_area = sum(b["width"] * b["height"] for b in blocks if b["type"] == 1)
    page_area = page.rect.width * page.rect.height or 1
    return chars, img_area / page_area

def classify_pdf(path):
    doc = fitz.open(path)
    total_chars, coverages = 0, []
    for page in doc:
        c, cov = score_page(page)
        total_chars += c
        coverages.append(cov)
    n = max(len(doc), 1)
    cpp = total_chars / n
    avg_cov = sum(coverages) / n
    if cpp > 400 and avg_cov < 0.08:
        tier = "SKIP"
    elif cpp < 30 and avg_cov > 0.75:
        tier = "OCR_REQUIRED"
    else:
        tier = "REVIEW"
    return {"file": path.name, "pages": n, "chars_per_page": round(cpp, 1),
            "image_coverage": round(avg_cov, 3), "tier": tier}

5. Enterprise Knowledge Base best practices

5.1 Upstream ingest rules

• Upload portal: declare native / scan / mixed;
• SHA-256 dedup with version pointers;
• Page limits and auto-split to avoid OCR OOM.

5.2 Quality gates

• SKIP: extract → language detect → chunk, 1% human readability sample;
• OCR_REQUIRED: sample CER, > 8% → human correction;
• Tag low-confidence chunks for rerank down-weighting.

5.3 Security

Run detection and Batch OCR inside VPC; DLP before cloud OCR APIs; store OCR text in the same permission domain as source PDFs.

5.4 Ops cadence

6. FAQ

Skan vs nativnyy PDF v detekcii?

Nativnye PDF: izvlechaemyy tekst—vysokiy chars_per_page, nizkiy image_coverage. Skany: bitmapy s pustym tekstom, image_coverage ~100%. Dvuhsloynye PDF nuzhny kachestvo teksta i dolya izobrazheniy.

Batch OCR na CPU ili GPU?

Do ~50k stranits/den s Tesseract—mnogoyaderny CPU. Svishe ~200k ili deep OCR (PaddleOCR, TrOCR)—GPU. Propusk ~60% nativnykh PDF cherez detekciyu ekonomit bolshe chem GPU.

Kak chasto zapuskat detekciyu?

Layer 1–2 pri ingest (millisekundy). Layer 3 vyborochno raz v nedelyu; polnyy rescan posle migraciy. hash + stranicy izbegayut dublikata OCR.

7. Summary

PDF OCR Detection separates native PDFs, scans, and gray zones; Batch OCR spends compute where it matters; Enterprise Knowledge Base governance keeps the system maintainable. Week 1: staging stats and threshold tuning; Week 2: queue + idempotent index on 1k sample PDFs; Week 3: quality gates and dashboards before full historical import.