PDF OCR 필요 여부 일괄 검출: 기업 지식 베이스 베스트 프랙티스

The most common budget sink in enterprise knowledge base projects is not the vector database—it is blind OCR on every PDF. In a batch of three thousand legal contracts, half may be native PDFs exported from Word with intact text layers ready for RAG chunking; the other half are scans that truly need Batch OCR. Skipping PDF OCR Detection and OCR-ing everything wastes GPU cycles and injects garbage chunks that hurt Enterprise Knowledge Base recall.

This guide is for platform engineers and tech leads building or operating enterprise knowledge bases. It presents a practical three-layer detection pipeline: metadata and text-layer fast filters, image-coverage and character-density scoring, then sampled OCR validation for gray-zone files. Combined with Batch OCR queue design and governance, it answers one critical question: does this PDF actually need OCR?

Bottom line: detect → route → OCR-on-demand typically saves 40%–70% versus full-corpus OCR while keeping retrieval quality stable.

1. Why full OCR is the most expensive mistake

Many teams equate "PDF ingest" with "OCR everything." That made sense in 2018 when scans dominated; in 2026 native PDFs often exceed half of an Enterprise Knowledge Base: ERP exports, Confluence prints, code-review PDFs, e-signed contracts. Re-OCR-ing them wastes money, adds noise, and blocks launch SLAs when million-page jobs run for days.

Dual-layer PDFs are worse: a poor OCR text layer sits above the real scan. A naive page.get_text() returns gibberish that looks like text. Systematic PDF OCR Detection beats a single "empty text → OCR" rule.

If you already run vector search plus Agent memory (see our TencentDB Agent Memory 퀵스타트: 10분 AI 장기 기억), dirty OCR pollutes long-term memory—wrong characters in L1 atomic facts are harder to debug than missing documents. Detection is the first quality gate.

2. Three-layer PDF OCR Detection model

Each PDF lands in SKIP, OCR_REQUIRED, or REVIEW. Layers run fast to slow.

2.1 Layer 1: metadata fast filter (< 10 ms/file)

Use Apache Tika or pdfinfo for page count, encryption, Producer/Creator strings, and XMP markers. Encrypted files → REVIEW; zero pages → quarantine; scanner Creator without font embedding → lean OCR_REQUIRED.

2.2 Layer 2: text-layer quality (< 200 ms/page)

With PyMuPDF (fitz) or pdfplumber, compute chars_per_page, word_ratio, image_coverage, and font_count.

2.3 Layer 3: gray-zone sample OCR

Sample 1–2 pages from REVIEW, run fast OCR, compare to Layer 2 text via Levenshtein/Jaccard. Similarity < 0.3 → OCR_REQUIRED; > 0.8 → SKIP. Disable blind trust of embedded OCR layers in Tika PDFParserConfig.

3. Batch OCR pipeline architecture

Decouple detection and OCR:

1. Ingest: object storage + queue (Kafka/SQS);
2. Detector workers: output {file_id, tier, scores, hash};
3. Router: text_extract, ocr_queue, human_review;
4. OCR workers: OCRmyPDF or cloud Document AI;
5. Indexer: chunk, embed, push to vector + full-text search.

Batch OCR essentials: idempotency on content_hash + ocr_engine_version; priority queues; DLQ for timeouts; monitor REVIEW ratio—if > 25%, tune Layer 2 thresholds.

4. Hands-on: Python batch detection

#!/usr/bin/env python3
import csv, sys
from pathlib import Path
import fitz

def score_page(page):
    text = page.get_text("text") or ""
    chars = sum(1 for c in text if c.isprintable())
    blocks = page.get_text("dict")["blocks"]
    img_area = sum(b["width"] * b["height"] for b in blocks if b["type"] == 1)
    page_area = page.rect.width * page.rect.height or 1
    return chars, img_area / page_area

def classify_pdf(path):
    doc = fitz.open(path)
    total_chars, coverages = 0, []
    for page in doc:
        c, cov = score_page(page)
        total_chars += c
        coverages.append(cov)
    n = max(len(doc), 1)
    cpp = total_chars / n
    avg_cov = sum(coverages) / n
    if cpp > 400 and avg_cov < 0.08:
        tier = "SKIP"
    elif cpp < 30 and avg_cov > 0.75:
        tier = "OCR_REQUIRED"
    else:
        tier = "REVIEW"
    return {"file": path.name, "pages": n, "chars_per_page": round(cpp, 1),
            "image_coverage": round(avg_cov, 3), "tier": tier}

5. Enterprise Knowledge Base best practices

5.1 Upstream ingest rules

• Upload portal: declare native / scan / mixed;
• SHA-256 dedup with version pointers;
• Page limits and auto-split to avoid OCR OOM.

5.2 Quality gates

• SKIP: extract → language detect → chunk, 1% human readability sample;
• OCR_REQUIRED: sample CER, > 8% → human correction;
• Tag low-confidence chunks for rerank down-weighting.

5.3 Security

Run detection and Batch OCR inside VPC; DLP before cloud OCR APIs; store OCR text in the same permission domain as source PDFs.

5.4 Ops cadence

6. FAQ

스캔 PDF와 네이티브 PDF 검출 차이는?

네이티브 PDF는 추출 가능한 텍스트 레이어(font embedding + ToUnicode)—chars_per_page 높고 image_coverage 낮음. 스캔은 비트맵 래퍼에 빈/깨진 텍스트 레이어, image_coverage 거의 100%. 이중 레이어 PDF는 텍스트 품질과 이미지 비율 모두 필요.

Batch OCR은 CPU vs GPU?

일 5만 페이지 미만·Tesseract 위주면 멀티코어 CPU+큐로 충분. 일 20만 페이지 초과 또는 딥 OCR(PaddleOCR, TrOCR) 시 GPU 추가. 검출로 네이티브 PDF 60% 스킵이 GPU 무작정 추가보다 저렴.

검출 스크립트 실행 빈도?

적재 시 Layer 1–2 실시간(밀리초). 전체 라이브러리 Layer 3 샘플링 주 1회. 대규모 마이그레이션 후 전량 재스캔. content hash + 페이지 수로 중복 OCR 방지.

7. Summary

PDF OCR Detection separates native PDFs, scans, and gray zones; Batch OCR spends compute where it matters; Enterprise Knowledge Base governance keeps the system maintainable. Week 1: staging stats and threshold tuning; Week 2: queue + idempotent index on 1k sample PDFs; Week 3: quality gates and dashboards before full historical import.