marker_api/TODO.md

TODO: marker-api Local Deployment

Status — 2026-06-07

• ✅ Container builds + runs with rocm/pytorch base (ROCm 7.2, PyTorch 2.9.1)
• ✅ CPU-mode conversions work (OCR + markdown extraction via marker-pdf)
• ✅ Ollama container runs locally on port 11435 with gfx1151 GPU
• ❌ PyTorch ROCm memory allocation segfaults (kernel ABI mismatch: ROCm 6.3 user-space vs 7.2 driver)

Remaining Work

1. Fix GPU acceleration for PyTorch

• PyTorch 2.9.1 is compiled against ROCm 6.3, host kernel driver is ROCm 7.2 — HIP kernel launches segfault
• Options:
  • Install PyTorch built for ROCm 7.2 (needs Python 3.12; not available on Fedora 44 host)
  • Use PyTorch from the rocm/pytorch container's venv (built for ROCm 7.2.4) — also failed with Memory in use
  • Patch HIP runtime to match kernel driver
  • Wait for Fedora / PyTorch to ship ROCm 7.2-aligned builds

2. Pull models into Ollama

• deepseek-ocr model not found (may be a custom model name)
• No internet access to pull models from ollama.com
• Need to pre-cache models or use an alternative OCR backend

3. Fix test fixtures

• Most PDFs in test_files/enisa/ are actually HTML error pages (proxy blocked original downloads)
• Only enisa-nis360-2026.pdf, nis2-technical-implementation-guidance.pdf are real PDFs
• test/test-pdf.pdf (81 pp.) works correctly

4. LLM correction pipeline

• use_llm=false skip works correctly (confirmed via diagnostic logs)
• use_llm=true needs Ollama model availability + GPU compute for correction prompt
• Build_options use_llm double-parameter fix applied (removed duplicate default)

5. Performance

• CPU-mode: ~6 s/page (first page, including model loading)
• Multi-page PDFs will be slow without GPU acceleration
• Consider MODEL_DTYPE=bfloat16 or smaller OCR models