oval
38 lines
1.8 KiB
Markdown
38 lines
1.8 KiB
Markdown
# TODO: marker-api Local Deployment
|
|
|
|
## Status — 2026-06-07
|
|
- ✅ Container builds + runs with `rocm/pytorch` base (ROCm 7.2, PyTorch 2.9.1)
|
|
- ✅ CPU-mode conversions work (OCR + markdown extraction via marker-pdf)
|
|
- ✅ Ollama container runs locally on port 11435 with gfx1151 GPU
|
|
- ❌ PyTorch ROCm memory allocation segfaults (kernel ABI mismatch: ROCm 6.3 user-space vs 7.2 driver)
|
|
|
|
## Remaining Work
|
|
|
|
### 1. Fix GPU acceleration for PyTorch
|
|
- PyTorch 2.9.1 is compiled against ROCm 6.3, host kernel driver is ROCm 7.2 — HIP kernel launches segfault
|
|
- Options:
|
|
- Install PyTorch built for ROCm 7.2 (needs Python 3.12; not available on Fedora 44 host)
|
|
- Use PyTorch from the `rocm/pytorch` container's venv (built for ROCm 7.2.4) — also failed with `Memory in use`
|
|
- Patch HIP runtime to match kernel driver
|
|
- Wait for Fedora / PyTorch to ship ROCm 7.2-aligned builds
|
|
|
|
### 2. Pull models into Ollama
|
|
- `deepseek-ocr` model not found (may be a custom model name)
|
|
- No internet access to pull models from ollama.com
|
|
- Need to pre-cache models or use an alternative OCR backend
|
|
|
|
### 3. Fix test fixtures
|
|
- Most PDFs in `test_files/enisa/` are actually HTML error pages (proxy blocked original downloads)
|
|
- Only `enisa-nis360-2026.pdf`, `nis2-technical-implementation-guidance.pdf` are real PDFs
|
|
- `test/test-pdf.pdf` (81 pp.) works correctly
|
|
|
|
### 4. LLM correction pipeline
|
|
- `use_llm=false` skip works correctly (confirmed via diagnostic logs)
|
|
- `use_llm=true` needs Ollama model availability + GPU compute for correction prompt
|
|
- Build_options `use_llm` double-parameter fix applied (removed duplicate default)
|
|
|
|
### 5. Performance
|
|
- CPU-mode: ~6 s/page (first page, including model loading)
|
|
- Multi-page PDFs will be slow without GPU acceleration
|
|
- Consider `MODEL_DTYPE=bfloat16` or smaller OCR models
|