oval
df8d0d6b74
- docker-compose.yml: default TORCH_DEVICE=cpu, PORT=8001, named volumes for model cache (marker-cache) and results (marker-results) with :Z flag - convert_both.py: API_URL defaults to port 8001, overridable via env - TODO.md: reflect final decisions (iGPU 30x slower than CPU, --privileged required for GPU, CPU mode is default)
39 lines
1.9 KiB
Markdown
39 lines
1.9 KiB
Markdown
# TODO: marker-api Local Deployment (Fedora 44 + Radeon 8060S iGPU)
|
||
|
||
## Status — 2026-06-07 (final)
|
||
- ✅ `marker-api:latest` container builds from `rocm/pytorch:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.9.1`
|
||
- ✅ CPU-mode conversions work correctly (~6 s/page for OCR + markdown)
|
||
- ✅ Persistent model cache volume with SELinux `:Z` relabeling (avoids re-download)
|
||
- ✅ Ollama container runs on port 11435 with gfx1151 GPU acceleration
|
||
- ✅ `convert_both.py` updated for port 8001 + env override
|
||
- ✅ `docker-compose.yml` defaults to CPU mode, port 8001, persistent named volumes
|
||
- ❌ GPU-mode PyTorch is 30× slower than CPU on gfx1151 iGPU (shared system RAM bottleneck)
|
||
- ❌ `--privileged` is the only way to make ROCm HIP allocate memory on gfx1151 (ACL issue)
|
||
|
||
## Remaining Work
|
||
|
||
### 1. GPU acceleration not worth pursuing for this iGPU
|
||
- Radeon 8060S iGPU shares system RAM — no dedicated VRAM
|
||
- GPU mode is 30× slower than CPU (3 min vs 6 s per page)
|
||
- CPU mode is the correct default: `TORCH_DEVICE=cpu`, `MODEL_DTYPE=float32`
|
||
- Ollama's ROCm backend works fine on this GPU for LLM inference
|
||
|
||
### 2. Pull models into Ollama
|
||
- `deepseek-ocr` ollama model still can't be pulled (no internet)
|
||
- Need to pre-cache models or use a different OCR backend for llm-assisted mode
|
||
- Ollama container runs locally and detects GPU correctly
|
||
|
||
### 3. Fix test fixtures
|
||
- `test_files/enisa/*.pdf` — most are HTML error pages from proxy
|
||
- Only `test/test-pdf.pdf` (81 pp. Cyber Resilience Act) is a reliable test document
|
||
- Need real ENISA PDFs for validation
|
||
|
||
### 4. LLM correction pipeline
|
||
- `use_llm=false` works (GPU-powered OCR, CPU correction skip)
|
||
- `use_llm=true` needs: (a) ollama model pulled, (b) `TORCH_DEVICE=cpu` for PyTorch, (c) correction LLM runs on GPU via ollama
|
||
|
||
### 5. Performance
|
||
- CPU-mode: ~6 s/page (first page, model already cached)
|
||
- With model cache volume, subsequent container restarts don't re-download
|
||
- Single worker is sufficient for this hardware
|