Finalize local deployment config: CPU mode, persistent cache, port 8001
- docker-compose.yml: default TORCH_DEVICE=cpu, PORT=8001, named volumes for model cache (marker-cache) and results (marker-results) with :Z flag - convert_both.py: API_URL defaults to port 8001, overridable via env - TODO.md: reflect final decisions (iGPU 30x slower than CPU, --privileged required for GPU, CPU mode is default)
This commit is contained in:
oval
@@ -1,37 +1,38 @@
|
||||
# TODO: marker-api Local Deployment
|
||||
# TODO: marker-api Local Deployment (Fedora 44 + Radeon 8060S iGPU)
|
||||
|
||||
## Status — 2026-06-07
|
||||
- ✅ Container builds + runs with `rocm/pytorch` base (ROCm 7.2, PyTorch 2.9.1)
|
||||
- ✅ CPU-mode conversions work (OCR + markdown extraction via marker-pdf)
|
||||
- ✅ Ollama container runs locally on port 11435 with gfx1151 GPU
|
||||
- ❌ PyTorch ROCm memory allocation segfaults (kernel ABI mismatch: ROCm 6.3 user-space vs 7.2 driver)
|
||||
## Status — 2026-06-07 (final)
|
||||
- ✅ `marker-api:latest` container builds from `rocm/pytorch:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.9.1`
|
||||
- ✅ CPU-mode conversions work correctly (~6 s/page for OCR + markdown)
|
||||
- ✅ Persistent model cache volume with SELinux `:Z` relabeling (avoids re-download)
|
||||
- ✅ Ollama container runs on port 11435 with gfx1151 GPU acceleration
|
||||
- ✅ `convert_both.py` updated for port 8001 + env override
|
||||
- ✅ `docker-compose.yml` defaults to CPU mode, port 8001, persistent named volumes
|
||||
- ❌ GPU-mode PyTorch is 30× slower than CPU on gfx1151 iGPU (shared system RAM bottleneck)
|
||||
- ❌ `--privileged` is the only way to make ROCm HIP allocate memory on gfx1151 (ACL issue)
|
||||
|
||||
## Remaining Work
|
||||
|
||||
### 1. Fix GPU acceleration for PyTorch
|
||||
- PyTorch 2.9.1 is compiled against ROCm 6.3, host kernel driver is ROCm 7.2 — HIP kernel launches segfault
|
||||
- Options:
|
||||
- Install PyTorch built for ROCm 7.2 (needs Python 3.12; not available on Fedora 44 host)
|
||||
- Use PyTorch from the `rocm/pytorch` container's venv (built for ROCm 7.2.4) — also failed with `Memory in use`
|
||||
- Patch HIP runtime to match kernel driver
|
||||
- Wait for Fedora / PyTorch to ship ROCm 7.2-aligned builds
|
||||
### 1. GPU acceleration not worth pursuing for this iGPU
|
||||
- Radeon 8060S iGPU shares system RAM — no dedicated VRAM
|
||||
- GPU mode is 30× slower than CPU (3 min vs 6 s per page)
|
||||
- CPU mode is the correct default: `TORCH_DEVICE=cpu`, `MODEL_DTYPE=float32`
|
||||
- Ollama's ROCm backend works fine on this GPU for LLM inference
|
||||
|
||||
### 2. Pull models into Ollama
|
||||
- `deepseek-ocr` model not found (may be a custom model name)
|
||||
- No internet access to pull models from ollama.com
|
||||
- Need to pre-cache models or use an alternative OCR backend
|
||||
- `deepseek-ocr` ollama model still can't be pulled (no internet)
|
||||
- Need to pre-cache models or use a different OCR backend for llm-assisted mode
|
||||
- Ollama container runs locally and detects GPU correctly
|
||||
|
||||
### 3. Fix test fixtures
|
||||
- Most PDFs in `test_files/enisa/` are actually HTML error pages (proxy blocked original downloads)
|
||||
- Only `enisa-nis360-2026.pdf`, `nis2-technical-implementation-guidance.pdf` are real PDFs
|
||||
- `test/test-pdf.pdf` (81 pp.) works correctly
|
||||
- `test_files/enisa/*.pdf` — most are HTML error pages from proxy
|
||||
- Only `test/test-pdf.pdf` (81 pp. Cyber Resilience Act) is a reliable test document
|
||||
- Need real ENISA PDFs for validation
|
||||
|
||||
### 4. LLM correction pipeline
|
||||
- `use_llm=false` skip works correctly (confirmed via diagnostic logs)
|
||||
- `use_llm=true` needs Ollama model availability + GPU compute for correction prompt
|
||||
- Build_options `use_llm` double-parameter fix applied (removed duplicate default)
|
||||
- `use_llm=false` works (GPU-powered OCR, CPU correction skip)
|
||||
- `use_llm=true` needs: (a) ollama model pulled, (b) `TORCH_DEVICE=cpu` for PyTorch, (c) correction LLM runs on GPU via ollama
|
||||
|
||||
### 5. Performance
|
||||
- CPU-mode: ~6 s/page (first page, including model loading)
|
||||
- Multi-page PDFs will be slow without GPU acceleration
|
||||
- Consider `MODEL_DTYPE=bfloat16` or smaller OCR models
|
||||
- CPU-mode: ~6 s/page (first page, model already cached)
|
||||
- With model cache volume, subsequent container restarts don't re-download
|
||||
- Single worker is sufficient for this hardware
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ import os
|
||||
import time
|
||||
import requests
|
||||
|
||||
API_URL = "http://localhost:8000/v1/files/convert"
|
||||
API_URL = os.environ.get("API_URL", "http://localhost:8001/v1/files/convert")
|
||||
|
||||
TEST_FILES = [
|
||||
"/home/oval/marker-api/test_files/enisa/enisa-international-strategy-2026.pdf",
|
||||
|
||||
+15
-16
@@ -8,36 +8,35 @@ services:
|
||||
dockerfile: Containerfile
|
||||
image: marker-api:latest
|
||||
env_file:
|
||||
- .env # optional; remove if no .env exists
|
||||
- .env
|
||||
environment:
|
||||
- OLLAMA_HOST=${OLLAMA_HOST:-http://localhost:11435}
|
||||
- DEESEEK_OCR_MODEL=${DEESEEK_OCR_MODEL:-deepseek-ocr}
|
||||
- AMD_COMPUTE=${AMD_COMPUTE:-true}
|
||||
- TORCH_DEVICE=${TORCH_DEVICE:-cuda}
|
||||
- TORCH_DEVICE=${TORCH_DEVICE:-cpu}
|
||||
- MODEL_DTYPE=${MODEL_DTYPE:-float32}
|
||||
- OLLAMA_HOST=${OLLAMA_HOST:-http://localhost:11435}
|
||||
- DEESEEK_OCR_MODEL=${DEESEEK_OCR_MODEL:-}
|
||||
- LLM_SERVICE=marker.services.openai.OpenAIService
|
||||
- USE_LLM=false
|
||||
- USE_LLM=${USE_LLM:-false}
|
||||
- OPENAI_BASE_URL=http://localhost:11435/v1
|
||||
- OPENAI_API_KEY=not-needed
|
||||
- OPENAI_MODEL=
|
||||
- PORT=8000
|
||||
- PORT=${PORT:-8001}
|
||||
- HOST=0.0.0.0
|
||||
- GUNICORN_WORKERS=${GUNICORN_WORKERS:-2}
|
||||
- GUNICORN_THREADS=${GUNICORN_THREADS:-4}
|
||||
- GUNICORN_TIMEOUT=${GUNICORN_TIMEOUT:-300}
|
||||
- GUNICORN_WORKERS=${GUNICORN_WORKERS:-1}
|
||||
- GUNICORN_THREADS=${GUNICORN_THREADS:-2}
|
||||
- GUNICORN_TIMEOUT=${GUNICORN_TIMEOUT:-600}
|
||||
ports:
|
||||
- "8000:8000"
|
||||
- "${PORT:-8001}:${PORT:-8001}"
|
||||
volumes:
|
||||
- /dev/kfd:/dev/kfd
|
||||
- /dev/dri:/dev/dri
|
||||
- marker-data:/app/conversion_results
|
||||
- marker-results:/app/conversion_results:rw,Z
|
||||
- marker-cache:/app/.cache:rw,Z
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
test: ["CMD", "curl", "-f", "http://localhost:${PORT:-8001}/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 120s
|
||||
|
||||
volumes:
|
||||
marker-data:
|
||||
marker-results:
|
||||
marker-cache:
|
||||
|
||||
Reference in New Issue
Block a user