oval/marker_api
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Finalize local deployment config: CPU mode, persistent cache, port 8001

Browse Source 
- docker-compose.yml: default TORCH_DEVICE=cpu, PORT=8001, named volumes
  for model cache (marker-cache) and results (marker-results) with :Z flag
- convert_both.py: API_URL defaults to port 8001, overridable via env
- TODO.md: reflect final decisions (iGPU 30x slower than CPU,
  --privileged required for GPU, CPU mode is default)
This commit is contained in:
oval
2026-06-07 23:37:04 +02:00
parent d1658d29b1
commit df8d0d6b74
3 changed files with 42 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TODO.md
+26 -25
View File
@@ -1,37 +1,38 @@
# TODO: marker-api Local Deployment # TODO: marker-api Local Deployment (Fedora 44 + Radeon 8060S iGPU)
## Status — 2026-06-07 ## Status — 2026-06-07 (final)
- ✅ Container builds + runs with `rocm/pytorch` base (ROCm 7.2, PyTorch 2.9.1) - ✅ `marker-api:latest` container builds from `rocm/pytorch:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.9.1`
- ✅ CPU-mode conversions work (OCR + markdown extraction via marker-pdf) - ✅ CPU-mode conversions work correctly (~6 s/page for OCR + markdown)
- ✅ Ollama container runs locally on port 11435 with gfx1151 GPU - ✅ Persistent model cache volume with SELinux `:Z` relabeling (avoids re-download)
- ❌ PyTorch ROCm memory allocation segfaults (kernel ABI mismatch: ROCm 6.3 user-space vs 7.2 driver) - ✅ Ollama container runs on port 11435 with gfx1151 GPU acceleration
- ✅ `convert_both.py` updated for port 8001 + env override
- ✅ `docker-compose.yml` defaults to CPU mode, port 8001, persistent named volumes
- ❌ GPU-mode PyTorch is 30× slower than CPU on gfx1151 iGPU (shared system RAM bottleneck)
- ❌ `--privileged` is the only way to make ROCm HIP allocate memory on gfx1151 (ACL issue)
## Remaining Work ## Remaining Work
### 1. Fix GPU acceleration for PyTorch ### 1. GPU acceleration not worth pursuing for this iGPU
- PyTorch 2.9.1 is compiled against ROCm 6.3, host kernel driver is ROCm 7.2 — HIP kernel launches segfault - Radeon 8060S iGPU shares system RAM — no dedicated VRAM
- Options: - GPU mode is 30× slower than CPU (3 min vs 6 s per page)
- Install PyTorch built for ROCm 7.2 (needs Python 3.12; not available on Fedora 44 host) - CPU mode is the correct default: `TORCH_DEVICE=cpu`, `MODEL_DTYPE=float32`
- Use PyTorch from the `rocm/pytorch` container's venv (built for ROCm 7.2.4) — also failed with `Memory in use` - Ollama's ROCm backend works fine on this GPU for LLM inference
- Patch HIP runtime to match kernel driver
- Wait for Fedora / PyTorch to ship ROCm 7.2-aligned builds
### 2. Pull models into Ollama ### 2. Pull models into Ollama
- `deepseek-ocr` model not found (may be a custom model name) - `deepseek-ocr` ollama model still can't be pulled (no internet)
- No internet access to pull models from ollama.com - Need to pre-cache models or use a different OCR backend for llm-assisted mode
- Need to pre-cache models or use an alternative OCR backend - Ollama container runs locally and detects GPU correctly
### 3. Fix test fixtures ### 3. Fix test fixtures
- Most PDFs in `test_files/enisa/` are actually HTML error pages (proxy blocked original downloads) - `test_files/enisa/*.pdf` — most are HTML error pages from proxy
- Only `enisa-nis360-2026.pdf`, `nis2-technical-implementation-guidance.pdf` are real PDFs - Only `test/test-pdf.pdf` (81 pp. Cyber Resilience Act) is a reliable test document
- `test/test-pdf.pdf` (81 pp.) works correctly - Need real ENISA PDFs for validation
### 4. LLM correction pipeline ### 4. LLM correction pipeline
- `use_llm=false` skip works correctly (confirmed via diagnostic logs) - `use_llm=false` works (GPU-powered OCR, CPU correction skip)
- `use_llm=true` needs Ollama model availability + GPU compute for correction prompt - `use_llm=true` needs: (a) ollama model pulled, (b) `TORCH_DEVICE=cpu` for PyTorch, (c) correction LLM runs on GPU via ollama
- Build_options `use_llm` double-parameter fix applied (removed duplicate default)
### 5. Performance ### 5. Performance
- CPU-mode: ~6 s/page (first page, including model loading) - CPU-mode: ~6 s/page (first page, model already cached)
- Multi-page PDFs will be slow without GPU acceleration - With model cache volume, subsequent container restarts don't re-download
- Consider `MODEL_DTYPE=bfloat16` or smaller OCR models - Single worker is sufficient for this hardware
convert_both.py
+1 -1
View File
@@ -4,7 +4,7 @@ import os
import time import time
import requests import requests
API_URL = "http://localhost:8000/v1/files/convert" API_URL = os.environ.get("API_URL", "http://localhost:8001/v1/files/convert")
TEST_FILES = [ TEST_FILES = [
"/home/oval/marker-api/test_files/enisa/enisa-international-strategy-2026.pdf", "/home/oval/marker-api/test_files/enisa/enisa-international-strategy-2026.pdf",
docker-compose.yml
+15 -16
View File
@@ -8,36 +8,35 @@ services:
dockerfile: Containerfile dockerfile: Containerfile
image: marker-api:latest image: marker-api:latest
env_file: env_file:
- .env # optional; remove if no .env exists - .env
environment: environment:
- OLLAMA_HOST=${OLLAMA_HOST:-http://localhost:11435} - TORCH_DEVICE=${TORCH_DEVICE:-cpu}
- DEESEEK_OCR_MODEL=${DEESEEK_OCR_MODEL:-deepseek-ocr}
- AMD_COMPUTE=${AMD_COMPUTE:-true}
- TORCH_DEVICE=${TORCH_DEVICE:-cuda}
- MODEL_DTYPE=${MODEL_DTYPE:-float32} - MODEL_DTYPE=${MODEL_DTYPE:-float32}
- OLLAMA_HOST=${OLLAMA_HOST:-http://localhost:11435}
- DEESEEK_OCR_MODEL=${DEESEEK_OCR_MODEL:-}
- LLM_SERVICE=marker.services.openai.OpenAIService - LLM_SERVICE=marker.services.openai.OpenAIService
- USE_LLM=false - USE_LLM=${USE_LLM:-false}
- OPENAI_BASE_URL=http://localhost:11435/v1 - OPENAI_BASE_URL=http://localhost:11435/v1
- OPENAI_API_KEY=not-needed - OPENAI_API_KEY=not-needed
- OPENAI_MODEL= - OPENAI_MODEL=
- PORT=8000 - PORT=${PORT:-8001}
- HOST=0.0.0.0 - HOST=0.0.0.0
- GUNICORN_WORKERS=${GUNICORN_WORKERS:-2} - GUNICORN_WORKERS=${GUNICORN_WORKERS:-1}
- GUNICORN_THREADS=${GUNICORN_THREADS:-4} - GUNICORN_THREADS=${GUNICORN_THREADS:-2}
- GUNICORN_TIMEOUT=${GUNICORN_TIMEOUT:-300} - GUNICORN_TIMEOUT=${GUNICORN_TIMEOUT:-600}
ports: ports:
- "8000:8000" - "${PORT:-8001}:${PORT:-8001}"
volumes: volumes:
- /dev/kfd:/dev/kfd - marker-results:/app/conversion_results:rw,Z
- /dev/dri:/dev/dri - marker-cache:/app/.cache:rw,Z
- marker-data:/app/conversion_results
restart: unless-stopped restart: unless-stopped
healthcheck: healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"] test: ["CMD", "curl", "-f", "http://localhost:${PORT:-8001}/health"]
interval: 30s interval: 30s
timeout: 10s timeout: 10s
retries: 3 retries: 3
start_period: 120s start_period: 120s
volumes: volumes:
marker-data: marker-results:
marker-cache: