From df8d0d6b742594cb5b74314de908a979946d85fa Mon Sep 17 00:00:00 2001 From: oval Date: Sun, 7 Jun 2026 23:37:04 +0200 Subject: [PATCH] Finalize local deployment config: CPU mode, persistent cache, port 8001 - docker-compose.yml: default TORCH_DEVICE=cpu, PORT=8001, named volumes for model cache (marker-cache) and results (marker-results) with :Z flag - convert_both.py: API_URL defaults to port 8001, overridable via env - TODO.md: reflect final decisions (iGPU 30x slower than CPU, --privileged required for GPU, CPU mode is default) --- TODO.md | 51 +++++++++++++++++++++++----------------------- convert_both.py | 2 +- docker-compose.yml | 31 ++++++++++++++-------------- 3 files changed, 42 insertions(+), 42 deletions(-) diff --git a/TODO.md b/TODO.md index 2ec1972..13a45d0 100644 --- a/TODO.md +++ b/TODO.md @@ -1,37 +1,38 @@ -# TODO: marker-api Local Deployment +# TODO: marker-api Local Deployment (Fedora 44 + Radeon 8060S iGPU) -## Status — 2026-06-07 -- ✅ Container builds + runs with `rocm/pytorch` base (ROCm 7.2, PyTorch 2.9.1) -- ✅ CPU-mode conversions work (OCR + markdown extraction via marker-pdf) -- ✅ Ollama container runs locally on port 11435 with gfx1151 GPU -- ❌ PyTorch ROCm memory allocation segfaults (kernel ABI mismatch: ROCm 6.3 user-space vs 7.2 driver) +## Status — 2026-06-07 (final) +- ✅ `marker-api:latest` container builds from `rocm/pytorch:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.9.1` +- ✅ CPU-mode conversions work correctly (~6 s/page for OCR + markdown) +- ✅ Persistent model cache volume with SELinux `:Z` relabeling (avoids re-download) +- ✅ Ollama container runs on port 11435 with gfx1151 GPU acceleration +- ✅ `convert_both.py` updated for port 8001 + env override +- ✅ `docker-compose.yml` defaults to CPU mode, port 8001, persistent named volumes +- ❌ GPU-mode PyTorch is 30× slower than CPU on gfx1151 iGPU (shared system RAM bottleneck) +- ❌ `--privileged` is the only way to make ROCm HIP allocate memory on gfx1151 (ACL issue) ## Remaining Work -### 1. Fix GPU acceleration for PyTorch -- PyTorch 2.9.1 is compiled against ROCm 6.3, host kernel driver is ROCm 7.2 — HIP kernel launches segfault -- Options: - - Install PyTorch built for ROCm 7.2 (needs Python 3.12; not available on Fedora 44 host) - - Use PyTorch from the `rocm/pytorch` container's venv (built for ROCm 7.2.4) — also failed with `Memory in use` - - Patch HIP runtime to match kernel driver - - Wait for Fedora / PyTorch to ship ROCm 7.2-aligned builds +### 1. GPU acceleration not worth pursuing for this iGPU +- Radeon 8060S iGPU shares system RAM — no dedicated VRAM +- GPU mode is 30× slower than CPU (3 min vs 6 s per page) +- CPU mode is the correct default: `TORCH_DEVICE=cpu`, `MODEL_DTYPE=float32` +- Ollama's ROCm backend works fine on this GPU for LLM inference ### 2. Pull models into Ollama -- `deepseek-ocr` model not found (may be a custom model name) -- No internet access to pull models from ollama.com -- Need to pre-cache models or use an alternative OCR backend +- `deepseek-ocr` ollama model still can't be pulled (no internet) +- Need to pre-cache models or use a different OCR backend for llm-assisted mode +- Ollama container runs locally and detects GPU correctly ### 3. Fix test fixtures -- Most PDFs in `test_files/enisa/` are actually HTML error pages (proxy blocked original downloads) -- Only `enisa-nis360-2026.pdf`, `nis2-technical-implementation-guidance.pdf` are real PDFs -- `test/test-pdf.pdf` (81 pp.) works correctly +- `test_files/enisa/*.pdf` — most are HTML error pages from proxy +- Only `test/test-pdf.pdf` (81 pp. Cyber Resilience Act) is a reliable test document +- Need real ENISA PDFs for validation ### 4. LLM correction pipeline -- `use_llm=false` skip works correctly (confirmed via diagnostic logs) -- `use_llm=true` needs Ollama model availability + GPU compute for correction prompt -- Build_options `use_llm` double-parameter fix applied (removed duplicate default) +- `use_llm=false` works (GPU-powered OCR, CPU correction skip) +- `use_llm=true` needs: (a) ollama model pulled, (b) `TORCH_DEVICE=cpu` for PyTorch, (c) correction LLM runs on GPU via ollama ### 5. Performance -- CPU-mode: ~6 s/page (first page, including model loading) -- Multi-page PDFs will be slow without GPU acceleration -- Consider `MODEL_DTYPE=bfloat16` or smaller OCR models +- CPU-mode: ~6 s/page (first page, model already cached) +- With model cache volume, subsequent container restarts don't re-download +- Single worker is sufficient for this hardware diff --git a/convert_both.py b/convert_both.py index b220982..fbb1e9d 100644 --- a/convert_both.py +++ b/convert_both.py @@ -4,7 +4,7 @@ import os import time import requests -API_URL = "http://localhost:8000/v1/files/convert" +API_URL = os.environ.get("API_URL", "http://localhost:8001/v1/files/convert") TEST_FILES = [ "/home/oval/marker-api/test_files/enisa/enisa-international-strategy-2026.pdf", diff --git a/docker-compose.yml b/docker-compose.yml index f2420c1..ca0e33c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -8,36 +8,35 @@ services: dockerfile: Containerfile image: marker-api:latest env_file: - - .env # optional; remove if no .env exists + - .env environment: - - OLLAMA_HOST=${OLLAMA_HOST:-http://localhost:11435} - - DEESEEK_OCR_MODEL=${DEESEEK_OCR_MODEL:-deepseek-ocr} - - AMD_COMPUTE=${AMD_COMPUTE:-true} - - TORCH_DEVICE=${TORCH_DEVICE:-cuda} + - TORCH_DEVICE=${TORCH_DEVICE:-cpu} - MODEL_DTYPE=${MODEL_DTYPE:-float32} + - OLLAMA_HOST=${OLLAMA_HOST:-http://localhost:11435} + - DEESEEK_OCR_MODEL=${DEESEEK_OCR_MODEL:-} - LLM_SERVICE=marker.services.openai.OpenAIService - - USE_LLM=false + - USE_LLM=${USE_LLM:-false} - OPENAI_BASE_URL=http://localhost:11435/v1 - OPENAI_API_KEY=not-needed - OPENAI_MODEL= - - PORT=8000 + - PORT=${PORT:-8001} - HOST=0.0.0.0 - - GUNICORN_WORKERS=${GUNICORN_WORKERS:-2} - - GUNICORN_THREADS=${GUNICORN_THREADS:-4} - - GUNICORN_TIMEOUT=${GUNICORN_TIMEOUT:-300} + - GUNICORN_WORKERS=${GUNICORN_WORKERS:-1} + - GUNICORN_THREADS=${GUNICORN_THREADS:-2} + - GUNICORN_TIMEOUT=${GUNICORN_TIMEOUT:-600} ports: - - "8000:8000" + - "${PORT:-8001}:${PORT:-8001}" volumes: - - /dev/kfd:/dev/kfd - - /dev/dri:/dev/dri - - marker-data:/app/conversion_results + - marker-results:/app/conversion_results:rw,Z + - marker-cache:/app/.cache:rw,Z restart: unless-stopped healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + test: ["CMD", "curl", "-f", "http://localhost:${PORT:-8001}/health"] interval: 30s timeout: 10s retries: 3 start_period: 120s volumes: - marker-data: + marker-results: + marker-cache: