marker_api/TODO.md

TODO: marker-api Local Deployment

Priority 1: Verify local GPU compute works

• PyTorch ROCm sees gfx1151 on host
• Ollama runs locally on port 11435 with ROCm
• Start marker-api container with rocm/pytorch base
• Verify /health shows torch_device: cuda
• Run convert_both.py — ensure non_llm outputs are produced

Priority 2: Fix LLM correction pipeline

• Remove Vega-specific HSA_OVERRIDE_GFX_VERSION from app.py ✅
• Confirm build_options() fix (rm use_llm dup) resolves Click warnings ✅
• Run convert_both.py and verify non_llm vs llm outputs differ
• Confirm LLM requests reach local ollama container

Priority 3: Hardening

• Add health check for LLM connectivity
• Ensure marker user can write to conversion_results directory
• Expose LLM metrics/status in /health endpoint

Known issues

• GPUs are considered "iGPU" by ollama (system memory, not VRAM)
• LD_LIBRARY_PATH in entrypoint may need /opt/rocm/lib from the rocm/pytorch image