11 Commits

Author SHA1 Message Date
oval 6b01b5a991 added datalab marker client sdk as a submodule 2026-06-10 08:16:51 +02:00
oval cb7dfae9c9 Containerfile: fix font download path and remove submodule dependency
- Font installed at the correct settings.FONT_PATH location
  (os.path.dirname(settings.FONT_PATH)), not relative to marker package dir
- Remove marker/ submodule reference; marker-pdf now installed from GitHub
- Updated header comment and default ENV values for public-readiness
- Add comment with alternative local-build pip command
2026-06-08 12:44:32 +02:00
oval 33997778f6 Prepare for public git: .gitignore, .env, sanitized configs
- Add .gitignore excluding .env, __pycache__, test_files (copyrighted PDFs),
  session logs, marker/ submodule, and other generated/private files
- Add .env with local config (gitignored); expand .env.example with all
  supported environment variables
- docker-compose.yml: all hardcoded values now use default pattern
- Containerfile: install marker-pdf from GitHub instead of local submodule;
  fix AMD_COMPUTE default to false; sanitize default OLLAMA_HOST
- Remove marker/ submodule from git tracking (keep local checkout via .gitignore)
- Remove session transcript and copyrighted ENISA PDFs from git tracking
- Sanitize local IPs in README, kubernetes deployment, deploy-podman.sh
- Fix hardcoded paths in convert_both.py and compare_conversions.py
- Update requirements.txt to reference GitHub instead of local --editable
2026-06-08 12:29:43 +02:00
oval 3601d48f5a Fix SDK compatibility: poll response field names and file metadata format 2026-06-08 12:11:48 +02:00
oval 5883565b87 Implement Datalab-compatible API with async endpoints and deepseek OCR backend
- New endpoints: /api/v1/convert, /api/v1/extract, /api/v1/segment,
  /api/v1/ocr, /api/v1/table_rec, /api/v1/marker (deprecated),
  /api/v1/create-document, /api/v1/files/*, /api/v1/thumbnails
- Async submit-and-poll pattern matching Datalab API spec
- X-API-Key authentication via API_KEY env var
- Filesystem-based request store for multi-worker support
- OCR_BACKEND=deepseek mode: uses deepseek-ocr via ollama for OCR
  (prompt: <|grounding|>Free OCR.)
- build_options() now propagates output_format to parsed_opts
- Fixed docs page CSS curly brace conflict with .format()
- Renamed marker_endpoint_referece.md -> marker_endpoint_reference.md
- Containerfile: added poppler-utils, pandoc; 3 gunicorn workers
- gunicorn.conf: post_fork hook for background worker threads
2026-06-08 11:12:55 +02:00
oval df8d0d6b74 Finalize local deployment config: CPU mode, persistent cache, port 8001
- docker-compose.yml: default TORCH_DEVICE=cpu, PORT=8001, named volumes
  for model cache (marker-cache) and results (marker-results) with :Z flag
- convert_both.py: API_URL defaults to port 8001, overridable via env
- TODO.md: reflect final decisions (iGPU 30x slower than CPU,
  --privileged required for GPU, CPU mode is default)
2026-06-07 23:37:04 +02:00
oval d1658d29b1 Update TODO.md with current status (CPU mode works, GPU blocked by ROCm ABI) 2026-06-07 23:08:01 +02:00
oval 0c4bcdc7d2 Switch from Vega20 (gfx906) base to ROCm 7.2 for gfx1151; set local ollama :11435 2026-06-07 22:39:28 +02:00
oval 94003f7dda Remove use_llm default from build_options() to prevent Click duplicate parameter warning 2026-06-06 02:14:53 +02:00
oval b14406f59f Add diagnostic logging to LLM correction pipeline 2026-06-05 18:31:50 +02:00
oval 460fc0003b Initial marker-api benchmark setup: Containerfile, app.py, convert_both.py, compare_conversions.py 2026-06-05 13:26:25 +02:00