first commit
This commit is contained in:
@@ -0,0 +1,21 @@
|
||||
# Agentic Research App Configuration
|
||||
OLLAMA_URL=http://10.0.1.127:11434
|
||||
GPT_OSS_MODEL=gpt-oss:20b
|
||||
MARKER_API_URL=http://localhost:8001
|
||||
OCR_URL=http://10.0.1.127:11434
|
||||
DEEPSEEK_OCR_URL=http://10.0.1.127:11434
|
||||
DEEPSEEK_OCR_MODEL=deepseek-ocr
|
||||
|
||||
# PostgreSQL / pgvector
|
||||
DB_HOST=localhost
|
||||
DB_PORT=5432
|
||||
DB_NAME=research
|
||||
DB_USER=research
|
||||
DB_PASSWORD=research123
|
||||
|
||||
# App
|
||||
APP_HOST=0.0.0.0
|
||||
APP_PORT=8000
|
||||
WORKSPACE_DIR=/home/oval/Projects/agentic/workspace
|
||||
DOCUMENTS_DIR=/home/oval/Projects/agentic/workspace/documents
|
||||
VECTOR_DIM=1536
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
*.pyc
|
||||
__pycache__/
|
||||
venv/
|
||||
.env
|
||||
workspace/
|
||||
marker_output/
|
||||
*.egg-info/
|
||||
dist/
|
||||
build/
|
||||
.eggs/
|
||||
node_modules/
|
||||
@@ -0,0 +1,3 @@
|
||||
[submodule "marker-api"]
|
||||
path = marker-api
|
||||
url = https://github.com/VikParuchuri/marker.git
|
||||
@@ -0,0 +1,107 @@
|
||||
---
|
||||
name: document-triage
|
||||
description: Scope a document research task, prioritize sources, and define an evidence-driven reading plan
|
||||
license: MIT
|
||||
compatibility: opencode
|
||||
metadata:
|
||||
audience: analysts
|
||||
workflow: research
|
||||
stage: triage
|
||||
---
|
||||
|
||||
## What I do
|
||||
|
||||
- Define the research objective in precise terms
|
||||
- Break the task into answerable sub-questions
|
||||
- Identify which documents are likely primary, secondary, or low-value
|
||||
- Propose a reading order based on expected relevance
|
||||
- Highlight likely evidence types such as requirements, decisions, risks, assumptions, and timelines
|
||||
- Create a focused plan before deep reading begins
|
||||
|
||||
## When to use me
|
||||
|
||||
Use this skill at the start of a document-based research task.
|
||||
|
||||
Examples:
|
||||
- You have a folder of policies, specs, meeting notes, or reports and need a systematic approach
|
||||
- You need to answer a question using internal documentation
|
||||
- You need to map a large corpus before extracting evidence
|
||||
- You want to avoid reading everything in full
|
||||
|
||||
## Inputs I expect
|
||||
|
||||
Provide as many of these as available:
|
||||
|
||||
- Research goal or question
|
||||
- List of documents, filenames, or brief descriptions
|
||||
- Any constraints such as deadline, topic boundaries, time period, or required output
|
||||
- Any known "must-cover" documents
|
||||
- Preferred output format if already defined
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Restate the research objective in one sentence
|
||||
2. Convert the objective into focused sub-questions
|
||||
3. Classify available documents by likely value:
|
||||
- Primary source
|
||||
- Supporting source
|
||||
- Background source
|
||||
- Likely irrelevant
|
||||
4. Identify expected signal in each source:
|
||||
- Requirements
|
||||
- Decisions
|
||||
- Responsibilities
|
||||
- Risks
|
||||
- Dates or milestones
|
||||
- Definitions
|
||||
- Exceptions
|
||||
5. Recommend an efficient reading order
|
||||
6. Define extraction criteria for the next stage
|
||||
7. Flag ambiguities, missing sources, and review risks
|
||||
|
||||
## Output format
|
||||
|
||||
Produce these sections:
|
||||
|
||||
### Research Objective
|
||||
A single concise statement of the exact question to answer.
|
||||
|
||||
### Sub-questions
|
||||
A numbered list of focused questions the research should resolve.
|
||||
|
||||
### Source Triage
|
||||
For each document:
|
||||
- Document name
|
||||
- Classification
|
||||
- Why it matters
|
||||
- What to look for
|
||||
|
||||
### Reading Plan
|
||||
A prioritized reading sequence with rationale.
|
||||
|
||||
### Extraction Criteria
|
||||
A checklist of evidence worth capturing in the next phase.
|
||||
|
||||
### Risks and Gaps
|
||||
Explicitly list missing documents, unclear scope, and likely blind spots.
|
||||
|
||||
## Quality bar
|
||||
|
||||
- Be specific, not generic
|
||||
- Prefer primary documents over commentary
|
||||
- Do not assume a document is authoritative unless the user says so
|
||||
- Separate fact-finding from interpretation
|
||||
- Explicitly note what cannot be determined yet
|
||||
|
||||
## Do not
|
||||
|
||||
- Do not summarize the documents in depth at this stage
|
||||
- Do not invent document contents
|
||||
- Do not claim conclusions before evidence extraction
|
||||
- Do not merge assumptions with facts
|
||||
|
||||
## Hand-off to companion skills
|
||||
|
||||
After this skill, use:
|
||||
- `evidence-extraction` to pull structured findings
|
||||
- `research-synthesis` to produce a final answer, brief, matrix, or report
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
name: evidence-extraction
|
||||
description: Extract structured evidence from documents with traceability, quotations, and confidence markers
|
||||
license: MIT
|
||||
compatibility: opencode
|
||||
metadata:
|
||||
audience: analysts
|
||||
workflow: research
|
||||
stage: extraction
|
||||
---
|
||||
|
||||
## What I do
|
||||
|
||||
- Read documents and extract evidence relevant to a defined research question
|
||||
- Separate direct evidence from interpretation
|
||||
- Capture exact quotations or precise paraphrases
|
||||
- Record document references for every extracted item
|
||||
- Structure findings so they can be audited and synthesized later
|
||||
- Mark uncertain or ambiguous findings for review
|
||||
|
||||
## When to use me
|
||||
|
||||
Use this after the research task has been scoped and prioritized.
|
||||
|
||||
Examples:
|
||||
- You need requirements, decisions, obligations, controls, or action items extracted from documents
|
||||
- You need a traceable evidence table
|
||||
- You need to compare what multiple documents say about the same issue
|
||||
- You need defensible notes before writing a summary or conclusion
|
||||
|
||||
## Inputs I expect
|
||||
|
||||
Provide:
|
||||
- The research objective or sub-question
|
||||
- One or more documents or excerpts
|
||||
- Optional triage output from `document-triage`
|
||||
- Optional extraction schema if the project already has one
|
||||
|
||||
## Core extraction rules
|
||||
|
||||
For every finding, preserve:
|
||||
|
||||
- Finding ID
|
||||
- Topic or theme
|
||||
- Evidence type
|
||||
- Source document
|
||||
- Section, heading, page, or paragraph reference if available
|
||||
- Extracted text
|
||||
- Short analyst note
|
||||
- Confidence level
|
||||
- Review flag where needed
|
||||
|
||||
Evidence types may include:
|
||||
- Requirement
|
||||
- Decision
|
||||
- Risk
|
||||
- Assumption
|
||||
- Responsibility
|
||||
- Constraint
|
||||
- Definition
|
||||
- Date or milestone
|
||||
- Dependency
|
||||
- Open issue
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Read the objective and extraction criteria
|
||||
2. Scan the document for relevant passages
|
||||
3. Extract only evidence tied to the research question
|
||||
4. Distinguish:
|
||||
- Direct statement
|
||||
- Implied interpretation
|
||||
- Missing information
|
||||
5. Attach traceable references for each item
|
||||
6. Normalize wording without losing meaning
|
||||
7. Flag ambiguity, contradiction, or incomplete support
|
||||
|
||||
## Output format
|
||||
|
||||
Use a structured table like this:
|
||||
|
||||
| # | Topic | Evidence Type | Description | Document Reference | Extracted Evidence | Analyst Note | Confidence | Review Needed |
|
||||
|---|---|----|-----------|-----|--------------------|------|----|---------|------|---------|-------|
|
||||
|
||||
Where:
|
||||
- **Description** is concise and factual
|
||||
- **Document Reference** is precise enough to locate the source again
|
||||
- **Extracted Evidence** is a quote or faithful paraphrase
|
||||
- **Analyst Note** explains relevance without overstating certainty
|
||||
- **Confidence** is one of: High, Medium, Low
|
||||
- **Review Needed** is `Yes` when ambiguity remains
|
||||
|
||||
## Comparison mode
|
||||
|
||||
If multiple documents cover the same topic, add a second table:
|
||||
|
||||
| Topic | Source A | Source B | Agreement | Conflict | Notes |
|
||||
|------|----------|----------|----------|----------|----|
|
||||
|
||||
Use this to identify contradictions, differences in wording, or missing alignment.
|
||||
|
||||
## Quality bar
|
||||
|
||||
- Every claim must be traceable to a source
|
||||
- Keep extraction atomic: one row per distinct finding
|
||||
- Preserve important qualifiers such as "must", "should", "may", "unless", and exceptions
|
||||
- Mark uncertainty explicitly instead of smoothing it over
|
||||
- Prefer exact citations over memory-based summaries
|
||||
|
||||
## Do not
|
||||
|
||||
- Do not write the final conclusion here
|
||||
- Do not collapse multiple findings into one vague row
|
||||
- Do not omit document references
|
||||
- Do not present interpretation as if it were a quote
|
||||
- Do not silently resolve contradictions
|
||||
|
||||
## Hand-off to companion skills
|
||||
|
||||
After this skill, use:
|
||||
- `research-synthesis` to turn extracted evidence into a conclusion, report, briefing, matrix, or recommendation
|
||||
@@ -0,0 +1,131 @@
|
||||
---
|
||||
name: research-synthesis
|
||||
description: Synthesize extracted document evidence into a clear, traceable conclusion, brief, or decision-ready report
|
||||
license: MIT
|
||||
compatibility: opencode
|
||||
metadata:
|
||||
audience: analysts
|
||||
workflow: research
|
||||
stage: synthesis
|
||||
---
|
||||
|
||||
## What I do
|
||||
|
||||
- Synthesize extracted evidence into a coherent answer
|
||||
- Build conclusions that remain traceable to source material
|
||||
- Identify consensus, conflicts, gaps, and unresolved questions
|
||||
- Produce outputs such as briefs, summaries, comparison matrices, decision memos, and research reports
|
||||
- Keep factual findings separate from recommendations or interpretation
|
||||
|
||||
## When to use me
|
||||
|
||||
Use this after evidence has been extracted and structured.
|
||||
|
||||
Examples:
|
||||
- You need a final answer to a document-based research question
|
||||
- You need an executive summary grounded in source material
|
||||
- You need a position paper or internal brief
|
||||
- You need a gap analysis or comparison across documents
|
||||
- You need a recommendation with explicit evidence support
|
||||
|
||||
## Inputs I expect
|
||||
|
||||
Provide:
|
||||
- Research objective
|
||||
- Extracted evidence table or notes
|
||||
- Optional comparison table
|
||||
- Desired output type, such as:
|
||||
- summary
|
||||
- brief
|
||||
- memo
|
||||
- report
|
||||
- gap analysis
|
||||
- comparison matrix
|
||||
- recommendation
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Restate the question being answered
|
||||
2. Group evidence by theme or sub-question
|
||||
3. Identify:
|
||||
- Supported conclusions
|
||||
- Partial support
|
||||
- Contradictions
|
||||
- Missing evidence
|
||||
4. Draft findings with explicit traceability
|
||||
5. Separate:
|
||||
- Facts from documents
|
||||
- Interpretation
|
||||
- Recommendations
|
||||
6. Produce a final structured output fit for the requested audience
|
||||
7. End with known limitations and review points
|
||||
|
||||
## Output modes
|
||||
|
||||
### A) Research Brief
|
||||
Use for concise decision support.
|
||||
|
||||
Structure:
|
||||
- Question
|
||||
- Short answer
|
||||
- Key findings
|
||||
- Conflicting evidence
|
||||
- Gaps
|
||||
- Recommended next step
|
||||
|
||||
### B) Research Report
|
||||
Use for fuller analytical output.
|
||||
|
||||
Structure:
|
||||
- Objective
|
||||
- Scope
|
||||
- Method
|
||||
- Findings by theme
|
||||
- Evidence-backed conclusion
|
||||
- Known gaps and limitations
|
||||
- Appendix with source references
|
||||
|
||||
### C) Gap Analysis
|
||||
Use when comparing required versus documented state.
|
||||
|
||||
Structure:
|
||||
- Assessment topic
|
||||
- Evidence found
|
||||
- Gap
|
||||
- Impact
|
||||
- Confidence
|
||||
- Review needed
|
||||
|
||||
## Traceability rule
|
||||
|
||||
For every substantive conclusion, include a source anchor such as:
|
||||
- document name
|
||||
- section or page
|
||||
- extracted finding number if available
|
||||
|
||||
If traceability is weak, say so explicitly.
|
||||
|
||||
## Quality bar
|
||||
|
||||
- Conclusions must follow from the extracted evidence
|
||||
- Contradictions must be surfaced, not hidden
|
||||
- Distinguish "document says" from "my recommendation"
|
||||
- State limits of the available corpus
|
||||
- Prefer precise wording over polished vagueness
|
||||
|
||||
## Do not
|
||||
|
||||
- Do not introduce facts that were not extracted
|
||||
- Do not overclaim certainty
|
||||
- Do not bury disagreements between documents
|
||||
- Do not produce recommendations without showing the evidence basis
|
||||
- Do not omit limitations
|
||||
|
||||
## Companion skill sequence
|
||||
|
||||
Recommended sequence:
|
||||
1. `document-triage`
|
||||
2. `evidence-extraction`
|
||||
3. `research-synthesis`
|
||||
|
||||
Use all three together for rigorous document-based research with clear traceability.
|
||||
@@ -0,0 +1,9 @@
|
||||
Collecting Pillow
|
||||
Downloading pillow-12.2.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (8.8 kB)
|
||||
Downloading pillow-12.2.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (7.1 MB)
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.1/7.1 MB 20.7 MB/s 0:00:00
|
||||
Installing collected packages: Pillow
|
||||
Successfully installed Pillow-12.2.0
|
||||
|
||||
[notice] A new release of pip is available: 26.0.1 -> 26.1.2
|
||||
[notice] To update, run: pip install --upgrade pip
|
||||
@@ -0,0 +1,362 @@
|
||||
# Deployment Guide
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Podman** (or Docker) with compose plugin
|
||||
- **Ollama** running with models: `gpt-oss:20b`, `qwen3-embedding:8b`
|
||||
- At least 8 GB RAM for the marker-api container (model loading)
|
||||
- GPU recommended but not required for marker-pdf OCR
|
||||
|
||||
## Quick Start (Podman/Docker Compose)
|
||||
|
||||
### 1. Update Environment Variables
|
||||
|
||||
Create `.env` in the project root:
|
||||
|
||||
```env
|
||||
OLLAMA_URL=http://10.0.1.127:11434
|
||||
GPT_OSS_MODEL=gpt-oss:20b
|
||||
MARKER_API_URL=http://marker-api:8001
|
||||
DB_HOST=postgres
|
||||
DB_PORT=5432
|
||||
DB_NAME=research
|
||||
DB_USER=research
|
||||
DB_PASSWORD=research123
|
||||
VECTOR_DIM=4096
|
||||
```
|
||||
|
||||
### 2. Update `podman-compose.yml` `VECTOR_DIM`
|
||||
|
||||
In `podman-compose.yml`, under `app.environment`, set:
|
||||
|
||||
```yaml
|
||||
VECTOR_DIM: 4096
|
||||
```
|
||||
|
||||
### 3. Build & Start
|
||||
|
||||
```bash
|
||||
# Build all images
|
||||
podman-compose build
|
||||
|
||||
# Start services (PostgreSQL, marker-api, app)
|
||||
podman-compose up -d
|
||||
|
||||
# Or with Docker:
|
||||
# docker compose up -d
|
||||
```
|
||||
|
||||
### 4. Verify Services
|
||||
|
||||
```bash
|
||||
# Health check
|
||||
curl http://localhost:8000/health
|
||||
|
||||
# Should return:
|
||||
# {"status":"ok","db":"connected","ollama":"connected","marker_api":"ok"}
|
||||
```
|
||||
|
||||
### 5. Upload Documents
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/documents/upload \
|
||||
-F "file=@document.pdf"
|
||||
```
|
||||
|
||||
### 6. Run Research
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/research/run \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query":"Your question","doc_id":"<doc_id_from_upload>","skills":["summarizer"]}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Manual Container Setup (Podman, host networking)
|
||||
|
||||
Use this if you need direct host network access (e.g., Ollama on `10.0.1.127`):
|
||||
|
||||
### PostgreSQL + pgvector
|
||||
|
||||
```bash
|
||||
podman run -d --name pv-db --network host \
|
||||
-e POSTGRES_DB=research \
|
||||
-e POSTGRES_USER=research \
|
||||
-e POSTGRES_PASSWORD=research123 \
|
||||
-v pgdata:/var/lib/postgresql/data \
|
||||
-v ./migrations/init.sql:/docker-entrypoint-initdb.d/01-init.sql \
|
||||
docker.io/pgvector/pgvector:pg16
|
||||
```
|
||||
|
||||
### Marker OCR API
|
||||
|
||||
```bash
|
||||
cd marker-api
|
||||
|
||||
# Build (slow first time - downloads marker-pdf + surya models)
|
||||
podman build -t marker-api .
|
||||
|
||||
# Run with model cache volume
|
||||
podman run -d --name marker-api --network host \
|
||||
-v marker_models:/root/.cache \
|
||||
-v marker_ws:/workspace \
|
||||
-e OLLAMA_URL=http://10.0.1.127:11434 \
|
||||
localhost/marker-api
|
||||
```
|
||||
|
||||
**Model warmup** (required before first use):
|
||||
|
||||
```bash
|
||||
podman exec marker-api poetry run python3 -c "
|
||||
from marker.scripts.convert_single import convert_single_cli
|
||||
import sys
|
||||
# Run once to trigger model download
|
||||
sys.argv = ['warmup', '/dev/null', '--output_dir', '/tmp/warmup', '--page_range', '0-0']
|
||||
try:
|
||||
convert_single_cli(sys.argv[1:], standalone_mode=False)
|
||||
except: pass
|
||||
print('Models downloaded')
|
||||
"
|
||||
```
|
||||
|
||||
Then commit the warm image to persist models:
|
||||
|
||||
```bash
|
||||
podman commit marker-api localhost/marker-api:latest
|
||||
```
|
||||
|
||||
### App Container
|
||||
|
||||
```bash
|
||||
podman build -t agentic-app .
|
||||
|
||||
podman run -d --name agentic-app --network host \
|
||||
-v app_ws:/workspace \
|
||||
-e OLLAMA_URL=http://10.0.1.127:11434 \
|
||||
-e GPT_OSS_MODEL=gpt-oss:20b \
|
||||
-e MARKER_API_URL=http://localhost:8001 \
|
||||
-e DB_HOST=localhost \
|
||||
-e DB_PORT=5432 \
|
||||
-e DB_NAME=research \
|
||||
-e DB_USER=research \
|
||||
-e DB_PASSWORD=research123 \
|
||||
-e VECTOR_DIM=4096 \
|
||||
localhost/agentic-app
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Kubernetes Deployment
|
||||
|
||||
### 1. PostgreSQL StatefulSet
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: postgres
|
||||
spec:
|
||||
serviceName: postgres
|
||||
replicas: 1
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: postgres
|
||||
image: pgvector/pgvector:pg16
|
||||
env:
|
||||
- name: POSTGRES_DB
|
||||
value: research
|
||||
- name: POSTGRES_USER
|
||||
value: research
|
||||
- name: POSTGRES_PASSWORD
|
||||
value: research123
|
||||
volumeMounts:
|
||||
- name: pgdata
|
||||
mountPath: /var/lib/postgresql/data
|
||||
- name: init-sql
|
||||
mountPath: /docker-entrypoint-initdb.d/01-init.sql
|
||||
subPath: init.sql
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: pgdata
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: init-sql
|
||||
data:
|
||||
init.sql: |
|
||||
CREATE EXTENSION IF NOT EXISTS vector;
|
||||
CREATE TABLE IF NOT EXISTS documents (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
filename TEXT NOT NULL,
|
||||
doc_id TEXT,
|
||||
mime_type TEXT,
|
||||
file_path TEXT,
|
||||
status TEXT DEFAULT 'pending',
|
||||
page_count INTEGER DEFAULT 0,
|
||||
full_text TEXT,
|
||||
metadata JSONB DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS chunks (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
doc_id UUID REFERENCES documents(id) ON DELETE CASCADE,
|
||||
content TEXT NOT NULL,
|
||||
vector VECTOR(4096),
|
||||
page_num INTEGER,
|
||||
block_index INTEGER,
|
||||
polygon JSONB,
|
||||
chunk_type TEXT DEFAULT 'text',
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
-- (full schema in migrations/init.sql)
|
||||
```
|
||||
|
||||
### 2. Marker-API Deployment
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: marker-api
|
||||
spec:
|
||||
replicas: 1
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: marker-api
|
||||
image: marker-api:latest
|
||||
ports:
|
||||
- containerPort: 8001
|
||||
env:
|
||||
- name: OLLAMA_URL
|
||||
value: http://ollama-service:11434
|
||||
volumeMounts:
|
||||
- name: model-cache
|
||||
mountPath: /root/.cache
|
||||
resources:
|
||||
requests:
|
||||
memory: 4Gi
|
||||
cpu: 2
|
||||
limits:
|
||||
memory: 8Gi
|
||||
```
|
||||
|
||||
### 3. App Deployment
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: agentic-app
|
||||
spec:
|
||||
replicas: 1
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: app
|
||||
image: agentic-app:latest
|
||||
ports:
|
||||
- containerPort: 8000
|
||||
env:
|
||||
- name: OLLAMA_URL
|
||||
value: http://ollama-service:11434
|
||||
- name: GPT_OSS_MODEL
|
||||
value: gpt-oss:20b
|
||||
- name: MARKER_API_URL
|
||||
value: http://marker-api:8001
|
||||
- name: DB_HOST
|
||||
value: postgres
|
||||
- name: DB_PORT
|
||||
value: "5432"
|
||||
- name: DB_NAME
|
||||
value: research
|
||||
- name: DB_USER
|
||||
value: research
|
||||
- name: DB_PASSWORD
|
||||
value: research123
|
||||
- name: VECTOR_DIM
|
||||
value: "4096"
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: agentic-app
|
||||
spec:
|
||||
ports:
|
||||
- port: 8000
|
||||
targetPort: 8000
|
||||
selector:
|
||||
app: agentic-app
|
||||
type: LoadBalancer
|
||||
```
|
||||
|
||||
### 4. Apply
|
||||
|
||||
```bash
|
||||
kubectl apply -f k8s/
|
||||
# Expose app
|
||||
kubectl port-forward svc/agentic-app 8000:8000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
│ Ollama │ │ marker-api │ │ PostgreSQL │
|
||||
│ Host │◄────┤ (OCR/PDF) │ │ + pgvector │
|
||||
│ :11434 │ │ :8001 │ │ :5432 │
|
||||
└─────────────┘ └──────┬───────┘ └──────┬───────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────────────────┐
|
||||
│ agentic-app (FastAPI) │
|
||||
│ :8000 │
|
||||
│ - Upload/chunk documents │
|
||||
│ - Run LLM research skills │
|
||||
│ - Vector search (pgvector) │
|
||||
│ - Pipeline mode (3 stages) │
|
||||
└──────────────────────────────┘
|
||||
```
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `OLLAMA_URL` | `http://10.0.1.127:11434` | Ollama API endpoint |
|
||||
| `GPT_OSS_MODEL` | `gpt-oss:20b` | LLM for research |
|
||||
| `MARKER_API_URL` | `http://localhost:8001` | Marker OCR service |
|
||||
| `DB_HOST` | `localhost` | PostgreSQL host |
|
||||
| `DB_PORT` | 5432 | PostgreSQL port |
|
||||
| `DB_NAME` | `research` | Database name |
|
||||
| `DB_USER` | `research` | Database user |
|
||||
| `DB_PASSWORD` | `research123` | Database password |
|
||||
| `VECTOR_DIM` | `4096` | Embedding dimension |
|
||||
| `APP_HOST` | `0.0.0.0` | App bind address |
|
||||
| `APP_PORT` | 8000 | App listen port |
|
||||
| `DOC_STORE_DIR` | `/workspace/documents` | Uploaded file storage |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely Cause | Fix |
|
||||
|---------|-------------|-----|
|
||||
| `vector dimension mismatch 1536 and 4096` | `VECTOR_DIM` env var is `1536` | Set `VECTOR_DIM=4096` in compose file & rebuild |
|
||||
| `'NoneType' object has no attribute 'batch_chunk'` | `db` module not initialized at import time | Restart app container |
|
||||
| `inconsistent types deduced for parameter $1` | UUID vs text type conflict in `upsert_document` | Use `$1::uuid` / `$1::text` casts (fixed in current code) |
|
||||
| `OCR failed: No JSON output` | Marker model not downloaded | Run warmup step, commit image |
|
||||
| `expected str, got list` for vector insert | asyncpg can't serialize Python list to `vector` type | Convert to string `[x,y,z]` with `::vector` cast (fixed in current code) |
|
||||
|
||||
## Notes
|
||||
|
||||
- The marker-api container downloads ~3.3 GB of surya models on first run (layout, text_recognition, table_recognition, text_detection, ocr_error_detection). Commit the container to an image to persist these across restarts.
|
||||
- The app container defaults to reading only pages 0-5 per PDF for fast development. To process full documents, remove `--page_range 0-5` in `marker-api/server.py:58`.
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
FROM python:3.11-slim
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY pyproject.toml ./
|
||||
RUN pip install --upgrade pip && pip install -e .
|
||||
|
||||
COPY app/ ./app/
|
||||
COPY frontend/ ./frontend/
|
||||
COPY main.py ./
|
||||
COPY migrations/ ./migrations/
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
@@ -0,0 +1,2 @@
|
||||
"""Agentic Research - Document Research Tool."""
|
||||
__version__ = "0.1.0"
|
||||
@@ -0,0 +1,417 @@
|
||||
"""Agentic research engine - coordinates agent skills and Ollama LLM interactions."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from typing import Any
|
||||
import httpx
|
||||
|
||||
from app.agents.skills import SKILLS, Skill
|
||||
from app.agents.tools import TOOLS
|
||||
from app.config import get_settings
|
||||
|
||||
|
||||
class Researcher:
|
||||
"""Main research orchestrator that coordinates agents and tools."""
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self.active_skills: list[Skill] = []
|
||||
self.findings: list[dict] = []
|
||||
self.context: dict = {}
|
||||
|
||||
def select_skills(self, query: str, skill_names: list[str] | None = None) -> list[str]:
|
||||
"""Select relevant agent skills for the query. Defaults to all if none specified."""
|
||||
if skill_names:
|
||||
selected = []
|
||||
for name in skill_names:
|
||||
s = SKILLS.get(name)
|
||||
if s:
|
||||
selected.append(name)
|
||||
return selected
|
||||
|
||||
query_lower = query.lower()
|
||||
selected = []
|
||||
for name, skill in SKILLS.items():
|
||||
for token in skill.verb_tokens:
|
||||
if token in query_lower:
|
||||
selected.append(name)
|
||||
break
|
||||
# If no token matched, default to researcher + qa
|
||||
if not selected:
|
||||
selected = ["researcher", "qa_agent"]
|
||||
return selected
|
||||
|
||||
async def call_ollama(self, prompt: str, model: str | None = None, system: str | None = None) -> str:
|
||||
"""Call Ollama LLM with streaming support."""
|
||||
model = model or self.settings.gpt_oss_model
|
||||
messages = []
|
||||
if system:
|
||||
messages.append({"role": "system", "content": system})
|
||||
messages.append({"role": "user", "content": prompt})
|
||||
|
||||
async with httpx.AsyncClient(timeout=300) as client:
|
||||
resp = await client.post(
|
||||
f"{self.settings.ollama_url}/api/chat",
|
||||
json={"model": model, "messages": messages, "stream": False},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json().get("message", {}).get("content", "")
|
||||
|
||||
async def run_skill(
|
||||
self, skill_name: str, query: str, doc_id: str | None = None
|
||||
) -> str:
|
||||
"""Execute a single agent skill."""
|
||||
skill = SKILLS.get(skill_name)
|
||||
if not skill:
|
||||
return f"[Unknown skill: {skill_name}]"
|
||||
|
||||
system_prompt = f"""You are the {skill_name.replace('_', ' ').title()} agent.
|
||||
{skill.instructions}"""
|
||||
|
||||
tool_prompts = []
|
||||
for tool_name in skill.tools:
|
||||
tool_fn = TOOLS.get(tool_name)
|
||||
if not tool_fn:
|
||||
continue
|
||||
import inspect
|
||||
sig = inspect.signature(tool_fn)
|
||||
try:
|
||||
if tool_name == "read_document" and doc_id:
|
||||
result = await tool_fn(doc_id)
|
||||
elif tool_name == "read_chunks" and doc_id:
|
||||
result = await tool_fn(doc_id, limit=20)
|
||||
elif tool_name == "get_page_text" and doc_id:
|
||||
result = await tool_fn(doc_id, page_num=0)
|
||||
elif tool_name == "extract_facts" and doc_id:
|
||||
result = await tool_fn(doc_id, question=query)
|
||||
elif tool_name == "list_documents":
|
||||
result = await tool_fn()
|
||||
elif tool_name in ("vector_search", "text_search", "memory_similarity_search"):
|
||||
kwargs = {"query": query}
|
||||
if doc_id and "doc_id" in sig.parameters:
|
||||
kwargs["doc_id"] = doc_id
|
||||
result = await tool_fn(**kwargs)
|
||||
elif tool_name in ("get_findings", "get_memories", "get_pipeline_state"):
|
||||
result = await tool_fn(session_id=doc_id or "")
|
||||
elif tool_name == "save_finding":
|
||||
result = await tool_fn(session_id=doc_id or "", question=query, answer="", finding_type="research", confidence=0.5)
|
||||
else:
|
||||
kwargs = {}
|
||||
if "doc_id" in sig.parameters and doc_id:
|
||||
kwargs["doc_id"] = doc_id
|
||||
if "query" in sig.parameters:
|
||||
kwargs["query"] = query
|
||||
result = await tool_fn(**kwargs)
|
||||
tool_prompts.append(f"\n--- {tool_name} output ---\n{result}")
|
||||
except Exception as e:
|
||||
tool_prompts.append(f"\n--- {tool_name} error ---\n{str(e)}")
|
||||
|
||||
context = "".join(tool_prompts)
|
||||
|
||||
prompt = f"""Research query: {query}
|
||||
Document ID: {doc_id}
|
||||
Context from tools:
|
||||
{context}
|
||||
|
||||
Provide your analysis following the {skill_name} protocol."""
|
||||
|
||||
response = await self.call_ollama(prompt, system=system_prompt)
|
||||
|
||||
self.findings.append({
|
||||
"skill": skill_name,
|
||||
"query": query,
|
||||
"response": response,
|
||||
"doc_id": doc_id,
|
||||
"timestamp": str(os.popen('date +%Y-%m-%dT%H:%M:%S').read()).strip(),
|
||||
})
|
||||
|
||||
return response
|
||||
|
||||
async def run_research_session(
|
||||
self, query: str, session_id: str,
|
||||
doc_id: str | None = None,
|
||||
skill_names: list[str] | None = None
|
||||
) -> dict[str, str]:
|
||||
"""Run a full research session with selected agents."""
|
||||
selected_skills = self.select_skills(query, skill_names)
|
||||
results = {}
|
||||
|
||||
for skill_name in selected_skills:
|
||||
results[skill_name] = await self.run_skill(skill_name, query, doc_id)
|
||||
|
||||
return results
|
||||
|
||||
async def cross_reference(self, query: str, doc_ids: list[str]) -> str:
|
||||
"""Cross-reference content across multiple documents."""
|
||||
contexts = []
|
||||
for doc_id in doc_ids:
|
||||
chunks = await TOOLS["read_chunks"](doc_id)
|
||||
contexts.append(f"--- {doc_id} ---\n{chunks[:2000]}")
|
||||
|
||||
combined = "\n\n".join(contexts)
|
||||
prompt = f"""Cross-reference these documents for the query: {query}
|
||||
|
||||
{combined}
|
||||
|
||||
Direct comparison. No preamble."""
|
||||
|
||||
return await self.call_ollama(prompt, system="You are a cross-reference analyst. Output concise, comparative findings.")
|
||||
|
||||
|
||||
# ── Pipeline skills ────────────────────────────────
|
||||
|
||||
PIPELINE_STAGES = ("document_triage", "evidence_extraction", "research_synthesis")
|
||||
|
||||
|
||||
async def _parse_triage_output(response: str) -> dict:
|
||||
"""Parse document-triage output into structured dict."""
|
||||
obj = {}
|
||||
for section in ["OBJECTIVE", "SUB-QUESTIONS", "CLASSIFICATION", "READING_ORDER", "EXTRACTION_CRITERIA", "RISKS_AND_GAPS"]:
|
||||
marker = f"[{section}]"
|
||||
# Find start of this section
|
||||
start = response.find(marker)
|
||||
end = response.find("[", start + len(marker)) if start != -1 else -1
|
||||
if end != -1:
|
||||
chunk = response[start + len(marker):end].strip()
|
||||
elif start != -1:
|
||||
chunk = response[start + len(marker):].strip()
|
||||
else:
|
||||
chunk = ""
|
||||
obj[section] = chunk
|
||||
return obj
|
||||
|
||||
|
||||
async def _parse_evidence_output(response: str) -> list[dict]:
|
||||
"""Parse evidence-extraction output into structured list."""
|
||||
rows = []
|
||||
# Find [EVIDENCE_ROWS] section
|
||||
start_marker = "[EVIDENCE_ROWS]"
|
||||
start = response.find(start_marker)
|
||||
# Check if there's a cross-comparison section
|
||||
cross_start = response.find("[CROSS_COMPARISON]")
|
||||
end = cross_start if cross_start != -1 else len(response)
|
||||
if start == -1:
|
||||
start = 0
|
||||
section = response[start:end].strip()
|
||||
|
||||
lines = section.split("\n")
|
||||
for line in lines:
|
||||
line = line.strip()
|
||||
if not line or line.startswith("["):
|
||||
continue
|
||||
parts = [p.strip() for p in line.split("|")]
|
||||
if len(parts) >= 7:
|
||||
rows.append({
|
||||
"topic": parts[1],
|
||||
"evidence_type": parts[2],
|
||||
"description": parts[3],
|
||||
"trace_ref": parts[4],
|
||||
"evidence": parts[5],
|
||||
"analyst_note": parts[6],
|
||||
"confidence": parts[7] if len(parts) > 7 else "Medium",
|
||||
"review_needed": parts[8] if len(parts) > 8 else "No",
|
||||
})
|
||||
return rows
|
||||
|
||||
|
||||
async def _parse_synthesis_mode(response: str) -> str:
|
||||
"""Determine output mode from synthesis response content."""
|
||||
for mode in ["brief", "report", "gap analysis", "matrix"]:
|
||||
if mode in response.lower():
|
||||
return mode
|
||||
return "brief"
|
||||
|
||||
|
||||
class ResearchPipeline:
|
||||
"""Pipeline orchestrator: document_triage → evidence_extraction → research_synthesis."""
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self.plan: dict = {}
|
||||
self.evidence: list[dict] = []
|
||||
self.synthesis: str = ""
|
||||
self.findings: list[dict] = []
|
||||
|
||||
async def run(
|
||||
self,
|
||||
query: str,
|
||||
session_id: str,
|
||||
db,
|
||||
doc_ids: list[str] | None = None,
|
||||
output_mode: str | None = None,
|
||||
) -> dict:
|
||||
"""Run all three pipeline stages sequentially, persisting after each."""
|
||||
results = {}
|
||||
|
||||
# ── Stage 1: document_triage ─────────────────────
|
||||
triage_skill = SKILLS["document_triage"]
|
||||
system = f"You are document_triage. {triage_skill.instructions}"
|
||||
|
||||
# Gather context from available documents
|
||||
if doc_ids:
|
||||
doc_context = []
|
||||
for did in doc_ids:
|
||||
doc_info = await TOOLS["read_document"](did)
|
||||
doc_context.append(str(doc_info))
|
||||
context_input = "\n".join(doc_context)
|
||||
else:
|
||||
doc_list = await TOOLS["list_documents"]()
|
||||
context_input = doc_list
|
||||
|
||||
triage_prompt = f"""Research query: {query}
|
||||
Context from tools:
|
||||
{context_input}
|
||||
|
||||
Apply the document-tireage protocol."""
|
||||
|
||||
triage_output = await self._call_ollama(triage_prompt, system=system)
|
||||
self.plan = await _parse_triage_output(triage_output)
|
||||
self.findings.append({
|
||||
"stage": "triage",
|
||||
"output": triage_output,
|
||||
"state": self.plan,
|
||||
})
|
||||
await db.save_pipeline_stage(session_id, "triage", triage_output, self.plan)
|
||||
results["triage"] = triage_output
|
||||
results["triage_state"] = self.plan
|
||||
|
||||
# ── Stage 2: evidence_extraction ────────────────
|
||||
evidence_skill = SKILLS["evidence_extraction"]
|
||||
system = f"""You are evidence_extraction. {evidence_skill.instructions}
|
||||
|
||||
Triage plan (from previous stage):
|
||||
{json.dumps(self.plan, indent=2, default=str)}
|
||||
|
||||
Focus on extracting evidence for the sub-questions and extraction criteria defined above."""
|
||||
|
||||
# Read chunks from all relevant documents
|
||||
all_chunks = []
|
||||
for did in (doc_ids or []):
|
||||
chunks = await TOOLS["read_chunks"](did)
|
||||
all_chunks.append(f"--- Document {did} ---\n{chunks}")
|
||||
|
||||
context_input = "\n\n".join(all_chunks) if all_chunks else "No documents loaded yet."
|
||||
|
||||
evidence_prompt = f"""Research query: {query}
|
||||
Sub-questions to address:
|
||||
{self.plan.get('SUB-QUESTIONS', 'N/A')}
|
||||
|
||||
Context from tools:
|
||||
{context_input}
|
||||
|
||||
Apply the evidence_extraction protocol. Return structured evidence rows."""
|
||||
|
||||
evidence_output = await self._call_ollama(evidence_prompt, system=system)
|
||||
self.evidence = await _parse_evidence_output(evidence_output)
|
||||
|
||||
# Persist to DB
|
||||
if self.evidence:
|
||||
await db.save_structured_evidence(session_id, self.evidence)
|
||||
await db.save_pipeline_stage(session_id, "evidence", evidence_output, {"row_count": len(self.evidence)})
|
||||
|
||||
results["evidence"] = evidence_output
|
||||
results["evidence_rows"] = self.evidence
|
||||
|
||||
# ── Stage 3: research_synthesis ─────────────────
|
||||
synthesis_skill = SKILLS["research_synthesis"]
|
||||
|
||||
mode = output_mode or await _parse_synthesis_mode(evidence_output)
|
||||
mode_prompts = {
|
||||
"brief": "Use Research Brief output mode.",
|
||||
"report": "Use Research Report output mode.",
|
||||
"gap": "Use Gap Analysis output mode.",
|
||||
"matrix": "Use Comparison Matrix output mode.",
|
||||
}
|
||||
mode_instruct = mode_prompts.get(mode, mode_prompts["brief"])
|
||||
|
||||
system = f"""You are research_synthesis. {synthesis_skill.instructions}
|
||||
|
||||
{mode_instruct}
|
||||
|
||||
Extracted evidence (from previous stage):
|
||||
{json.dumps(self.evidence, indent=2, default=str)[:15000]}
|
||||
|
||||
Apply the research_synthesis protocol."""
|
||||
|
||||
synthesis_prompt = f"""Research query: {query}
|
||||
|
||||
Sub-questions:
|
||||
{self.plan.get('SUB-QUESTIONS', 'N/A')}
|
||||
|
||||
Apply the research_synthesis protocol."""
|
||||
|
||||
synthesis_output = await self._call_ollama(synthesis_prompt, system=system)
|
||||
|
||||
self.synthesis = synthesis_output
|
||||
synthesis_state = {
|
||||
"mode": mode,
|
||||
"sub_questions": self.plan.get("SUB-QUESTIONS", ""),
|
||||
"evidence_count": len(self.evidence),
|
||||
}
|
||||
await db.save_pipeline_stage(session_id, "synthesis", synthesis_output, synthesis_state)
|
||||
|
||||
results["synthesis"] = synthesis_output
|
||||
results["output_mode"] = mode
|
||||
results["pipeline_complete"] = True
|
||||
|
||||
self.findings.append({
|
||||
"stage": "synthesis",
|
||||
"output": synthesis_output,
|
||||
"state": synthesis_state,
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
async def _call_ollama(self, prompt: str, system: str) -> str:
|
||||
"""Call Ollama LLM."""
|
||||
messages = [
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": prompt},
|
||||
]
|
||||
async with httpx.AsyncClient(timeout=600) as client:
|
||||
resp = await client.post(
|
||||
f"{self.settings.ollama_url}/api/chat",
|
||||
json={"model": self.settings.gpt_oss_model, "messages": messages, "stream": False},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json().get("message", {}).get("content", "")
|
||||
|
||||
|
||||
async def render_pipeline_results(self, results: dict) -> list[dict]:
|
||||
"""Render pipeline results for frontend display."""
|
||||
sections = []
|
||||
|
||||
# Triage stage
|
||||
if "triage_state" in results:
|
||||
plan = results["triage_state"]
|
||||
sections.append({
|
||||
"stage": "triage",
|
||||
"title": "Stage 1: Source Triage",
|
||||
"objective": plan.get("OBJECTIVE", ""),
|
||||
"sub_questions": plan.get("SUB-QUESTIONS", ""),
|
||||
"classification": plan.get("CLASSIFICATION", ""),
|
||||
"reading_order": plan.get("READING_ORDER", ""),
|
||||
"extraction_criteria": plan.get("EXTRACTION_CRITERIA", ""),
|
||||
"risks_gaps": plan.get("RISKS_AND_GAPS", ""),
|
||||
})
|
||||
|
||||
# Evidence stage
|
||||
if "evidence_rows" in results:
|
||||
rows = results["evidence_rows"]
|
||||
sections.append({
|
||||
"stage": "evidence",
|
||||
"title": f"Stage 2: Extracted Evidence ({len(rows)} rows)",
|
||||
"rows": rows,
|
||||
})
|
||||
|
||||
# Synthesis stage
|
||||
if "synthesis" in results:
|
||||
sections.append({
|
||||
"stage": "synthesis",
|
||||
"title": f"Stage 3: Research Synthesis (mode: {results.get('output_mode', 'auto')})",
|
||||
"synthesis": results["synthesis"],
|
||||
})
|
||||
|
||||
return sections
|
||||
@@ -0,0 +1,319 @@
|
||||
"""Agent skills - modular capabilities for the agentic research engine."""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
@dataclass
|
||||
class Skill:
|
||||
"""A single agent capability."""
|
||||
name: str
|
||||
description: str
|
||||
instructions: str
|
||||
verb_tokens: list[str] = field(default_factory=list)
|
||||
tools: list[str] = field(default_factory=list)
|
||||
output_format: str = "json"
|
||||
|
||||
|
||||
SKILLS = {
|
||||
"summarizer": Skill(
|
||||
name="summarizer",
|
||||
description="Create concise summaries of documents or sections",
|
||||
instructions="""
|
||||
You are a research summarizer. Create brief, factual summaries.
|
||||
- Lead with the main claim or finding
|
||||
- Omit examples and elaborations
|
||||
- Max 3 sentences for executive summary
|
||||
- List only critical facts
|
||||
- Use bullet points for key points
|
||||
- No preamble or hedging
|
||||
""",
|
||||
verb_tokens=["summarize", "summary", "overview", "abstract"],
|
||||
tools=["read_document", "read_chunks", "get_page_text", "extract_facts"],
|
||||
),
|
||||
"extractor": Skill(
|
||||
name="extractor",
|
||||
description="Extract specific facts, figures, entities, or information types from documents",
|
||||
instructions="""
|
||||
You are an information extraction agent. Be terse and direct.
|
||||
- Output raw facts only
|
||||
- One fact per line
|
||||
- Format: [entity] → [value]
|
||||
- Skip qualifiers like "likely", "possibly"
|
||||
- No introductions, conclusions, or commentary
|
||||
- Direct format only:
|
||||
• Key terms
|
||||
• Definitions
|
||||
• Statistics
|
||||
• Names, dates, organizations
|
||||
• Relationships
|
||||
""",
|
||||
verb_tokens=["extract", "extracted", "list", "find", "identify", "enumerate"],
|
||||
tools=["read_chunks", "vector_search", "extract_facts", "get_page_text"],
|
||||
),
|
||||
"comparator": Skill(
|
||||
name="comparator",
|
||||
description="Compare documents, sections, or concepts and identify differences/similarities",
|
||||
instructions="""
|
||||
You are a comparison analyst. Output findings rapidly.
|
||||
- Direct table format when possible
|
||||
- Left align, right align
|
||||
- Use ✓ and ✗ markers
|
||||
- One line per difference
|
||||
- No fluff, no filler
|
||||
""",
|
||||
verb_tokens=["compare", "contrast", "similarities", "differences", "vs", "versus"],
|
||||
tools=["read_chunks", "vector_search", "get_page_text", "read_document"],
|
||||
),
|
||||
"critic": Skill(
|
||||
name="critic",
|
||||
description="Analyze arguments, findings, or claims for quality, validity, and gaps",
|
||||
instructions="""
|
||||
You are a critical analyst. Evaluate quickly and directly.
|
||||
- Strengths: 2-3 items max
|
||||
- Weaknesses: 2-3 items max
|
||||
- Gaps: list only critical ones
|
||||
- Assumptions: call out directly
|
||||
- Confidence: high/medium/low
|
||||
- Be blunt but accurate
|
||||
""",
|
||||
verb_tokens=["critique", "criticize", "evaluate", "assess", "review", "analyze validity"],
|
||||
tools=["read_chunks", "vector_search", "get_document", "extract_facts"],
|
||||
),
|
||||
"researcher": Skill(
|
||||
name="researcher",
|
||||
description="Conduct deep research on a topic using document collection, cross-referencing, and synthesis",
|
||||
instructions="""
|
||||
You are a research specialist. Work methodically.
|
||||
1. Parse the query for key concepts
|
||||
2. Retrieve relevant chunks via vector search
|
||||
3. Extract supporting evidence
|
||||
4. Note contradictions within sources
|
||||
5. Synthesize findings directly
|
||||
Format:
|
||||
- Context: 1 line
|
||||
- Evidence: bullet citations
|
||||
- Findings: 2-3 bullets
|
||||
- Limitations: 1 line
|
||||
- Output: concise, no padding
|
||||
""",
|
||||
verb_tokens=["research", "investigate", "explore", "dig into", "study", "analyze"],
|
||||
tools=["vector_search", "text_search", "read_chunks", "read_document", "memory_similarity_search"],
|
||||
),
|
||||
"context_agent": Skill(
|
||||
name="context_agent",
|
||||
description="Gather background context and establish research framing from available documents",
|
||||
instructions="""
|
||||
You establish research context. Direct output.
|
||||
- Document landscape: scope, domain, volume
|
||||
- Key themes (3-5)
|
||||
- Document types and relevance
|
||||
- Gaps in coverage
|
||||
- Suggested research angles
|
||||
All points. No paragraphs.
|
||||
""",
|
||||
verb_tokens=["context", "background", "landscape", "overview", "scope"],
|
||||
tools=["read_document", "list_documents", "vector_search", "get_memories"],
|
||||
),
|
||||
"qa_agent": Skill(
|
||||
name="qa_agent",
|
||||
description="Answer specific questions about document content with source-anchored responses",
|
||||
instructions="""
|
||||
You answer research questions directly.
|
||||
1. Locate relevant evidence
|
||||
2. Quote sources inline [doc_name:page]
|
||||
3. Synthesize one direct answer
|
||||
4. Note uncertainty
|
||||
Format:
|
||||
Q: [restated briefly]
|
||||
A: [direct answer]
|
||||
Sources: [citations]
|
||||
""",
|
||||
verb_tokens=["answer", "who", "what", "when", "where", "why", "how", "question"],
|
||||
tools=["vector_search", "text_search", "read_chunks", "get_page_text"],
|
||||
),
|
||||
"aggregator": Skill(
|
||||
name="aggregator",
|
||||
description="Combine findings across multiple documents into a unified research synthesis",
|
||||
instructions="""
|
||||
You synthesize cross-document findings.
|
||||
- Group by topic, not by document
|
||||
- Consensus findings first
|
||||
- Conflicts second
|
||||
- Novel insights third
|
||||
- Confidence ratings on each cluster
|
||||
- One-line summaries only
|
||||
- Bold key terms
|
||||
""",
|
||||
verb_tokens=["aggregate", "synthesize", "merge", "combine", "consolidate", "integrate"],
|
||||
tools=["read_document", "vector_search", "get_findings", "get_memories"],
|
||||
),
|
||||
# ── Pipeline skills ───────────────────────────────────
|
||||
|
||||
"document_triage": Skill(
|
||||
name="document_triage",
|
||||
description="Scope a document research task, prioritize sources, and define an evidence-driven reading plan",
|
||||
instructions="""\
|
||||
You are the document-triage agent for rigorous document-based research.
|
||||
Your job is to decide what matters and how to approach the corpus BEFORE deep reading.
|
||||
|
||||
FOLLOW THIS WORKFLOW:
|
||||
|
||||
1. Restate the research objective in one sentence
|
||||
2. Convert the objective into focused numbered sub-questions
|
||||
3. Classify available documents by value:
|
||||
PRIMARY — essential to answering the objective
|
||||
SECONDARY — supports or contextualizes
|
||||
BACKGROUND — peripheral reference
|
||||
LIKELY IRRELEVANT — skip unless needed
|
||||
4. For each classified document, state what evidence types to look for:
|
||||
requirements, decisions, risks, assumptions, responsibilities,
|
||||
timelines, definitions, constraints, dependencies, open issues
|
||||
5. Recommend an efficient reading order with rationale
|
||||
6. Define extraction criteria for the next phase
|
||||
7. Flag ambiguities, missing sources, and review risks
|
||||
|
||||
OUTPUT FORMAT (return exactly these sections):
|
||||
|
||||
[OBJECTIVE]: One-sentence research goal
|
||||
[SUB-QUESTIONS]: 1. ... 2. ... ...
|
||||
[CLASSIFICATION]: doc_name | PRIMARY/SECONDARY/BACKGROUND/IRRELEVANT | reason | signal types
|
||||
[READING ORDER]: 1. docA -> docB ... with rationale
|
||||
[EXTRACTION CRITERIA]: Checklist of evidence to capture
|
||||
[RISKS AND GAPS]: Missing docs, unclear scope, blind spots
|
||||
|
||||
RULES:
|
||||
- Be specific, not generic
|
||||
- Prefer primary documents over commentary
|
||||
- Do not summarize documents in depth
|
||||
- Do not invent document contents
|
||||
- Do not claim conclusions before extraction
|
||||
- Separate fact-finding from interpretation
|
||||
""",
|
||||
verb_tokens=["triage", "scope", "plan", "classi", "prioriti", "reading order", "assessment"],
|
||||
tools=["list_documents", "read_document", "read_chunks"],
|
||||
),
|
||||
|
||||
"evidence_extraction": Skill(
|
||||
name="evidence_extraction",
|
||||
description="Extract structured evidence from documents with traceability, quotations, and confidence markers",
|
||||
instructions="""\
|
||||
You are the evidence-extraction agent. Read documents and extract
|
||||
evidence relevant to the defined research question.
|
||||
|
||||
CORE RULES:
|
||||
- Separate direct evidence from interpretation
|
||||
- Capture exact quotations or precise paraphrases
|
||||
- Record document references for every finding
|
||||
- Mark uncertain or ambiguous findings
|
||||
- Distinguish: direct statement, implied interpretation, missing info
|
||||
- Normalize wording without losing meaning
|
||||
- Flag ambiguity, contradiction, or incomplete support
|
||||
|
||||
For EVERY finding output these fields:
|
||||
- Topic: the theme
|
||||
- Evidence type: requirement|decision|risk|assumption|responsibility|constraint|definition|date|dependency|open_issue
|
||||
- Description: concise factual summary
|
||||
- Document reference: source name, section, page
|
||||
- Extracted evidence: quote or faithful paraphrase
|
||||
- Analyst note: relevance without overstating certainty
|
||||
- Confidence: one of High|Medium|Low
|
||||
- Review needed: Yes|No
|
||||
|
||||
If multiple documents cover the same topic, also output a comparison table:
|
||||
Topic | Source A | Source B | Agreement | Conflict | Notes
|
||||
|
||||
OUTPUT FORMAT (return exactly):
|
||||
|
||||
[EVIDENCE_ROWS]:
|
||||
# topic | evidence_type | description | doc_ref | evidence | analyst_note | confidence | review_needed
|
||||
1 | ... | ... | ... | ... | ... | ... | ...
|
||||
...
|
||||
|
||||
[CROSS_COMPARISON] (only if multiple sources cover same topic):
|
||||
topic | source_a | source_b | agreement | conflict | notes
|
||||
|
||||
RULES:
|
||||
- Keep extraction atomic: one row per distinct finding
|
||||
- Preserve qualifiers: must, should, may, unless
|
||||
- Mark uncertainty explicitly
|
||||
- Prefer exact citations over memory-based summaries
|
||||
- Do not write the final conclusion
|
||||
- Do not collapse multiple findings into vague rows
|
||||
- Do not omit document references
|
||||
- Do not present interpretation as if it were a quote
|
||||
- Do not silently resolve contradictions
|
||||
""",
|
||||
verb_tokens=["evidence extract", "extract evidence", "traceable", "auditable", "structured extract"],
|
||||
tools=["read_chunks", "get_page_text", "read_document", "vector_search"],
|
||||
),
|
||||
|
||||
"research_synthesis": Skill(
|
||||
name="research_synthesis",
|
||||
description="Synthesize extracted document evidence into a clear, traceable conclusion, brief, or decision-ready report",
|
||||
instructions="""\
|
||||
You are the research-synthesis agent. Turn extracted evidence into
|
||||
a coherent, traceable final output.
|
||||
|
||||
INPUT: research objective + extracted evidence table + optional comparison table
|
||||
|
||||
WORKFLOW:
|
||||
1. Restate the question
|
||||
2. Group evidence by theme or sub-question
|
||||
3. Identify: supported conclusions, partial support, contradictions, missing evidence
|
||||
4. Draft findings with explicit traceability
|
||||
5. Separate: facts from documents | interpretation | recommendations
|
||||
6. Produce final structured output matching the requested output mode
|
||||
7. End with known limitations and review points
|
||||
|
||||
OUTPUT MODES:
|
||||
|
||||
### A) Research Brief (default for "brief")
|
||||
- Question
|
||||
- Short answer
|
||||
- Key findings (each with source anchor)
|
||||
- Conflicting evidence
|
||||
- Gaps
|
||||
- Recommended next step
|
||||
|
||||
### B) Research Report (default for "report")
|
||||
- Objective
|
||||
- Scope
|
||||
- Method
|
||||
- Findings by theme
|
||||
- Evidence-backed conclusion
|
||||
- Known gaps and limitations
|
||||
- Appendix with source references
|
||||
|
||||
### C) Gap Analysis (default for "gap")
|
||||
- Assessment topic
|
||||
- Evidence found
|
||||
- Gap
|
||||
- Impact
|
||||
- Confidence
|
||||
- Review needed
|
||||
|
||||
### D) Comparison Matrix
|
||||
- Topic | Source 1 | Source 2 | ... | Consensus | Conflict
|
||||
|
||||
TRACEABILITY RULE:
|
||||
Every substantive conclusion must include a source anchor:
|
||||
document name, section/page, or extracted finding number.
|
||||
If traceability is weak, say so explicitly.
|
||||
|
||||
RULES:
|
||||
- Conclusions must follow from extracted evidence
|
||||
- Contradictions must be surfaced, not hidden
|
||||
- Distinguish "document says" from "my recommendation"
|
||||
- State limits of the available corpus
|
||||
- Prefer precise wording over polished vagueness
|
||||
- Do not introduce facts not extracted
|
||||
- Do not overclaim certainty
|
||||
- Do not bury disagreements between documents
|
||||
- Do not produce recommendations without showing evidence basis
|
||||
- Do not omit limitations
|
||||
""",
|
||||
verb_tokens=["synthesize", "synthesis", "brief", "report", "gap analysis", "conclusion", "summary report"],
|
||||
tools=["read_chunks", "get_findings", "get_memories"],
|
||||
),
|
||||
}
|
||||
@@ -0,0 +1,250 @@
|
||||
"""Agent tools - callable operations for the agentic research engine."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
import httpx
|
||||
|
||||
from app.config import get_settings
|
||||
from app.db.database import db
|
||||
|
||||
|
||||
async def read_document(doc_id: str) -> dict[str, Any]:
|
||||
"""Read complete document record."""
|
||||
doc = await db.get_document(doc_id)
|
||||
if not doc:
|
||||
return {"error": "Document not found"}
|
||||
return {"document": doc}
|
||||
|
||||
|
||||
async def read_chunks(doc_id: str, page_num: int | None = None, limit: int = 50) -> str:
|
||||
"""Read text chunks for a document."""
|
||||
chunks = await db.get_doc_chunks(doc_id)
|
||||
if page_num is not None:
|
||||
chunks = [c for c in chunks if c.get("page_num") == page_num]
|
||||
chunks = chunks[:limit]
|
||||
parts = []
|
||||
for c in chunks:
|
||||
page_part = f"[p{c['page_num']}]" if c.get("page_num") else ""
|
||||
parts.append(f"{page_part} {c['content'][:500]}")
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
async def get_page_text(doc_id: str, page_num: int) -> str:
|
||||
"""Extract text for a specific page, including polygon metadata."""
|
||||
chunks = await db.get_doc_chunks(doc_id)
|
||||
page_chunks = [c for c in chunks if c.get("page_num") == page_num]
|
||||
if not page_chunks:
|
||||
return f"No content available for page {page_num}"
|
||||
|
||||
# Build page view with polygon info
|
||||
text_parts = []
|
||||
for c in page_chunks:
|
||||
polygon = c.get("polygon")
|
||||
poly_info = ""
|
||||
if polygon and isinstance(polygon, dict):
|
||||
bbox = polygon.get("bbox", [])
|
||||
if bbox:
|
||||
poly_info = f" [bbox:{bbox[0]:.0f},{bbox[1]:.0f},{bbox[2]:.0f},{bbox[3]:.0f}]"
|
||||
text_parts.append(f"[{c['block_index']}{poly_info}] {c['content'][:300]}")
|
||||
|
||||
return f"--- Page {page_num} ---\n" + "\n".join(text_parts)
|
||||
|
||||
|
||||
async def vector_search(query: str, doc_id: str | None = None, limit: int = 20) -> str:
|
||||
"""Search document chunks by semantic similarity."""
|
||||
settings = get_settings()
|
||||
async with httpx.AsyncClient(timeout=30) as client:
|
||||
query_vec = []
|
||||
# Try to get embedding via Ollama
|
||||
try:
|
||||
resp = await client.post(
|
||||
f"{settings.ollama_url}/api/embeddings",
|
||||
json={"model": "nomic-embed-text", "prompt": query},
|
||||
)
|
||||
query_vec = resp.json().get("embedding", [])
|
||||
except Exception:
|
||||
# Fall back to local embedding
|
||||
from app.core import get_embedding
|
||||
query_vec = get_embedding(query)
|
||||
|
||||
if not query_vec:
|
||||
return "No embeddings available. Using text search fallback."
|
||||
|
||||
results = await db.vector_search(query_vec, doc_id=doc_id, limit=limit)
|
||||
parts = []
|
||||
for r in results:
|
||||
score = r.get("similarity", 0)
|
||||
page = r.get("page_num", "?")
|
||||
parts.append(f"[{score:.2f}] (p{page}) {r['content'][:200]}")
|
||||
return "\n".join(parts) if parts else "No similar content found."
|
||||
|
||||
|
||||
async def text_search(query: str, doc_id: str | None = None, limit: int = 20) -> str:
|
||||
"""Search chunks by keyword match."""
|
||||
results = await db.search_chunks_text(query, doc_id=doc_id, limit=limit)
|
||||
parts = []
|
||||
for r in results:
|
||||
rank = r.get("rank", 0)
|
||||
page = r.get("page_num", "?")
|
||||
parts.append(f"[{rank:.2f}] (p{page}) {r['content'][:200]}")
|
||||
return "\n".join(parts) if parts else "No text matches found."
|
||||
|
||||
|
||||
async def extract_facts(doc_id: str, question: str | None = None) -> str:
|
||||
"""Extract key facts from document."""
|
||||
chunks = await db.get_doc_chunks(doc_id)
|
||||
# Extract from chunk metadata
|
||||
facts = []
|
||||
for c in chunks:
|
||||
content = c.get("content", "")
|
||||
if question and question.lower() not in content.lower():
|
||||
continue
|
||||
if "fact" in c.get("chunk_type", "").lower():
|
||||
facts.append(content[:200])
|
||||
elif len(content.strip()) > 50:
|
||||
facts.append(content[:200])
|
||||
|
||||
seen = set()
|
||||
unique = []
|
||||
for f in facts:
|
||||
key = f[:40]
|
||||
if key not in seen:
|
||||
seen.add(key)
|
||||
unique.append(f)
|
||||
|
||||
return "\n".join(f"I: {f}" for f in unique[:20])
|
||||
|
||||
|
||||
async def get_polygon_view(doc_id: str, page_num: int, block_index: int | None = None) -> dict:
|
||||
"""Get page with polygon coordinates for visualization."""
|
||||
chunks = await db.get_doc_chunks(doc_id)
|
||||
page_chunks = [c for c in chunks if c.get("page_num") == page_num]
|
||||
|
||||
blocks = []
|
||||
for c in page_chunks:
|
||||
poly = c.get("polygon")
|
||||
if poly and isinstance(poly, dict):
|
||||
blocks.append({
|
||||
"index": c.get("block_index", 0),
|
||||
"page": page_num,
|
||||
"polygon": poly,
|
||||
"content": c.get("content", "")[:200],
|
||||
"type": c.get("chunk_type", "text"),
|
||||
})
|
||||
elif block_index is None or c.get("block_index") == block_index:
|
||||
blocks.append({
|
||||
"index": c.get("block_index", 0),
|
||||
"page": page_num,
|
||||
"polygon": {"bbox": [0, 0, 1000, 1000]},
|
||||
"content": c.get("content", "")[:200],
|
||||
"type": c.get("chunk_type", "text"),
|
||||
})
|
||||
|
||||
# Compute page bounding box from all polygons
|
||||
all_polys = [b["polygon"] for b in blocks if b["polygon"] and isinstance(b["polygon"], dict)]
|
||||
|
||||
pages_data = {"page_num": page_num, "blocks": blocks}
|
||||
|
||||
if all_polys:
|
||||
min_x = min(p.get("bbox", [0, 0, 0, 0])[:2] for p in all_polys)
|
||||
max_x = max(p.get("bbox", [0, 0, 0, 0])[2:] for p in all_polys)
|
||||
pages_data["page_bbox"] = {
|
||||
"x_min": min(min_x),
|
||||
"y_min": min(min_x, key=lambda x: x[1])[1],
|
||||
"width": max(max_x) - min(min_x),
|
||||
"height": max(max_x, key=lambda x: x[1])[1] - min(min_x, key=lambda x: x[1])[1],
|
||||
}
|
||||
|
||||
return pages_data
|
||||
|
||||
|
||||
async def get_memories(session_id: str) -> list[dict]:
|
||||
"""Get research memories from pgvector."""
|
||||
return await db.get_memories(session_id)
|
||||
|
||||
|
||||
async def memory_similarity_search(query: str, limit: int = 10) -> str:
|
||||
"""Search across all research memories."""
|
||||
results = await db.memory_similarity_search(query, limit=limit)
|
||||
parts = []
|
||||
for r in results:
|
||||
parts.append(f"[{r.get('importance', '?')}] ({r.get('memory_type', '?')}) {r['content'][:200]}")
|
||||
return "\n".join(parts) if parts else "No memories found."
|
||||
|
||||
|
||||
async def list_documents(status: str | None = None) -> str:
|
||||
"""List all indexed documents."""
|
||||
docs = await db.list_documents(status=status)
|
||||
parts = []
|
||||
for d in docs:
|
||||
parts.append(f"- [{d['status']}] {d['filename']} (p{d.get('page_count', '?')}, {d['created_at']})")
|
||||
return "\n".join(parts) if parts else "No documents found."
|
||||
|
||||
|
||||
async def save_finding(session_id: str, question: str, answer: str,
|
||||
summary: str, confidence: float = 0.8,
|
||||
relevant_chunks: list | None = None) -> str:
|
||||
"""Save a research finding to the database."""
|
||||
finding_id = await db.store_finding(
|
||||
session_id, question, answer, summary,
|
||||
"researcher", confidence, relevant_chunks
|
||||
)
|
||||
await db.update_session(session_id, findings={"count": 1})
|
||||
return finding_id
|
||||
|
||||
|
||||
async def save_structured_evidence(
|
||||
session_id: str, rows: list[dict]
|
||||
) -> list[str]:
|
||||
"""Save evidence-extraction rows with traceability fields."""
|
||||
ids = await db.save_structured_evidence(session_id, rows)
|
||||
return ids
|
||||
|
||||
|
||||
async def get_pipeline_state(session_id: str) -> list[dict]:
|
||||
"""Get saved pipeline intermediate stages."""
|
||||
return await db.get_pipeline_stages(session_id)
|
||||
|
||||
|
||||
async def merge_evidence(rows_a: list[dict], rows_b: list[dict]) -> list[dict]:
|
||||
"""Merge two evidence sets, flagging agreement/conflict by topic."""
|
||||
by_topic: dict[str, list] = {}
|
||||
for r in rows_a + rows_b:
|
||||
topic = r.get("topic", "unknown")
|
||||
by_topic.setdefault(topic, []).append(r)
|
||||
merged = []
|
||||
for topic, rs in by_topic.items():
|
||||
if len(rs) == 2:
|
||||
merged.append({
|
||||
"topic": topic,
|
||||
"source_a": rs[0].get("evidence", ""),
|
||||
"source_b": rs[1].get("evidence", ""),
|
||||
"agreement": "Yes" if rs[0].get("evidence") == rs[1].get("evidence") else "No",
|
||||
"conflict": "Yes" if rs[0].get("confidence") != rs[1].get("confidence") else "No",
|
||||
"notes": f"Combined from {rs[0].get('source_doc','')} and {rs[1].get('source_doc','')}",
|
||||
})
|
||||
else:
|
||||
merged.append({k: rs[0].get(k, "") for k in ("topic", "description", "trace_ref", "evidence", "confidence")})
|
||||
return merged
|
||||
|
||||
|
||||
# Tool registry
|
||||
TOOLS = {
|
||||
"read_document": read_document,
|
||||
"read_chunks": read_chunks,
|
||||
"get_page_text": get_page_text,
|
||||
"vector_search": vector_search,
|
||||
"text_search": text_search,
|
||||
"extract_facts": extract_facts,
|
||||
"get_polygon_view": get_polygon_view,
|
||||
"get_memories": get_memories,
|
||||
"memory_similarity_search": memory_similarity_search,
|
||||
"list_documents": list_documents,
|
||||
"save_finding": save_finding,
|
||||
"save_structured_evidence": save_structured_evidence,
|
||||
"get_pipeline_state": get_pipeline_state,
|
||||
"merge_evidence": merge_evidence,
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Configuration loader for agentic research app."""
|
||||
from pydantic_settings import BaseSettings
|
||||
from functools import lru_cache
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
ollama_url: str = "http://10.0.1.127:11434"
|
||||
gpt_oss_model: str = "gpt-oss:20b"
|
||||
marker_api_url: str = "http://localhost:8001"
|
||||
ocr_url: str = "http://10.0.1.127:11434"
|
||||
deepseek_ocr_url: str = "http://10.0.1.127:11434"
|
||||
deepseek_ocr_model: str = "deepseek-ocr"
|
||||
db_host: str = "localhost"
|
||||
db_port: int = 5432
|
||||
db_name: str = "research"
|
||||
db_user: str = "research"
|
||||
db_password: str = "research123"
|
||||
app_host: str = "0.0.0.0"
|
||||
app_port: int = 8000
|
||||
workspace_dir: str = "/home/oval/Projects/agentic/workspace"
|
||||
documents_dir: str = "/home/oval/Projects/agentic/workspace/documents"
|
||||
vector_dim: int = 4096
|
||||
|
||||
embedding_model: str = "qwen3-embedding:8b"
|
||||
|
||||
model_config = {"env_file": ".env"}
|
||||
|
||||
|
||||
@lru_cache()
|
||||
def get_settings() -> Settings:
|
||||
return Settings()
|
||||
@@ -0,0 +1,4 @@
|
||||
"""Embedding service — uses Ollama qwen3-embedding:8b."""
|
||||
from app.core.embedding_engine import get_embedding_sync
|
||||
|
||||
get_embedding = get_embedding_sync
|
||||
@@ -0,0 +1,79 @@
|
||||
"""Embedding generation via Ollama's qwen3-embedding:8b model."""
|
||||
import hashlib
|
||||
import numpy as np
|
||||
|
||||
|
||||
def get_embedding_sync(text: str, ollama_url: str = "http://10.0.1.127:11434") -> list[float]:
|
||||
"""Synchronous embedding call via Ollama /qwen3-embedding:8b."""
|
||||
try:
|
||||
import httpx
|
||||
with httpx.Client(timeout=30) as client:
|
||||
resp = client.post(
|
||||
f"{ollama_url}/api/embed",
|
||||
json={"model": "qwen3-embedding:8b", "input": text},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
vectors = data.get("embeddings", [])
|
||||
if vectors:
|
||||
# Ollama may return multiple inputs; use first
|
||||
emb = vectors[0] if isinstance(vectors[0], list) else vectors
|
||||
return emb
|
||||
except Exception:
|
||||
pass
|
||||
# Fallback to deterministic feature vector if Ollama unavailable
|
||||
return _hash_embedding(text)
|
||||
|
||||
|
||||
def get_embedding(text: str, ollama_url: str = "http://10.0.1.127:11434") -> list[float]:
|
||||
"""Generate an embedding — calls Ollama qwen3-embedding:8b."""
|
||||
return get_embedding_sync(text, ollama_url)
|
||||
|
||||
|
||||
async def get_embedding_async(text: str, ollama_url: str = "http://10.0.1.127:11434") -> list[float]:
|
||||
"""Async embedding call via Ollama /qwen3-embedding:8b."""
|
||||
try:
|
||||
import httpx
|
||||
async with httpx.AsyncClient(timeout=30) as client:
|
||||
resp = await client.post(
|
||||
f"{ollama_url}/api/embed",
|
||||
json={"model": "qwen3-embedding:8b", "input": text},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
vectors = data.get("embeddings", [])
|
||||
if vectors:
|
||||
emb = vectors[0] if isinstance(vectors[0], list) else vectors
|
||||
return emb
|
||||
except Exception:
|
||||
pass
|
||||
return _hash_embedding(text)
|
||||
|
||||
|
||||
def _hash_embedding(text: str) -> list[float]:
|
||||
"""Deterministic 4096-dim feature vector fallback (no Ollama needed)."""
|
||||
feature_dim = 4096
|
||||
vec = np.zeros(feature_dim, dtype=np.float32)
|
||||
for n in [1, 2, 3, 4]:
|
||||
tokens = [text[i:i+n] for i in range(len(text)-n+1)]
|
||||
for token in tokens[:200]:
|
||||
h = hashlib.md5(token.encode()).hexdigest()
|
||||
for i in range(0, 12, 3):
|
||||
val = (int(h[i:i+2], 16) - 128) / 128.0
|
||||
feature_idx = (int(h[i+2:i+4], 16) * 37) % feature_dim
|
||||
vec[feature_idx] += val
|
||||
norm = np.linalg.norm(vec)
|
||||
if norm > 0:
|
||||
vec /= norm
|
||||
return vec.tolist()
|
||||
|
||||
|
||||
def compute_similarity(vec1: list[float], vec2: list[float]) -> float:
|
||||
"""Cosine similarity between two vectors."""
|
||||
v1 = np.array(vec1, dtype=np.float32)
|
||||
v2 = np.array(vec2, dtype=np.float32)
|
||||
if v1.shape[0] != v2.shape[0]:
|
||||
return 0.0
|
||||
v1 /= np.linalg.norm(v1) + 1e-8
|
||||
v2 /= np.linalg.norm(v2) + 1e-8
|
||||
return float(np.dot(v1, v2))
|
||||
@@ -0,0 +1,110 @@
|
||||
"""Marker/OCR integration for document processing."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import asyncio
|
||||
import io
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
import httpx
|
||||
|
||||
from app.config import get_settings
|
||||
import app.db.database as _database_mod
|
||||
|
||||
|
||||
class MarkerProcessor:
|
||||
"""Handles OCR via Marker API with polygon output."""
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
|
||||
async def process_pdf(self, file_data: bytes, filename: str) -> dict[str, Any]:
|
||||
"""Send PDF to Marker API for OCR with polygon extraction."""
|
||||
async with httpx.AsyncClient(timeout=300) as client:
|
||||
resp = await client.post(
|
||||
f"{self.settings.marker_api_url}/convert",
|
||||
files={"file": (filename, file_data, "application/pdf")},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
async def process_pdf_url(self, pdf_url: str, filename: str) -> dict[str, Any]:
|
||||
"""Process PDF from URL via Marker API."""
|
||||
async with httpx.AsyncClient(timeout=300) as client:
|
||||
resp = await client.post(
|
||||
f"{self.settings.marker_api_url}/convert",
|
||||
json={"pdf_url": pdf_url},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
async def parse_marker_json(self, marker_output: dict, doc_id: str) -> list[dict]:
|
||||
"""Parse Marker JSON output into vectorized chunks with polygon data."""
|
||||
pages = marker_output.get("pages", [])
|
||||
chunks = []
|
||||
|
||||
for page in pages:
|
||||
page_num = page.get("meta", {}).get("page_num", page.get("page", 0))
|
||||
text_lines = page.get("text_lines", [])
|
||||
|
||||
for block_idx, tl in enumerate(text_lines):
|
||||
content = tl.get("text", "")
|
||||
polygon = tl.get("bbox") or tl.get("polygon")
|
||||
block_type = tl.get("type", "text")
|
||||
|
||||
if not content or not isinstance(content, str) or not content.strip():
|
||||
continue
|
||||
|
||||
chunks.append({
|
||||
"content": content.strip(),
|
||||
"page_num": page_num,
|
||||
"block_index": block_idx,
|
||||
"polygon": polygon,
|
||||
"chunk_type": block_type,
|
||||
})
|
||||
|
||||
if chunks:
|
||||
await _database_mod.db.batch_chunk(doc_id, chunks)
|
||||
|
||||
return chunks
|
||||
|
||||
async def process_document_file(
|
||||
self, file_data: bytes, filename: str, doc_id: str
|
||||
) -> dict[str, Any]:
|
||||
"""Process document file and return structured result."""
|
||||
ext = Path(filename).suffix.lower()
|
||||
|
||||
if ext == ".pdf":
|
||||
result = await self.process_pdf(file_data, filename)
|
||||
pages_data = result.get("pages", [])
|
||||
|
||||
chunks = await self.parse_marker_json({"pages": pages_data}, doc_id)
|
||||
return {
|
||||
"success": result.get("success", True),
|
||||
"doc_id": doc_id,
|
||||
"page_count": result.get("page_count", len(pages_data)),
|
||||
"chunks": len(chunks),
|
||||
"ocr_model": result.get("ocr_model", "deepseek-ocr"),
|
||||
}
|
||||
|
||||
elif ext in (".txt", ".pdf", ".md"):
|
||||
text = file_data.decode("utf-8", errors="replace")
|
||||
paragraphs = re.split(r'\n\s*\n', text)
|
||||
chunks = []
|
||||
for i, para in enumerate(paragraphs):
|
||||
if len(para.strip()) > 20:
|
||||
chunks.append({
|
||||
"content": para.strip(),
|
||||
"page_num": 0,
|
||||
"block_index": i,
|
||||
"polygon": None,
|
||||
"chunk_type": "text",
|
||||
})
|
||||
|
||||
if chunks:
|
||||
await _database_mod.db.batch_chunk(doc_id, chunks)
|
||||
|
||||
return {"success": True, "doc_id": doc_id, "chunks": len(chunks), "page_count": 1}
|
||||
|
||||
return {"success": False, "error": f"Unsupported file type: {ext}"}
|
||||
@@ -0,0 +1,402 @@
|
||||
"""Async PostgreSQL database layer with pgvector support."""
|
||||
import json
|
||||
import uuid
|
||||
from typing import Any
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
import asyncpg
|
||||
import numpy as np
|
||||
|
||||
from app.config import get_settings
|
||||
|
||||
|
||||
class Database:
|
||||
"""Handles all PostgreSQL/pgvector operations."""
|
||||
|
||||
def __init__(self, pool: asyncpg.Pool):
|
||||
self.pool = pool
|
||||
|
||||
@classmethod
|
||||
async def create_pool(cls) -> asyncpg.Pool:
|
||||
settings = get_settings()
|
||||
pool = await asyncpg.create_pool(
|
||||
host=settings.db_host,
|
||||
port=settings.db_port,
|
||||
database=settings.db_name,
|
||||
user=settings.db_user,
|
||||
password=settings.db_password,
|
||||
min_size=2,
|
||||
max_size=10,
|
||||
)
|
||||
return pool
|
||||
|
||||
@classmethod
|
||||
@asynccontextmanager
|
||||
async def connection(cls):
|
||||
pool = await cls.create_pool()
|
||||
async with pool.acquire() as conn:
|
||||
yield conn
|
||||
await pool.close()
|
||||
|
||||
async def init_schema(self):
|
||||
"""Run initial SQL schema from migrations."""
|
||||
import os
|
||||
migrations_path = os.path.join(
|
||||
os.path.dirname(__file__), "..", "..", "migrations", "init.sql"
|
||||
)
|
||||
async with self.connection() as conn:
|
||||
with open(migrations_path) as f:
|
||||
await conn.execute(f.read())
|
||||
|
||||
async def upsert_document(
|
||||
self, filename: str, doc_id: str | None, mime_type: str,
|
||||
file_path: str, status: str, page_count: int,
|
||||
full_text: str, metadata: dict
|
||||
) -> str:
|
||||
async with self.connection() as conn:
|
||||
pk = doc_id or str(uuid.uuid4())
|
||||
await conn.execute(
|
||||
"""INSERT INTO documents (id, filename, doc_id, mime_type, file_path,
|
||||
status, page_count, full_text, metadata)
|
||||
VALUES ($1::uuid, $2, $1::text, $3, $4, $5, $6, $7, $8::jsonb)
|
||||
ON CONFLICT (filename) DO UPDATE SET
|
||||
status=EXCLUDED.status, page_count=EXCLUDED.page_count,
|
||||
full_text=EXCLUDED.full_text, metadata=EXCLUDED.metadata,
|
||||
updated_at=NOW()""",
|
||||
str(pk), filename, mime_type, file_path,
|
||||
status, page_count, full_text, json.dumps(metadata),
|
||||
)
|
||||
return pk
|
||||
|
||||
async def get_document(self, doc_id: str | uuid.UUID) -> dict | None:
|
||||
async with self.connection() as conn:
|
||||
row = await conn.fetchrow(
|
||||
"SELECT * FROM documents WHERE id=$1", doc_id
|
||||
)
|
||||
return dict(row) if row else None
|
||||
|
||||
async def list_documents(self, status: str | None = None) -> list[dict]:
|
||||
async with self.connection() as conn:
|
||||
if status:
|
||||
rows = await conn.fetch(
|
||||
"SELECT * FROM documents WHERE $1=ANY(string_to_array(status, ',')) ORDER BY created_at DESC",
|
||||
status,
|
||||
)
|
||||
else:
|
||||
rows = await conn.fetch("SELECT * FROM documents ORDER BY created_at DESC")
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
async def chunk_document(
|
||||
self, doc_id: str, content: str, polygon: dict | None,
|
||||
page_num: int, block_index: int, chunk_type: str = "text"
|
||||
) -> str:
|
||||
vec_str = "[" + ",".join(str(x) for x in self._extract_vector(content)) + "]"
|
||||
chunk_id = str(uuid.uuid4())
|
||||
async with self.connection() as conn:
|
||||
await conn.execute(
|
||||
"""INSERT INTO chunks (id, doc_id, content, vector, page_num,
|
||||
block_index, polygon, chunk_type)
|
||||
VALUES ($1, $2, $3, $4::vector, $5, $6, $7::jsonb, $8)""",
|
||||
chunk_id, str(doc_id), content, vec_str, page_num,
|
||||
block_index, json.dumps(polygon) if polygon else None, chunk_type,
|
||||
)
|
||||
return chunk_id
|
||||
|
||||
async def batch_chunk(self, doc_id: str, data: list[dict]):
|
||||
"""Insert multiple chunks at once."""
|
||||
vec_data = []
|
||||
for d in data:
|
||||
content = d.get("content", "")
|
||||
vec_str = "[" + ",".join(str(x) for x in self._extract_vector(content)) + "]"
|
||||
vec_data.append((
|
||||
str(doc_id), content, vec_str,
|
||||
d.get("page_num", 0), d.get("block_index", 0),
|
||||
json.dumps(d.get("polygon")) if d.get("polygon") else None,
|
||||
d.get("chunk_type", "text"),
|
||||
))
|
||||
async with self.connection() as conn:
|
||||
await conn.executemany(
|
||||
"""INSERT INTO chunks (doc_id, content, vector, page_num,
|
||||
block_index, polygon, chunk_type)
|
||||
VALUES ($1, $2, $3::vector, $4, $5, $6::jsonb, $7)""",
|
||||
vec_data
|
||||
)
|
||||
|
||||
async def vector_search(
|
||||
self, query_vector: list[float], doc_id: str | None = None,
|
||||
limit: int = 20, min_score: float = 0.0
|
||||
) -> list[dict]:
|
||||
"""Find similar chunks using pgvector cosine similarity."""
|
||||
query_vec = "[" + ",".join(str(x) for x in query_vector) + "]"
|
||||
async with self.connection() as conn:
|
||||
if doc_id:
|
||||
rows = await conn.fetch(
|
||||
"""SELECT id, content, doc_id, page_num, polygon,
|
||||
(1 - (vector <-> $4::vector) / 2) as similarity
|
||||
FROM chunks WHERE doc_id = $1
|
||||
AND (1 - (vector <-> $4::vector) / 2) >= $3
|
||||
ORDER BY vector <-> $4 LIMIT $2""",
|
||||
str(doc_id), limit, min_score, query_vec,
|
||||
)
|
||||
else:
|
||||
rows = await conn.fetch(
|
||||
"""SELECT id, content, doc_id, page_num, polygon,
|
||||
(1 - (vector <-> $3::vector) / 2) as similarity
|
||||
FROM chunks
|
||||
WHERE (1 - (vector <-> $3::vector) / 2) >= $2
|
||||
ORDER BY vector <-> $3 LIMIT $1""",
|
||||
limit, min_score, query_vec,
|
||||
)
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
async def semantic_search(
|
||||
self, query_text: str, limit: int = 20, min_score: float = 0.0
|
||||
) -> list[dict]:
|
||||
"""Semantic search by embedding the query text."""
|
||||
from app.core import get_embedding
|
||||
query_vec = get_embedding(query_text)
|
||||
return await self.vector_search(query_vec, limit=limit, min_score=min_score)
|
||||
|
||||
async def search_chunks_text(
|
||||
self, query: str, doc_id: str | None = None, limit: int = 20
|
||||
) -> list[dict]:
|
||||
"""Text-based search using trigram similarity."""
|
||||
async with self.connection() as conn:
|
||||
if doc_id:
|
||||
rows = await conn.fetch(
|
||||
"""SELECT id, content, doc_id, page_num,
|
||||
ts_rank(to_tsvector('simple', content),
|
||||
plainto_tsquery('simple', $4)) as rank
|
||||
FROM chunks WHERE doc_id = $1
|
||||
AND to_tsvector('simple', content) @@ plainto_tsquery('simple', $4)
|
||||
ORDER BY rank DESC LIMIT $2""",
|
||||
str(doc_id), limit, query,
|
||||
)
|
||||
else:
|
||||
rows = await conn.fetch(
|
||||
"""SELECT id, content, doc_id, page_num,
|
||||
ts_rank(to_tsvector('simple', content),
|
||||
plainto_tsquery('simple', $3)) as rank
|
||||
FROM chunks
|
||||
WHERE to_tsvector('simple', content) @@ plainto_tsquery('simple', $3)
|
||||
ORDER BY rank DESC LIMIT $1""",
|
||||
limit, query,
|
||||
)
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
async def store_memory(
|
||||
self, session_id: str, content: str, memory_type: str = "fact",
|
||||
importance: int = 3, source_doc_id: str | None = None
|
||||
) -> str:
|
||||
vec = self._extract_vector(content)
|
||||
mem_id = str(uuid.uuid4())
|
||||
async with self.connection() as conn:
|
||||
await conn.execute(
|
||||
"""INSERT INTO memories (id, session_id, content, vector,
|
||||
memory_type, importance, source_doc_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7)""",
|
||||
mem_id, str(session_id), content, vec, memory_type,
|
||||
importance, str(source_doc_id) if source_doc_id else None,
|
||||
)
|
||||
return mem_id
|
||||
|
||||
async def get_memories(
|
||||
self, session_id: str, limit: int = 50
|
||||
) -> list[dict]:
|
||||
async with self.connection() as conn:
|
||||
rows = await conn.fetch(
|
||||
"SELECT * FROM memories WHERE session_id=$1 ORDER BY importance DESC, created_at DESC LIMIT $2",
|
||||
str(session_id), limit,
|
||||
)
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
async def memory_similarity_search(
|
||||
self, query_text: str, limit: int = 10
|
||||
) -> list[dict]:
|
||||
from app.core import get_embedding
|
||||
query_vec = get_embedding(query_text)
|
||||
query_str = "[" + ",".join(str(x) for x in query_vec) + "]"
|
||||
async with self.connection() as conn:
|
||||
rows = await conn.fetch(
|
||||
"""SELECT id, content, memory_type, importance,
|
||||
(1 - (vector <-> $2::vector) / 2) as similarity
|
||||
FROM memories ORDER BY vector <-> $2 LIMIT $1""",
|
||||
limit, query_str,
|
||||
)
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
async def store_finding(
|
||||
self, session_id: str, question: str, answer: str,
|
||||
summary: str, agent_name: str, confidence: float,
|
||||
relevant_chunks: list | None = None
|
||||
) -> str:
|
||||
question_vec_str = "[" + ",".join(str(x) for x in self._extract_vector(answer)) + "]"
|
||||
finding_id = str(uuid.uuid4())
|
||||
async with self.connection() as conn:
|
||||
await conn.execute(
|
||||
"""INSERT INTO findings (id, session_id, question, answer,
|
||||
summary, vector, relevant_chunks, agent_name, confidence)
|
||||
VALUES ($1, $2, $3, $4, $5, $6::vector, $7::jsonb, $8, $9)""",
|
||||
finding_id, str(session_id), question, answer,
|
||||
summary, question_vec_str, json.dumps(relevant_chunks or []),
|
||||
agent_name, confidence,
|
||||
)
|
||||
return finding_id
|
||||
|
||||
async def get_findings(self, session_id: str) -> list[dict]:
|
||||
async with self.connection() as conn:
|
||||
rows = await conn.fetch(
|
||||
"SELECT * FROM findings WHERE session_id=$1 ORDER BY created_at DESC",
|
||||
str(session_id),
|
||||
)
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
async def create_session(self, query: str) -> str:
|
||||
session_id = str(uuid.uuid4())
|
||||
async with self.connection() as conn:
|
||||
await conn.execute(
|
||||
"INSERT INTO research_sessions (id, query, status, documents, findings) VALUES ($1, $2, $3, '[]', '[]')",
|
||||
session_id, query, "running",
|
||||
)
|
||||
return session_id
|
||||
|
||||
async def update_session(
|
||||
self, session_id: str, status: str | None = None,
|
||||
documents: list | dict | None = None, findings: list | dict | None = None
|
||||
):
|
||||
async with self.connection() as conn:
|
||||
if status:
|
||||
if documents or findings:
|
||||
await conn.execute(
|
||||
"UPDATE research_sessions SET status=$1, documents=$2::jsonb, findings=$3::jsonb, completed_at=NOW() WHERE id=$4",
|
||||
status, json.dumps(documents or []), json.dumps(findings or []),
|
||||
session_id,
|
||||
)
|
||||
else:
|
||||
await conn.execute(
|
||||
"UPDATE research_sessions SET status=$1 WHERE id=$2",
|
||||
status, session_id,
|
||||
)
|
||||
else:
|
||||
await conn.execute(
|
||||
"UPDATE research_sessions SET documents=$1::jsonb, findings=$2::jsonb WHERE id=$3",
|
||||
json.dumps(documents or []), json.dumps(findings or []),
|
||||
session_id,
|
||||
)
|
||||
|
||||
async def get_session(self, session_id: str) -> dict | None:
|
||||
async with self.connection() as conn:
|
||||
row = await conn.fetchrow(
|
||||
"SELECT * FROM research_sessions WHERE id=$1", session_id
|
||||
)
|
||||
return dict(row) if row else None
|
||||
|
||||
async def get_chunk(self, chunk_id: str) -> dict | None:
|
||||
async with self.connection() as conn:
|
||||
row = await conn.fetchrow(
|
||||
"SELECT * FROM chunks WHERE id=$1", chunk_id
|
||||
)
|
||||
return dict(row) if row else None
|
||||
|
||||
@staticmethod
|
||||
def _extract_vector(text: str) -> list[float]:
|
||||
"""Generate a lightweight embedding vector directly (no external call for speed)."""
|
||||
# Use a fast hash-based feature vector as fallback
|
||||
# In production this calls the server; locally we use a fast approximation
|
||||
import hashlib
|
||||
feature_dim = 4096
|
||||
vec = np.zeros(feature_dim, dtype=np.float32)
|
||||
# Create deterministic features from character trigrams
|
||||
trigrams = [text[i:i+3] for i in range(len(text)-2)]
|
||||
for i, tri in enumerate(trigrams):
|
||||
hash_val = hash(tri) & 0xFFFFFFFF
|
||||
# convert to signed 32-bit
|
||||
if hash_val >= 0x80000000:
|
||||
hash_val -= 0x100000000
|
||||
start_idx = (hash_val % feature_dim)
|
||||
end_idx = min(start_idx + 5, feature_dim)
|
||||
for j, byte in enumerate(hash_val.to_bytes(4, "big", signed=True)):
|
||||
idx = (start_idx + j) % feature_dim
|
||||
vec[idx] = (byte / 127.0) * np.sin(i * 0.1)
|
||||
norm = np.linalg.norm(vec)
|
||||
if norm > 0:
|
||||
vec = vec / norm
|
||||
return vec.tolist()
|
||||
|
||||
async def get_doc_chunks(self, doc_id: str) -> list[dict]:
|
||||
"""Get all chunks for a document."""
|
||||
async with self.connection() as conn:
|
||||
rows = await conn.fetch(
|
||||
"SELECT * FROM chunks WHERE doc_id=$1 ORDER BY page_num, block_index",
|
||||
str(doc_id),
|
||||
)
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
# ── Pipeline persistence ───────────────────────────────
|
||||
|
||||
async def save_pipeline_stage(
|
||||
self, session_id: str, stage: str, output: str, state: dict
|
||||
) -> str:
|
||||
"""Persist intermediate pipeline stage (triage/evidence/synthesis)."""
|
||||
stage_id = str(uuid.uuid4())
|
||||
async with self.connection() as conn:
|
||||
await conn.execute(
|
||||
"""INSERT INTO pipeline_stages (id, session_id, stage, output, state)
|
||||
VALUES ($1, $2, $3, $4, $5::jsonb)""",
|
||||
stage_id, str(session_id), stage, output, json.dumps(state),
|
||||
)
|
||||
return stage_id
|
||||
|
||||
async def get_pipeline_stages(self, session_id: str) -> list[dict]:
|
||||
"""Get all pipeline stages for a session in order."""
|
||||
async with self.connection() as conn:
|
||||
rows = await conn.fetch(
|
||||
"SELECT * FROM pipeline_stages WHERE session_id=$1 ORDER BY stage, created_at",
|
||||
str(session_id),
|
||||
)
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
async def save_structured_evidence(
|
||||
self, session_id: str,
|
||||
rows: list[dict]
|
||||
) -> list[str]:
|
||||
"""Save evidence-extraction rows with full traceability fields."""
|
||||
ids = []
|
||||
for row in rows:
|
||||
finding_id = str(uuid.uuid4())
|
||||
async with self.connection() as conn:
|
||||
await conn.execute(
|
||||
"""INSERT INTO findings (id, session_id, question, answer, summary,
|
||||
agent_name, confidence, relevant_chunks)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)""",
|
||||
finding_id, str(session_id),
|
||||
row.get("topic", ""),
|
||||
row.get("evidence", ""),
|
||||
row.get("description", ""),
|
||||
"evidence_extraction",
|
||||
{"High": 0.9, "Medium": 0.6, "Low": 0.3}.get(row.get("confidence", "Medium"), 0.6),
|
||||
json.dumps({
|
||||
"evidence_type": row.get("evidence_type"),
|
||||
"trace_ref": row.get("trace_ref"),
|
||||
"review_needed": row.get("review_needed", False),
|
||||
"confidence": row.get("confidence"),
|
||||
"source_doc": row.get("source_doc"),
|
||||
}),
|
||||
)
|
||||
ids.append(finding_id)
|
||||
return ids
|
||||
|
||||
async def get_structured_findings(self, session_id: str) -> list[dict]:
|
||||
"""Get structured evidence findings for a session."""
|
||||
async with self.connection() as conn:
|
||||
rows = await conn.fetch(
|
||||
"""SELECT f.*, f.relevant_chunks::jsonb as meta
|
||||
FROM findings f
|
||||
WHERE f.session_id=$1 AND f.agent_name='evidence_extraction'
|
||||
ORDER BY f.created_at""",
|
||||
str(session_id),
|
||||
)
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
|
||||
db: Database | None = None
|
||||
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
# Quick local dev mode - no podman
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$SCRIPT_DIR"
|
||||
|
||||
if [ ! -f ".env" ]; then
|
||||
cp .env.example .env
|
||||
fi
|
||||
|
||||
# Ensure workspace dirs exist
|
||||
mkdir -p workspace/documents
|
||||
|
||||
echo "Starting Agentic Research in dev mode..."
|
||||
echo " UI: http://localhost:8000"
|
||||
echo " Docs: http://localhost:8000/docs"
|
||||
echo " Marker API: http://localhost:8001"
|
||||
echo ""
|
||||
|
||||
# Start marker API in background
|
||||
cd marker-api
|
||||
python3 server.py &
|
||||
MARKER_PID=$!
|
||||
cd ../
|
||||
|
||||
sleep 1
|
||||
python3 -m uvicorn main:app --host 0.0.0.0 --port 8000 --reload
|
||||
|
||||
kill $MARKER_PID 2>/dev/null || true
|
||||
@@ -0,0 +1,675 @@
|
||||
:root {
|
||||
--bg-primary: #0d1117;
|
||||
--bg-secondary: #161b22;
|
||||
--bg-tertiary: #1c2333;
|
||||
--bg-card: #1a1f2e;
|
||||
--bg-hover: #252d3a;
|
||||
--bg-input: #0d1117;
|
||||
--border-color: #30363d;
|
||||
--border-active: #58a6ff;
|
||||
--text-primary: #e6edf3;
|
||||
--text-secondary: #8b949e;
|
||||
--text-muted: #484f58;
|
||||
--accent-blue: #58a6ff;
|
||||
--accent-green: #3fb950;
|
||||
--accent-purple: #a371f7;
|
||||
--accent-orange: #f0883e;
|
||||
--accent-red: #f85149;
|
||||
--accent-yellow: #d29922;
|
||||
--accent-cyan: #39d2c0;
|
||||
--sidebar-width: 320px;
|
||||
--toolbar-height: 44px;
|
||||
}
|
||||
|
||||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, sans-serif;
|
||||
background: var(--bg-primary);
|
||||
color: var(--text-primary);
|
||||
line-height: 1.5;
|
||||
overflow: hidden;
|
||||
height: 100vh;
|
||||
}
|
||||
|
||||
#app {
|
||||
display: flex;
|
||||
height: 100vh;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
/* ── Sidebar ───────────────────────────────────────────*/
|
||||
|
||||
.sidebar {
|
||||
width: var(--sidebar-width);
|
||||
min-width: var(--sidebar-width);
|
||||
background: var(--bg-secondary);
|
||||
border-right: 1px solid var(--border-color);
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow-y: auto;
|
||||
}
|
||||
|
||||
.sidebar-header {
|
||||
padding: 16px;
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
}
|
||||
|
||||
.sidebar-title {
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
color: var(--accent-blue);
|
||||
}
|
||||
|
||||
.sidebar-stats {
|
||||
font-size: 11px;
|
||||
color: var(--text-muted);
|
||||
margin-top: 4px;
|
||||
}
|
||||
|
||||
.sidebar-section {
|
||||
padding: 12px 16px;
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
}
|
||||
|
||||
.sidebar-section-header {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
.sidebar-label {
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
color: var(--text-secondary);
|
||||
}
|
||||
|
||||
.sidebar-actions { margin-top: 8px; }
|
||||
|
||||
.skill-toggles {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 4px;
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
.skill-toggle {
|
||||
font-size: 11px;
|
||||
padding: 3px 8px;
|
||||
background: var(--bg-tertiary);
|
||||
border-radius: 6px;
|
||||
cursor: pointer;
|
||||
color: var(--text-secondary);
|
||||
border: 1px solid transparent;
|
||||
transition: all 0.15s;
|
||||
}
|
||||
|
||||
.skill-toggle:hover { border-color: var(--border-color); color: var(--text-primary); }
|
||||
.skill-toggle input { display: none; }
|
||||
.skill-toggle input:checked + span,
|
||||
.skill-toggle:has(input:checked) {
|
||||
background: var(--accent-blue);
|
||||
color: #fff;
|
||||
border-color: var(--accent-blue);
|
||||
}
|
||||
|
||||
textarea, input[type="text"], input[type="number"], select {
|
||||
width: 100%;
|
||||
background: var(--bg-input);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 6px;
|
||||
color: var(--text-primary);
|
||||
padding: 8px 10px;
|
||||
font-size: 13px;
|
||||
font-family: inherit;
|
||||
outline: none;
|
||||
transition: border-color 0.15s;
|
||||
}
|
||||
|
||||
textarea:focus, input:focus, select:focus { border-color: var(--border-active); }
|
||||
|
||||
.btn {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
gap: 6px;
|
||||
padding: 6px 14px;
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 6px;
|
||||
font-size: 13px;
|
||||
font-weight: 500;
|
||||
cursor: pointer;
|
||||
background: var(--bg-tertiary);
|
||||
color: var(--text-primary);
|
||||
transition: all 0.15s;
|
||||
font-family: inherit;
|
||||
}
|
||||
|
||||
.btn:hover { background: var(--bg-hover); border-color: var(--text-muted); }
|
||||
.btn-primary { background: var(--accent-blue); color: #fff; border-color: var(--accent-blue); }
|
||||
.btn-primary:hover { opacity: 0.85; }
|
||||
.btn-secondary { background: var(--bg-tertiary); }
|
||||
.btn-small { padding: 4px 10px; font-size: 11px; }
|
||||
|
||||
.upload-label .btn { font-size: 11px; }
|
||||
|
||||
.doc-list { margin-top: 6px; }
|
||||
|
||||
.doc-item, .session-item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
padding: 6px 8px;
|
||||
border-radius: 4px;
|
||||
cursor: pointer;
|
||||
font-size: 12px;
|
||||
transition: background 0.1s;
|
||||
border-left: 2px solid transparent;
|
||||
}
|
||||
|
||||
.doc-item:hover, .session-item:hover { background: var(--bg-hover); }
|
||||
.doc-item.active, .session-item.active { background: var(--bg-tertiary); border-left-color: var(--accent-blue); }
|
||||
|
||||
.doc-item .file-icon { font-size: 14px; flex-shrink: 0; }
|
||||
.doc-item .file-name { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; flex: 1; }
|
||||
.doc-item .file-status { font-size: 10px; padding: 1px 5px; border-radius: 4px; flex-shrink: 0; }
|
||||
.doc-item .file-status.indexed { background: rgba(63,185,80,0.15); color: var(--accent-green); }
|
||||
.doc-item .file-status.processing { background: rgba(240,136,62,0.15); color: var(--accent-orange); }
|
||||
|
||||
.doc-item .file-remove {
|
||||
width: 16px; height: 16px;
|
||||
background: none; border: none;
|
||||
color: var(--text-muted);
|
||||
cursor: pointer;
|
||||
font-size: 14px;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
.doc-item .file-remove:hover { color: var(--accent-red); }
|
||||
|
||||
.sidebar-footer { padding: 12px; margin-top: auto; }
|
||||
.health-indicator {
|
||||
display: flex; align-items: center; gap: 6px;
|
||||
font-size: 11px; color: var(--text-secondary);
|
||||
}
|
||||
.health-dot {
|
||||
width: 7px; height: 7px; border-radius: 50%;
|
||||
background: var(--accent-red);
|
||||
}
|
||||
.health-dot.healthy { background: var(--accent-green); animation: pulse 2s infinite; }
|
||||
|
||||
@keyframes pulse {
|
||||
0%, 100% { opacity: 1; }
|
||||
50% { opacity: 0.4; }
|
||||
}
|
||||
|
||||
/* ── Main Content ───────────────────────────────────────*/
|
||||
|
||||
.main-content {
|
||||
flex: 1;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
background: var(--bg-primary);
|
||||
}
|
||||
|
||||
.toolbar {
|
||||
height: var(--toolbar-height);
|
||||
background: var(--bg-secondary);
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
display: flex;
|
||||
align-items: center;
|
||||
padding: 0 16px;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.toolbar-tabs { display: flex; gap: 2px; }
|
||||
|
||||
.tab {
|
||||
padding: 8px 16px;
|
||||
border: none;
|
||||
background: transparent;
|
||||
color: var(--text-secondary);
|
||||
font-size: 13px;
|
||||
cursor: pointer;
|
||||
border-bottom: 2px solid transparent;
|
||||
transition: all 0.15s;
|
||||
font-family: inherit;
|
||||
}
|
||||
|
||||
.tab:hover { color: var(--text-primary); }
|
||||
.tab.active { color: var(--accent-blue); border-bottom-color: var(--accent-blue); }
|
||||
|
||||
.tab-panes { flex: 1; overflow: hidden; position: relative; }
|
||||
.tab-pane { display: none; height: calc(100vh - var(--toolbar-height)); overflow-y: auto; padding: 16px; }
|
||||
.tab-pane.active { display: block; }
|
||||
|
||||
/* ── Analysis ─────────────────────────────────────────────────*/
|
||||
|
||||
.analysis-container { height: 100%; }
|
||||
|
||||
.research-output {
|
||||
background: var(--bg-card);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 8px;
|
||||
padding: 20px;
|
||||
height: 100%;
|
||||
overflow-y: auto;
|
||||
font-size: 13px;
|
||||
line-height: 1.6;
|
||||
}
|
||||
|
||||
.research-section { margin-bottom: 20px; }
|
||||
.research-section h3 { color: var(--accent-blue); font-size: 14px; margin-bottom: 8px; }
|
||||
.research-section h4 { color: var(--accent-purple); font-size: 13px; margin: 12px 0 6px; }
|
||||
.research-section p { margin-bottom: 8px; color: var(--text-primary); }
|
||||
|
||||
.research-section .citation {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
background: var(--bg-tertiary);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 4px;
|
||||
padding: 2px 8px;
|
||||
font-size: 11px;
|
||||
color: var(--accent-cyan);
|
||||
cursor: pointer;
|
||||
margin: 2px;
|
||||
}
|
||||
.research-section .citation:hover { border-color: var(--accent-blue); }
|
||||
|
||||
.research-section ul { padding-left: 18px; margin: 6px 0; }
|
||||
.research-section li { margin: 3px 0; }
|
||||
.research-section li::marker { color: var(--accent-blue); }
|
||||
|
||||
.research-section table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
margin: 8px 0;
|
||||
font-size: 12px;
|
||||
}
|
||||
.research-section th { text-align: left; padding: 6px 10px; background: var(--bg-tertiary); color: var(--text-secondary); border: 1px solid var(--border-color); font-weight: 600; font-size: 11px; }
|
||||
.research-section td { padding: 5px 10px; border: 1px solid var(--border-color); color: var(--text-primary); }
|
||||
|
||||
.research-section .block {
|
||||
background: var(--bg-tertiary);
|
||||
border-left: 3px solid var(--accent-blue);
|
||||
padding: 10px 14px;
|
||||
border-radius: 0 6px 6px 0;
|
||||
margin: 8px 0;
|
||||
}
|
||||
|
||||
.empty-state {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
height: 100%;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
.empty-icon { font-size: 40px; margin-bottom: 12px; opacity: 0.5; }
|
||||
.empty-text { font-size: 14px; }
|
||||
|
||||
/* ── Viewer ─────────────────────────────────────────────────*/
|
||||
|
||||
.viewer-container { height: 100%; display: flex; flex-direction: column; }
|
||||
|
||||
#viewer-toolbar {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
padding: 10px 14px;
|
||||
background: var(--bg-tertiary);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 8px;
|
||||
margin-bottom: 12px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
#viewer-doc-name { font-weight: 600; color: var(--accent-blue); font-size: 14px; }
|
||||
#viewer-page-info { color: var(--text-secondary); font-size: 12px; }
|
||||
|
||||
.viewer-controls {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
}
|
||||
|
||||
.page-input {
|
||||
width: 60px;
|
||||
text-align: center;
|
||||
padding: 4px 6px;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.viewer-toggle {
|
||||
display: flex;
|
||||
gap: 12px;
|
||||
margin-left: auto;
|
||||
}
|
||||
.viewer-toggle label {
|
||||
font-size: 12px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
color: var(--text-secondary);
|
||||
}
|
||||
.viewer-toggle input { cursor: pointer; }
|
||||
|
||||
.polygon-viewer {
|
||||
flex: 1;
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 8px;
|
||||
overflow: hidden;
|
||||
position: relative;
|
||||
background: #1a1a2e;
|
||||
}
|
||||
|
||||
.polygon-svg, .polygon-canvas {
|
||||
position: absolute;
|
||||
top: 0;
|
||||
left: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
}
|
||||
|
||||
/* ── Search ────────────────────────────────────────────*/
|
||||
|
||||
.search-container { max-width: 800px; }
|
||||
|
||||
.search-input-row {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
margin-bottom: 16px;
|
||||
}
|
||||
|
||||
.search-input { flex: 1; }
|
||||
.search-type-select { width: 120px; }
|
||||
|
||||
.search-results-list { }
|
||||
|
||||
.search-result-item {
|
||||
background: var(--bg-card);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 6px;
|
||||
padding: 12px;
|
||||
margin-bottom: 8px;
|
||||
cursor: pointer;
|
||||
transition: all 0.15s;
|
||||
}
|
||||
.search-result-item:hover {
|
||||
border-color: var(--accent-blue);
|
||||
background: var(--bg-hover);
|
||||
}
|
||||
.search-result-item .result-score { font-size: 11px; color: var(--accent-blue); margin-bottom: 4px; }
|
||||
.search-result-item .result-content { font-size: 13px; color: var(--text-primary); }
|
||||
.search-result-item .result-content mark { background: var(--accent-orange); color: #000; }
|
||||
.search-result-item .result-meta { font-size: 11px; color: var(--text-muted); margin-top: 4px; }
|
||||
|
||||
/* ── Memories ─────────────────────────────────────────*/
|
||||
|
||||
.memories-container { max-width: 800px; }
|
||||
.memories-container h3 { color: var(--accent-purple); margin-bottom: 12px; }
|
||||
|
||||
.memories-list { }
|
||||
|
||||
.memory-item {
|
||||
background: var(--bg-card);
|
||||
border: 1px solid var(--border-color);
|
||||
border-left: 3px solid var(--accent-purple);
|
||||
border-radius: 0 6px 6px 0;
|
||||
padding: 10px 14px;
|
||||
margin-bottom: 8px;
|
||||
font-size: 13px;
|
||||
}
|
||||
.memory-item .memory-type { font-size: 10px; color: var(--accent-orange); font-weight: 600; }
|
||||
.memory-item .memory-content { margin-top: 4px; color: var(--text-primary); }
|
||||
|
||||
/* ── Findings ────────────────────────────────────────────*/
|
||||
|
||||
.findings-container { max-width: 800px; }
|
||||
.findings-container h3 { color: var(--accent-cyan); margin-bottom: 12px; }
|
||||
|
||||
.findings-list { }
|
||||
.finding-item {
|
||||
background: var(--bg-card);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 6px;
|
||||
padding: 12px;
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
.finding-item .finding-agent { font-size: 11px; color: var(--accent-purple); font-weight: 600; }
|
||||
.finding-item .finding-query { font-size: 12px; color: var(--text-secondary); margin: 4px 0; }
|
||||
.finding-item .finding-answer { font-size: 13px; color: var(--text-primary); margin-top: 6px; }
|
||||
|
||||
/* ── Chat Overlay ───────────────────────────────────────
|
||||
|
||||
.chat-overlay {
|
||||
position: fixed;
|
||||
top: 0; right: 0;
|
||||
width: 420px;
|
||||
height: 100vh;
|
||||
background: var(--bg-secondary);
|
||||
border-left: 1px solid var(--border-color);
|
||||
z-index: 100;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
.chat-window {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
height: 100%;
|
||||
}
|
||||
|
||||
.chat-header {
|
||||
padding: 12px 16px;
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
font-size: 14px;
|
||||
font-weight: 600;
|
||||
color: var(--accent-blue);
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
}
|
||||
.chat-header .btn-close {
|
||||
background: none; border: none;
|
||||
color: var(--text-secondary);
|
||||
font-size: 20px;
|
||||
cursor: pointer;
|
||||
padding: 0 4px;
|
||||
}
|
||||
.chat-header .btn-close:hover { color: var(--text-primary); }
|
||||
|
||||
.chat-body {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 12px;
|
||||
}
|
||||
|
||||
.chat-message {
|
||||
margin-bottom: 12px;
|
||||
font-size: 13px;
|
||||
}
|
||||
.chat-message.sender { margin-left: 32px; }
|
||||
.chat-message.received { margin-right: 32px; }
|
||||
.chat-message .msg-label { font-size: 10px; font-weight: 600; margin-bottom: 2px; }
|
||||
.chat-message.sender .msg-label { color: var(--accent-purple); }
|
||||
.chat-message.received .msg-label { color: var(--accent-cyan); }
|
||||
|
||||
.chat-input-row {
|
||||
padding: 12px;
|
||||
border-top: 1px solid var(--border-color);
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.chat-input {
|
||||
flex: 1;
|
||||
padding: 8px 10px;
|
||||
}
|
||||
|
||||
/* ── Loading ──────────────────────────────────────
|
||||
|
||||
.spinner {
|
||||
display: inline-block;
|
||||
width: 16px; height: 16px;
|
||||
border: 2px solid var(--border-color);
|
||||
border-top-color: var(--accent-blue);
|
||||
border-radius: 50%;
|
||||
animation: spin 0.7s linear infinite;
|
||||
}
|
||||
@keyframes spin { to { transform: rotate(360deg); } }
|
||||
|
||||
.typing { color: var(--text-muted); font-style: italic; font-size: 12px; }
|
||||
|
||||
/* ── Pipeline ─────────────────────────────────────*/
|
||||
|
||||
.mode-section {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
.mode-toggle {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
font-size: 11px;
|
||||
color: var(--text-secondary);
|
||||
cursor: pointer;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.mode-toggle input { cursor: pointer; margin: 0; }
|
||||
|
||||
.mode-select {
|
||||
font-size: 11px;
|
||||
padding: 3px 8px;
|
||||
background: var(--bg-tertiary);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 4px;
|
||||
color: var(--text-primary);
|
||||
}
|
||||
|
||||
.pipeline-container { max-width: 960px; }
|
||||
|
||||
.pipeline-header-row {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
margin-bottom: 16px;
|
||||
}
|
||||
.pipeline-header-row h3 { color: var(--accent-green); margin: 0; }
|
||||
.pipeline-controls { display: flex; align-items: center; gap: 10px; }
|
||||
|
||||
.pipeline-output { }
|
||||
|
||||
.pipeline-stage-card {
|
||||
background: var(--bg-card);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 8px;
|
||||
margin-bottom: 16px;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.pipeline-stage-header {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
padding: 10px 16px;
|
||||
background: var(--bg-tertiary);
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
cursor: pointer;
|
||||
user-select: none;
|
||||
}
|
||||
.pipeline-stage-header:hover { background: var(--bg-hover); }
|
||||
.pipeline-stage-header .stage-title { font-size: 13px; font-weight: 600; }
|
||||
.pipeline-stage-header .stage-status { font-size: 11px; padding: 2px 8px; border-radius: 4px; }
|
||||
.pipeline-stage-header .stage-status.complete { background: rgba(63,185,80,0.15); color: var(--accent-green); }
|
||||
.pipeline-stage-header .stage-body { padding: 14px 16px; font-size: 13px; color: var(--text-primary); max-height: 800px; overflow-y: auto; }
|
||||
.pipeline-stage-header .stage-body.collapsed { display: none; }
|
||||
|
||||
.pipeline-section-label {
|
||||
font-size: 11px;
|
||||
font-weight: 600;
|
||||
color: var(--accent-blue);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
margin: 10px 0 4px;
|
||||
}
|
||||
|
||||
.pipeline-section-content {
|
||||
padding: 6px 0;
|
||||
color: var(--text-primary);
|
||||
}
|
||||
|
||||
.evidence-table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
margin: 8px 0;
|
||||
font-size: 12px;
|
||||
}
|
||||
.evidence-table th {
|
||||
text-align: left;
|
||||
padding: 6px 10px;
|
||||
background: var(--bg-tertiary);
|
||||
color: var(--text-secondary);
|
||||
border: 1px solid var(--border-color);
|
||||
font-weight: 600;
|
||||
font-size: 10px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.3px;
|
||||
}
|
||||
.evidence-table td {
|
||||
padding: 6px 10px;
|
||||
border: 1px solid var(--border-color);
|
||||
color: var(--text-primary);
|
||||
vertical-align: top;
|
||||
}
|
||||
.evidence-table td.confidence-high { color: var(--accent-green); }
|
||||
.evidence-table td.confidence-medium { color: var(--accent-yellow); }
|
||||
.evidence-table td.confidence-low { color: var(--accent-red); }
|
||||
|
||||
.synthesis-card {
|
||||
background: var(--bg-tertiary);
|
||||
border-left: 3px solid var(--accent-green);
|
||||
padding: 12px 16px;
|
||||
border-radius: 0 6px 6px 0;
|
||||
margin: 8px 0;
|
||||
}
|
||||
.synthesis-card h4 { color: var(--accent-green); font-size: 14px; margin: 0 0 8px; }
|
||||
|
||||
.gap-card {
|
||||
background: var(--bg-tertiary);
|
||||
border-left: 3px solid var(--accent-orange);
|
||||
padding: 10px 14px;
|
||||
border-radius: 0 6px 6px 0;
|
||||
margin: 6px 0;
|
||||
}
|
||||
.gap-card .gap-topic { font-weight: 600; color: var(--accent-orange); font-size: 12px; }
|
||||
.gap-card .gap-detail { font-size: 12px; color: var(--text-primary); margin-top: 4px; }
|
||||
|
||||
.matrix-table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
margin: 8px 0;
|
||||
font-size: 12px;
|
||||
}
|
||||
.matrix-table th {
|
||||
text-align: left;
|
||||
padding: 6px 10px;
|
||||
background: var(--bg-tertiary);
|
||||
color: var(--text-secondary);
|
||||
border: 1px solid var(--border-color);
|
||||
font-weight: 600;
|
||||
font-size: 10px;
|
||||
}
|
||||
.matrix-table td {
|
||||
padding: 5px 10px;
|
||||
border: 1px solid var(--border-color);
|
||||
color: var(--text-primary);
|
||||
}
|
||||
@@ -0,0 +1,218 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Agentic Research</title>
|
||||
<link rel="stylesheet" href="/static/css/styles.css">
|
||||
</head>
|
||||
<body>
|
||||
<div id="app">
|
||||
<!-- Sidebar -->
|
||||
<div id="sidebar" class="sidebar">
|
||||
<div class="sidebar-header">
|
||||
<h2 class="sidebar-title">Agentic Research</h2>
|
||||
<div class="sidebar-stats" id="sidebar-stats"></div>
|
||||
</div>
|
||||
|
||||
<!-- Query -->
|
||||
<div class="sidebar-section">
|
||||
<div class="sidebar-label">Research Query</div>
|
||||
<textarea id="query-input" placeholder="Enter your research question..." rows="3"></textarea>
|
||||
<div class="sidebar-actions">
|
||||
<div class="mode-section">
|
||||
<label class="mode-toggle">
|
||||
<input type="checkbox" id="pipeline-mode">
|
||||
<span>Pipeline Mode</span>
|
||||
</label>
|
||||
<select id="pipeline-output-mode" class="mode-select" style="display:none">
|
||||
<option value="">Auto (detect)</option>
|
||||
<option value="brief">Brief</option>
|
||||
<option value="report">Report</option>
|
||||
<option value="gap">Gap Analysis</option>
|
||||
<option value="matrix">Matrix</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="skill-toggles" id="skill-toggles">
|
||||
<label class="skill-toggle"><input type="checkbox" value="researcher" checked> Research</label>
|
||||
<label class="skill-toggle"><input type="checkbox" value="extractor"> Extract</label>
|
||||
<label class="skill-toggle"><input type="checkbox" value="summarizer"> Summarize</label>
|
||||
<label class="skill-toggle"><input type="checkbox" value="qa_agent"> QA</label>
|
||||
<label class="skill-toggle"><input type="checkbox" value="critic"> Critique</label>
|
||||
<label class="skill-toggle"><input type="checkbox" value="comparator"> Compare</label>
|
||||
<label class="skill-toggle"><input type="checkbox" value="aggregator"> Aggregate</label>
|
||||
<label class="skill-toggle"><input type="checkbox" value="context_agent"> Context</label>
|
||||
</div>
|
||||
</div>
|
||||
<button id="run-research" class="btn btn-primary">Run Research</button>
|
||||
</div>
|
||||
|
||||
<!-- Documents -->
|
||||
<div class="sidebar-section">
|
||||
<div class="sidebar-section-header">
|
||||
<span class="sidebar-label">Documents</span>
|
||||
<label class="upload-label">
|
||||
<input type="file" id="doc-upload" accept=".pdf,.txt,.md,.doc" style="display:none">
|
||||
<span class="btn btn-secondary">Upload</span>
|
||||
</label>
|
||||
</div>
|
||||
<div id="doc-list" class="doc-list"></div>
|
||||
</div>
|
||||
|
||||
<!-- Sessions -->
|
||||
<div class="sidebar-section">
|
||||
<div class="sidebar-section-header">
|
||||
<span class="sidebar-label">Sessions</span>
|
||||
</div>
|
||||
<div id="session-list" class="doc-list"></div>
|
||||
</div>
|
||||
|
||||
<!-- Models -->
|
||||
<div class="sidebar-section">
|
||||
<div class="sidebar-section-header">
|
||||
<span class="sidebar-label">Models</span>
|
||||
<button id="check-models" class="btn btn-small">Check</button>
|
||||
</div>
|
||||
<div id="model-list" class="doc-list"></div>
|
||||
</div>
|
||||
|
||||
<div class="sidebar-footer">
|
||||
<div id="health-indicator" class="health-indicator"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Main Content -->
|
||||
<div id="main" class="main-content">
|
||||
<div id="main-toolbar" class="toolbar">
|
||||
<div class="toolbar-tabs" id="main-tabs">
|
||||
<button class="tab active" data-tab="analysis">Analysis</button>
|
||||
<button class="tab" data-tab="pipeline">Pipeline</button>
|
||||
<button class="tab" data-tab="viewer">Document Viewer</button>
|
||||
<button class="tab" data-tab="search">Search</button>
|
||||
<button class="tab" data-tab="memories">Memories</button>
|
||||
<button class="tab" data-tab="findings">Findings</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Tabs -->
|
||||
<div id="tab-content" class="tab-panes">
|
||||
<!-- Analysis Tab -->
|
||||
<div id="tab-analysis" class="tab-pane active">
|
||||
<div class="analysis-container">
|
||||
<div id="research-output" class="research-output">
|
||||
<div class="empty-state">
|
||||
<div class="empty-icon">⚙</div>
|
||||
<div class="empty-text">Run a research query to see results</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Viewer Tab -->
|
||||
<div id="tab-viewer" class="tab-pane">
|
||||
<div class="viewer-container">
|
||||
<div id="viewer-toolbar">
|
||||
<span id="viewer-doc-name">No document selected</span>
|
||||
<span id="viewer-page-info">Page 0 / 0</span>
|
||||
<div class="viewer-controls">
|
||||
<button id="viewer-prev" class="btn btn-small">◀ Prev</button>
|
||||
<input id="viewer-page-input" type="number" min="0" value="0" class="page-input">
|
||||
<button id="viewer-next" class="btn btn-small">Next ▶</button>
|
||||
</div>
|
||||
<div class="viewer-toggle">
|
||||
<label><input type="checkbox" id="show-polygons" checked> Polygons</label>
|
||||
<label><input type="checkbox" id="show-text-layers" checked> Text</label>
|
||||
<label><input type="checkbox" id="show-chunk-labels" checked> Chunk labels</label>
|
||||
</div>
|
||||
</div>
|
||||
<div id="polygon-viewer" class="polygon-viewer">
|
||||
<svg id="polygon-svg" class="polygon-svg"></svg>
|
||||
<canvas id="polygon-canvas" class="polygon-canvas"></canvas>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Search Tab -->
|
||||
<div id="tab-search" class="tab-pane">
|
||||
<div class="search-container">
|
||||
<div class="search-input-row">
|
||||
<input id="search-query" type="text" placeholder="Search documents (text or semantic)..." class="search-input">
|
||||
<select id="search-type" class="search-type-select">
|
||||
<option value="semantic">Semantic</option>
|
||||
<option value="text">Text</option>
|
||||
</select>
|
||||
<button id="execute-search" class="btn btn-primary">Search</button>
|
||||
</div>
|
||||
<div id="search-results" class="search-results-list"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Memories Tab -->
|
||||
<div id="tab-memories" class="tab-pane">
|
||||
<div class="memories-container">
|
||||
<div class="memories-section">
|
||||
<h3>Saved Memories</h3>
|
||||
<div id="memories-list" class="memories-list"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Findings Tab -->
|
||||
<div id="tab-findings" class="tab-pane">
|
||||
<div class="findings-container">
|
||||
<h3>Research Findings</h3>
|
||||
<div id="findings-list" class="findings-list"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Pipeline Tab -->
|
||||
<div id="tab-pipeline" class="tab-pane">
|
||||
<div class="pipeline-container">
|
||||
<div class="pipeline-header-row">
|
||||
<h3>Research Pipeline</h3>
|
||||
<div class="pipeline-controls">
|
||||
<select id="pipeline-output-mode-tab" class="mode-select">
|
||||
<option value="">Auto (detect)</option>
|
||||
<option value="brief">Brief</option>
|
||||
<option value="report">Report</option>
|
||||
<option value="gap">Gap Analysis</option>
|
||||
<option value="matrix">Matrix</option>
|
||||
</select>
|
||||
<button id="rebuild-pipeline" class="btn btn-secondary">Rebuild Pipeline</button>
|
||||
</div>
|
||||
</div>
|
||||
<div id="pipeline-output" class="pipeline-output">
|
||||
<div class="empty-state">
|
||||
<div class="empty-icon">⚙</div>
|
||||
<div class="empty-text">Run a research query in Pipeline mode to see staged results here</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Chat overlay -->
|
||||
<div id="chat-overlay" class="chat-overlay" style="display:none">
|
||||
<div class="chat-window">
|
||||
<div class="chat-header">
|
||||
<span>Agent Chat</span>
|
||||
<button class="btn-close" id="chat-close">×</button>
|
||||
</div>
|
||||
<div class="chat-body" id="chat-body"></div>
|
||||
<div class="chat-input-row">
|
||||
<input id="chat-input" type="text" placeholder="Ask the agent..." class="chat-input">
|
||||
<button id="chat-send" class="btn btn-primary">Send</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script src="/static/js/store.js"></script>
|
||||
<script src="/static/js/utils.js"></script>
|
||||
<script src="/static/js/doc-viewer.js"></script>
|
||||
<script src="/static/js/research.js"></script>
|
||||
<script src="/static/js/ui.js"></script>
|
||||
<script src="/static/js/app.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,4 @@
|
||||
// App initialization
|
||||
document.addEventListener("DOMContentLoaded", () => {
|
||||
UI.init();
|
||||
});
|
||||
@@ -0,0 +1,105 @@
|
||||
// Document upload and list management
|
||||
const DOC_UI = {
|
||||
async init() {
|
||||
$("#doc-upload").addEventListener("change", (e) => this.handleUpload(e));
|
||||
|
||||
// Periodically refresh document list
|
||||
setInterval(() => this.refreshList(), 30000);
|
||||
},
|
||||
|
||||
async handleUpload(e) {
|
||||
const file = e.target.files[0];
|
||||
if (!file) return;
|
||||
|
||||
const output = $("#research-output");
|
||||
if (!output.querySelector(".empty-state")) output.innerHTML = "";
|
||||
|
||||
output.innerHTML = '<div class="spinner"></div> <span class="typing">Reading file...</span>';
|
||||
|
||||
const formData = new FormData();
|
||||
formData.append("file", file);
|
||||
|
||||
const res = await fetch("/api/documents/upload", { method: "POST", body: formData });
|
||||
const data = await res.json();
|
||||
|
||||
if (res.ok) {
|
||||
showToast(`Document indexed: ${data.chunks || "unknown"} chunks`, "info");
|
||||
STORE.documents = [];
|
||||
await this.refreshList();
|
||||
|
||||
// Select the new document
|
||||
if (data.doc_id) {
|
||||
STORE.currentDocId = data.doc_id;
|
||||
this.renderDocList();
|
||||
$("#viewer-doc-name").textContent = file.name;
|
||||
}
|
||||
} else {
|
||||
output.innerHTML = `<p style="color:var(--accent-red)">Upload failed: ${data.detail || data.error || "unknown"}</p>`;
|
||||
}
|
||||
|
||||
e.target.value = "";
|
||||
},
|
||||
|
||||
async refreshList() {
|
||||
try {
|
||||
const res = await API.get("/api/documents");
|
||||
STORE.documents = res.documents || [];
|
||||
this.renderDocList();
|
||||
|
||||
// Update stats
|
||||
const stats = $("#sidebar-stats");
|
||||
const total = STORE.documents.length;
|
||||
const indexed = STORE.documents.filter(d => d.status === "indexed").length;
|
||||
stats.textContent = `${indexed}/${total} documents indexed`;
|
||||
} catch (e) {
|
||||
console.warn("Doc list refresh failed:", e);
|
||||
}
|
||||
},
|
||||
|
||||
async selectDoc(docId) {
|
||||
STORE.currentDocId = docId;
|
||||
this.renderDocList();
|
||||
|
||||
// Load into viewer
|
||||
if (STORE.viewerDoc) {
|
||||
DOC_VIEWER.loadDocument(docId);
|
||||
}
|
||||
|
||||
// Switch to viewer tab
|
||||
$$(".tab").forEach(t => t.classList.remove("active"));
|
||||
$$(".tab-pane").forEach(p => p.classList.remove("active"));
|
||||
document.querySelector('[data-tab="viewer"]').classList.add("active");
|
||||
$("#tab-viewer").classList.add("active");
|
||||
},
|
||||
|
||||
renderDocList() {
|
||||
const list = $("#doc-list");
|
||||
if (!list) return;
|
||||
|
||||
if (!STORE.documents.length) {
|
||||
list.innerHTML = '<div style="font-size:12px;color:var(--text-muted);padding:8px">No documents</div>';
|
||||
return;
|
||||
}
|
||||
|
||||
list.innerHTML = STORE.documents.map(doc => {
|
||||
const ext = doc.filename.split(".").pop().toLowerCase();
|
||||
const icons = { pdf: "📄", txt: "📝", md: "📋", doc: "📑" };
|
||||
const icon = icons[ext] || "📄";
|
||||
const isActive = doc.id === STORE.currentDocId;
|
||||
return `
|
||||
<div class="doc-item ${isActive ? 'active' : ''}" data-doc-id="${doc.id}">
|
||||
<span class="file-icon">${icon}</span>
|
||||
<span class="file-name" title="${doc.filename}">${doc.filename.length > 25 ? doc.filename.substring(0, 25) + '...' : doc.filename}</span>
|
||||
<span class="file-status ${doc.status}">${doc.status.charAt(0).toUpperCase() + doc.status.slice(1)}</span>
|
||||
</div>`;
|
||||
}).join('');
|
||||
|
||||
// Click handlers
|
||||
list.querySelectorAll(".doc-item").forEach(item => {
|
||||
item.addEventListener("click", (e) => {
|
||||
if (e.target.closest(".file-remove")) return;
|
||||
this.selectDoc(item.dataset.docId);
|
||||
});
|
||||
});
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,298 @@
|
||||
// Doc viewer with polygon visualization
|
||||
const DOC_VIEWER = {
|
||||
svgEl: null,
|
||||
canvasEl: null,
|
||||
|
||||
init() {
|
||||
this.svgEl = $("#polygon-svg");
|
||||
this.canvasEl = $("#polygon-canvas");
|
||||
|
||||
$("#viewer-prev").addEventListener("click", () => this.goPage(-1));
|
||||
$("#viewer-next").addEventListener("click", () => this.goPage(1));
|
||||
$("#show-polygons").addEventListener("change", () => this.redraw());
|
||||
$("#show-text-layers").addEventListener("change", () => this.redraw());
|
||||
$("#show-chunk-labels").addEventListener("change", () => this.redraw());
|
||||
|
||||
$("input[id='viewer-page-input']").addEventListener("change", (e) => {
|
||||
const page = parseInt(e.target.value, 10);
|
||||
if (page >= 0 && page <= STORE.totalPages) {
|
||||
this.currentPage = page;
|
||||
this.loadPage(page);
|
||||
}
|
||||
});
|
||||
|
||||
// Click on polygon to show chunk info
|
||||
this.svgEl.addEventListener("click", (e) => {
|
||||
const target = e.target;
|
||||
if (target.dataset.chunkId) {
|
||||
this.showChunkInfo(target.dataset.chunkId);
|
||||
}
|
||||
});
|
||||
},
|
||||
|
||||
async loadDocument(docId) {
|
||||
STORE.currentDocId = docId;
|
||||
const res = await API.get(`/api/documents/${docId}/chunks`);
|
||||
STORE.viewerDoc = res.chunks || [];
|
||||
|
||||
// Compute page range
|
||||
const pages = new Set(STORE.viewerDoc.map(c => c.page_num));
|
||||
STORE.totalPages = Math.max(...pages) + 1;
|
||||
STORE.currentPage = 0;
|
||||
|
||||
$("#viewer-doc-name").textContent = STORE.documents.find(d => d.id === docId)?.filename || "Unknown";
|
||||
$("#viewer-page-input").max = STORE.totalPages - 1;
|
||||
this.updatePageInfo();
|
||||
|
||||
await this.loadPage(0);
|
||||
},
|
||||
|
||||
async loadPage(page) {
|
||||
STORE.currentPage = page;
|
||||
this.updatePageInfo();
|
||||
|
||||
// Get chunks for this page
|
||||
const chunks = STORE.viewerDoc.filter(c => c.page_num === page);
|
||||
if (!chunks.length) {
|
||||
$("#viewer-toolbar > span:nth-child(2)").textContent = "No content";
|
||||
this.clearSvg();
|
||||
return;
|
||||
}
|
||||
|
||||
// Get polygon data
|
||||
const polyRes = await API.get(`/api/documents/${STORE.currentDocId}/polygon-view?page=${page}`);
|
||||
STORE.polygons = polyRes.blocks || chunks.map((c, i) => ({
|
||||
...c,
|
||||
polygon: c.polygon || { x_min: 50, y_min: 50 + (i * 30), width: 900, height: 25 },
|
||||
}));
|
||||
|
||||
this.redraw();
|
||||
},
|
||||
|
||||
updatePageInfo() {
|
||||
$("#viewer-page-input").value = STORE.currentPage;
|
||||
$("#viewer-page-info").textContent = `Page ${STORE.currentPage + 1} / ${STORE.totalPages}`;
|
||||
},
|
||||
|
||||
goPage(dir) {
|
||||
const newPage = STORE.currentPage + dir;
|
||||
if (newPage >= 0 && newPage < STORE.totalPages) {
|
||||
this.loadPage(newPage);
|
||||
}
|
||||
},
|
||||
|
||||
clearSvg() {
|
||||
if (this.svgEl) {
|
||||
this.svgEl.innerHTML = "";
|
||||
}
|
||||
},
|
||||
|
||||
redraw() {
|
||||
const chunks = STORE.polygons.filter(c => c.polygon);
|
||||
if (!chunks.length || !this.svgEl) return;
|
||||
|
||||
this.clearSvg();
|
||||
|
||||
// Calculate bounding box from all polygons
|
||||
const bbox = chunks.reduce((acc, c) => {
|
||||
const p = c.polygon;
|
||||
if (!p || !p.bbox) return acc;
|
||||
const [x1, y1, x2, y2] = p.bbox;
|
||||
acc.x1 = Math.min(acc.x1, x1);
|
||||
acc.y1 = Math.min(acc.y1, y1);
|
||||
acc.x2 = Math.max(acc.x2, x2);
|
||||
acc.y2 = Math.max(acc.y2, y2);
|
||||
return acc;
|
||||
}, { x1: Infinity, y1: Infinity, x2: -Infinity, y2: -Infinity });
|
||||
|
||||
const padding = 20;
|
||||
const svgWidth = 1000;
|
||||
const svgHeight = 1200;
|
||||
const scaleX = svgWidth / Math.max(bbox.x2 - bbox.x1, 100);
|
||||
const scaleY = svgHeight / Math.max(bbox.y2 - bbox.y1, 100);
|
||||
const scale = Math.min(scaleX, scaleY);
|
||||
|
||||
this.svgEl.setAttribute("viewBox", `0 0 ${svgWidth} ${svgHeight}`);
|
||||
|
||||
// Page background
|
||||
const bg = document.createElementNS("http://www.w3.org/2000/svg", "rect");
|
||||
bg.setAttribute("x", padding);
|
||||
bg.setAttribute("y", padding);
|
||||
bg.setAttribute("width", svgWidth - padding * 2);
|
||||
bg.setAttribute("height", svgHeight - padding * 2);
|
||||
bg.setAttribute("fill", "#1e1e2e");
|
||||
bg.setAttribute("rx", "4");
|
||||
this.svgEl.appendChild(bg);
|
||||
|
||||
const showPolys = $("#show-polygons").checked;
|
||||
const showText = $("#show-text-layers").checked;
|
||||
const showLabels = $("#show-chunk-labels").checked;
|
||||
|
||||
// Draw each chunk as a polygon/rect
|
||||
chunks.forEach((chunk, idx) => {
|
||||
let rect;
|
||||
let x, y, w, h;
|
||||
|
||||
// Handle both bbox format [x1, y1, x2, y2] and explicit polygon format
|
||||
let coords;
|
||||
if (chunk.polygon.bbox && Array.isArray(chunk.polygon.bbox)) {
|
||||
const [x1, y1, x2, y2] = chunk.polygon.bbox;
|
||||
x = padding + (x1 - bbox.x1) * scale;
|
||||
y = padding + (y1 - bbox.y1) * scale;
|
||||
w = Math.max((x2 - x1) * scale, 10);
|
||||
h = Math.max((y2 - y1) * scale, 5);
|
||||
} else if (chunk.polygon.polygon && Array.isArray(chunk.polygon.polygon)) {
|
||||
const pol = chunk.polygon.polygon;
|
||||
const minX = Math.min(...pol.map(p => p[0]));
|
||||
const minY = Math.min(...pol.map(p => p[1]));
|
||||
const maxX = Math.max(...pol.map(p => p[0]));
|
||||
const maxY = Math.max(...pol.map(p => p[1]));
|
||||
x = padding + (minX - bbox.x1) * scale;
|
||||
y = padding + (minY - bbox.y1) * scale;
|
||||
w = Math.max((maxX - minX) * scale, 10);
|
||||
h = Math.max((maxY - minY) * scale, 5);
|
||||
} else if (chunk.polygon.x_min != null) {
|
||||
x = padding + (chunk.polygon.x_min - bbox.x1) * scale;
|
||||
y = padding + (chunk.polygon.y_min - bbox.y1) * scale;
|
||||
w = Math.max((chunk.polygon.width || 100) * scale, 50);
|
||||
h = Math.max((chunk.polygon.height || 20) * scale, 5);
|
||||
} else {
|
||||
return; // Skip if no polygon data
|
||||
}
|
||||
|
||||
// Polygon shape
|
||||
if (showPolys) {
|
||||
rect = document.createElementNS("http://www.w3.org/2000/svg", "rect");
|
||||
rect.setAttribute("x", x);
|
||||
rect.setAttribute("y", y);
|
||||
rect.setAttribute("width", w);
|
||||
rect.setAttribute("height", h);
|
||||
rect.setAttribute("rx", "2");
|
||||
|
||||
// Color by chunk type
|
||||
const colors = {
|
||||
"text": "#58a6ff",
|
||||
"figure": "#3fb950",
|
||||
"table": "#a371f7",
|
||||
"footnote": "#f0883e",
|
||||
"equation": "#f85149",
|
||||
"list": "#d29922",
|
||||
"caption": "#39d2c0",
|
||||
"title": "#e6edf3",
|
||||
};
|
||||
const color = colors[chunk.type] || colors["text"];
|
||||
|
||||
rect.setAttribute("fill", `${color}15`);
|
||||
rect.setAttribute("stroke", `${color}80`);
|
||||
rect.setAttribute("stroke-width", "1");
|
||||
rect.setAttribute("stroke-dasharray", "3,2");
|
||||
rect.dataset.chunkId = chunk.id;
|
||||
rect.style.cursor = "pointer";
|
||||
rect.addEventListener("mouseenter", () => {
|
||||
rect.setAttribute("stroke-width", "2");
|
||||
rect.setAttribute("fill", `${color}30`);
|
||||
});
|
||||
rect.addEventListener("mouseleave", () => {
|
||||
rect.setAttribute("stroke-width", "1");
|
||||
rect.setAttribute("fill", `${color}15`);
|
||||
});
|
||||
|
||||
// Store text content as tooltip
|
||||
const title = document.createElementNS("http://www.w3.org/2000/svg", "title");
|
||||
title.textContent = chunk.content ? chunk.content.substring(0, 100) : "";
|
||||
rect.appendChild(title);
|
||||
|
||||
this.svgEl.appendChild(rect);
|
||||
}
|
||||
|
||||
// Text label overlay
|
||||
if (showText && chunk.content) {
|
||||
const fontSize = Math.max(6, Math.min(12, h * 0.6));
|
||||
if (h > 12) {
|
||||
const textEl = document.createElementNS("http://www.w3.org/2000/svg", "text");
|
||||
textEl.setAttribute("x", x + 3);
|
||||
textEl.setAttribute("y", y + fontSize + 1);
|
||||
textEl.setAttribute("fill", "#e6edf3");
|
||||
textEl.setAttribute("font-size", fontSize);
|
||||
textEl.setAttribute("font-family", "monospace");
|
||||
|
||||
// Truncate text to fit
|
||||
const maxChars = Math.max(2, Math.floor(w / (fontSize * 0.55)));
|
||||
let displayText = chunk.content.substring(0, maxChars);
|
||||
if (chunk.content.length > maxChars) displayText += "...";
|
||||
|
||||
textEl.textContent = displayText;
|
||||
this.svgEl.appendChild(textEl);
|
||||
}
|
||||
}
|
||||
|
||||
// Chunk index label
|
||||
if (showLabels) {
|
||||
const label = document.createElementNS("http://www.w3.org/2000/svg", "text");
|
||||
label.setAttribute("x", x);
|
||||
label.setAttribute("y", Math.max(y - 4, 12));
|
||||
label.setAttribute("fill", `${color || "#58a6ff"}aa`);
|
||||
label.setAttribute("font-size", "8");
|
||||
label.setAttribute("font-family", "monospace");
|
||||
label.textContent = `${idx}`;
|
||||
this.svgEl.appendChild(label);
|
||||
}
|
||||
});
|
||||
|
||||
// Legend
|
||||
const legendY = svgHeight - 30;
|
||||
const legendBg = document.createElementNS("http://www.w3.org/2000/svg", "rect");
|
||||
legendBg.setAttribute("x", padding);
|
||||
legendBg.setAttribute("y", legendY);
|
||||
legendBg.setAttribute("width", "180");
|
||||
legendBg.setAttribute("height", "22");
|
||||
legendBg.setAttribute("fill", "var(--bg-primary)");
|
||||
legendBg.setAttribute("rx", "3");
|
||||
this.svgEl.appendChild(legendBg);
|
||||
|
||||
const legendText = document.createElementNS("http://www.w3.org/2000/svg", "text");
|
||||
legendText.setAttribute("x", padding + 8);
|
||||
legendText.setAttribute("y", legendY + 14);
|
||||
legendText.setAttribute("fill", "#8b949e");
|
||||
legendText.setAttribute("font-size", "9");
|
||||
legendText.textContent = "Click a region to view content";
|
||||
this.svgEl.appendChild(legendText);
|
||||
},
|
||||
|
||||
showChunkInfo(chunkId) {
|
||||
const chunk = STORE.viewerDoc.find(c => c.id === chunkId) ||
|
||||
STORE.polygons.find(c => c.id === chunkId);
|
||||
if (!chunk) return;
|
||||
|
||||
// Switch to analysis tab and show chunk
|
||||
const content = chunk.content || "(no text content)";
|
||||
const page = chunk.page_num ?? "?";
|
||||
|
||||
const div = document.createElement("div");
|
||||
div.className = "block";
|
||||
div.style.cursor = "pointer";
|
||||
div.innerHTML = `<div style="font-size:10px;color:var(--accent-cyan);margin-bottom:4px">[p${page}]${chunk.polygon ? ' (polygon)' : ''}</div><div style="font-size:12px;color:var(--text-primary)">${content.substring(0, 300)}</div>`;
|
||||
|
||||
div.addEventListener("click", () => {
|
||||
// Open in research tab
|
||||
const output = $("#research-output");
|
||||
const section = document.createElement("div");
|
||||
section.className = "research-section";
|
||||
section.innerHTML = `<h3>Chunk Viewer - Page ${page}</h3>
|
||||
<div style="background:var(--bg-tertiary);padding:10px;border-radius:6px;margin:8px 0;font-family:monospace;font-size:12px;color:var(--text-secondary);white-space:pre-wrap;max-height:300px;overflow-y:auto">${content}</div>`;
|
||||
|
||||
if (output.querySelector(".empty-state")) {
|
||||
output.innerHTML = "";
|
||||
}
|
||||
output.appendChild(section);
|
||||
|
||||
// Switch to analysis tab
|
||||
$$(".tab").forEach(t => t.classList.remove("active"));
|
||||
$$(".tab-pane").forEach(p => p.classList.remove("active"));
|
||||
document.querySelector('[data-tab="analysis"]').classList.add("active");
|
||||
$("#tab-analysis").classList.add("active");
|
||||
});
|
||||
|
||||
showToast(`Chunk p${page}: ${content.substring(0, 50)}...`, "info");
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,299 @@
|
||||
// Research orchestration UI
|
||||
const RESEARCH_UI = {
|
||||
async init() {
|
||||
$("#run-research").addEventListener("click", () => this.runResearch());
|
||||
$("#query-input").addEventListener("keydown", (e) => {
|
||||
if (e.key === "Enter" && e.ctrlKey) this.runResearch();
|
||||
});
|
||||
},
|
||||
|
||||
async runResearch() {
|
||||
const query = $("#query-input").value.trim();
|
||||
if (!query) return showToast("Enter a research query", "error");
|
||||
|
||||
// Get selected skills
|
||||
const checked = [...document.querySelectorAll('.skill-toggle input:checked')].map(c => c.value);
|
||||
|
||||
const output = $("#research-output");
|
||||
output.innerHTML = "";
|
||||
showSpinner(output);
|
||||
|
||||
// Create session
|
||||
let sessionId = STORE.currentSessionId;
|
||||
if (!sessionId) {
|
||||
try {
|
||||
const session = await API.post("/api/research/session", { query });
|
||||
STORE.currentSessionId = session.session_id;
|
||||
sessionId = session.session_id;
|
||||
} catch (e) {
|
||||
showToast("Failed to create session", "error");
|
||||
output.innerHTML = `<p class='typing'>Session error: ${e.message}</p>`;
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
// Run research
|
||||
try {
|
||||
const results = await API.post("/api/research/run", {
|
||||
query,
|
||||
doc_id: STORE.currentDocId,
|
||||
skills: checked.length ? checked : undefined,
|
||||
});
|
||||
|
||||
output.innerHTML = "";
|
||||
|
||||
// Session info
|
||||
const info = document.createElement("div");
|
||||
info.className = "research-section";
|
||||
info.innerHTML = `
|
||||
<div style="display:flex;justify-content:space-between;align-items:center">
|
||||
<h3 style="color:var(--accent-blue)">Research Results</h3>
|
||||
<span style="font-size:11px;color:var(--text-muted)">Session: ${sessionId.substring(0, 8)}</span>
|
||||
</div>
|
||||
<p style="color:var(--text-secondary)">Query: ${query}</p>
|
||||
<hr style="border:none;border-top:1px solid var(--border-color);margin:8px 0">`;
|
||||
output.appendChild(info);
|
||||
|
||||
// Results from each skill
|
||||
if (results.results) {
|
||||
for (const [skill, response] of Object.entries(results.results)) {
|
||||
const section = document.createElement("div");
|
||||
section.className = "research-section";
|
||||
section.id = `section-${skill}`;
|
||||
section.innerHTML = `<h3>${skill.replace(/_/g, ' ').replace(/\b\w/g, c => c.toUpperCase())}</h3>`;
|
||||
|
||||
const content = document.createElement("div");
|
||||
content.innerHTML = renderContent(response);
|
||||
section.appendChild(content);
|
||||
output.appendChild(section);
|
||||
}
|
||||
}
|
||||
|
||||
STORE.currentSessionId = sessionId;
|
||||
showToast("Research complete", "info");
|
||||
|
||||
} catch (e) {
|
||||
output.innerHTML = `<p class='typing' style="color:var(--accent-red)">Error: ${e.message}</p>`;
|
||||
showToast("Research failed", "error");
|
||||
}
|
||||
},
|
||||
|
||||
async runSemanticSearch(query, docId = null) {
|
||||
const results = await API.post("/api/research/semantic", { query, doc_id: docId, limit: 20 });
|
||||
return results.results || [];
|
||||
},
|
||||
|
||||
async runTextSearch(query, docId = null) {
|
||||
const results = await API.post("/api/research/text-search", { query, doc_id: docId, limit: 20 });
|
||||
return results.results || [];
|
||||
},
|
||||
|
||||
async getFindings(sessionId) {
|
||||
if (!sessionId) return [];
|
||||
try {
|
||||
const res = await API.get(`/api/research/findings?session_id=${sessionId}`);
|
||||
return res.findings || [];
|
||||
} catch { return []; }
|
||||
},
|
||||
|
||||
async saveMemory(sessionId, content, type = "fact", importance = 3) {
|
||||
if (!sessionId) return;
|
||||
try {
|
||||
await API.post("/api/memories/save", {
|
||||
session_id: sessionId,
|
||||
content,
|
||||
memory_type: type,
|
||||
importance,
|
||||
source_doc_id: STORE.currentDocId,
|
||||
});
|
||||
showToast("Memory saved", "info");
|
||||
} catch (e) { showToast("Save failed: " + e.message, "error"); }
|
||||
},
|
||||
|
||||
async runPipeline() {
|
||||
const query = $("#query-input").value.trim();
|
||||
if (!query) return showToast("Enter a research query", "error");
|
||||
|
||||
const mode = $("#pipeline-output-mode").value || "";
|
||||
|
||||
// Create session
|
||||
let sessionId = STORE.currentSessionId;
|
||||
if (!sessionId) {
|
||||
try {
|
||||
const session = await API.post("/api/research/session", { query });
|
||||
sessionId = session.session_id;
|
||||
STORE.currentSessionId = sessionId;
|
||||
} catch (e) {
|
||||
showToast("Failed to create session", "error");
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const pipelineEl = $("#pipeline-output");
|
||||
pipelineEl.innerHTML = '<div class="spinner"></div> <span class="typing">Running: triage → evidence extraction → synthesis...</span>';
|
||||
|
||||
// Run the pipeline
|
||||
try {
|
||||
const results = await API.post("/api/research/pipeline", {
|
||||
query,
|
||||
doc_ids: STORE.currentDocId ? [STORE.currentDocId] : undefined,
|
||||
output_mode: mode || undefined,
|
||||
});
|
||||
|
||||
// Save session to the first doc or temp session
|
||||
STORE.currentSessionId = sessionId;
|
||||
|
||||
// Render pipeline sections
|
||||
// First render: get structured sections from results
|
||||
const sections = RESEARCH_UI.renderPipelineSections(results);
|
||||
pipelineEl.innerHTML = sections.map(s => RESEARCH_UI.renderSectionHtml(s)).join('');
|
||||
|
||||
// Wire up collapse toggles
|
||||
pipelineEl.querySelectorAll(".pipeline-stage-header").forEach(header => {
|
||||
header.addEventListener("click", () => {
|
||||
const body = header.nextElementSibling;
|
||||
body.classList.toggle("collapsed");
|
||||
const arrow = header.querySelector(".stage-arrow");
|
||||
if (arrow) arrow.textContent = body.classList.contains("collapsed") ? "▶" : "▼";
|
||||
});
|
||||
});
|
||||
|
||||
showToast("Pipeline complete", "info");
|
||||
|
||||
} catch (e) {
|
||||
pipelineEl.innerHTML = `<p class='typing' style="color:var(--accent-red)">Error: ${e.message}</p>`;
|
||||
showToast("Pipeline failed", "error");
|
||||
}
|
||||
},
|
||||
|
||||
renderPipelineSections(results) {
|
||||
const sections = [];
|
||||
|
||||
// Triage stage
|
||||
if (results.triage_state) {
|
||||
const plan = results.triage_state;
|
||||
sections.push({
|
||||
stage: "triage",
|
||||
title: "Stage 1: Source Triage",
|
||||
status: "complete",
|
||||
sections: [
|
||||
{ label: "Objective", content: plan.OBJECTIVE || "" },
|
||||
{ label: "Sub-questions", content: plan["SUB-QUESTIONS"] || "" },
|
||||
{ label: "Classification", content: plan.CLASSIFICATION || "" },
|
||||
{ label: "Reading Order", content: plan.READING_ORDER || "" },
|
||||
{ label: "Extraction Criteria", content: plan["EXTRACTION CRITERIA"] || "" },
|
||||
{ label: "Risks & Gaps", content: plan["RISKS and GAPS"] || plan.RISKS_AND_GAPS || "" },
|
||||
],
|
||||
});
|
||||
}
|
||||
|
||||
// Evidence stage
|
||||
if (results.evidence_rows && results.evidence_rows.length > 0) {
|
||||
sections.push({
|
||||
stage: "evidence",
|
||||
title: `Stage 2: Extracted Evidence (${results.evidence_rows.length} rows)`,
|
||||
status: "complete",
|
||||
sections: [
|
||||
{ label: "", content: "table", rows: results.evidence_rows },
|
||||
],
|
||||
});
|
||||
}
|
||||
|
||||
// Synthesis stage
|
||||
if (results.synthesis) {
|
||||
sections.push({
|
||||
stage: "synthesis",
|
||||
title: `Stage 3: Synthesis (mode: ${results.output_mode || "auto"})`,
|
||||
status: "complete",
|
||||
sections: [
|
||||
{ label: "Result", content: results.synthesis, mode: results.output_mode },
|
||||
],
|
||||
});
|
||||
}
|
||||
|
||||
return sections;
|
||||
},
|
||||
|
||||
renderSectionHtml(section) {
|
||||
let cardsHtml = "";
|
||||
section.sections.forEach(sec => {
|
||||
if (sec.content === "table") {
|
||||
const rows = sec.rows || [];
|
||||
const hasComparison = rows[0] && rows[0].source_a !== undefined;
|
||||
if (hasComparison) {
|
||||
cardsHtml += `<div class="pipeline-section-label">Cross-Comparison</div>`;
|
||||
cardsHtml += `<table class="matrix-table"><thead><tr><th>Topic</th><th>Source A</th><th>Source B</th><th>Agreement</th><th>Conflict</th><th>Notes</th></tr></thead><tbody>`;
|
||||
rows.forEach(r => {
|
||||
cardsHtml += `<tr>
|
||||
<td>${r.topic || ""}</td>
|
||||
<td>${(r.source_a || "").substring(0, 200)}</td>
|
||||
<td>${(r.source_b || "").substring(0, 200)}</td>
|
||||
<td>${r.agreement || ""}</td>
|
||||
<td>${r.conflict || ""}</td>
|
||||
<td>${r.notes || ""}</td>
|
||||
</tr>`;
|
||||
});
|
||||
cardsHtml += `</tbody></table>`;
|
||||
} else {
|
||||
const cols = ["#", "Topic", "Evidence Type", "Description", "Doc Ref", "Evidence", "Analyst Note", "Confidence", "Review"];
|
||||
cardsHtml += `<div class="pipeline-section-label">Extracted Evidence</div>`;
|
||||
cardsHtml += `<table class="evidence-table"><thead><tr>${cols.map(c => `<th>${c}</th>`).join("")}</tr></thead><tbody>`;
|
||||
rows.forEach((r, i) => {
|
||||
const confClass = (r.confidence || "medium").toLowerCase().replace(" ", "-");
|
||||
cardsHtml += `<tr>
|
||||
<td>${i + 1}</td>
|
||||
<td>${r.topic || ""}</td>
|
||||
<td>${r.evidence_type || ""}</td>
|
||||
<td>${(r.description || "").substring(0, 150)}</td>
|
||||
<td>${(r.trace_ref || "").substring(0, 150)}</td>
|
||||
<td>${(r.evidence || "").substring(0, 200)}</td>
|
||||
<td>${(r.analyst_note || "").substring(0, 150)}</td>
|
||||
<td class="confidence-${confClass}">${r.confidence || "Medium"}</td>
|
||||
<td>${r.review_needed || "No"}</td>
|
||||
</tr>`;
|
||||
});
|
||||
cardsHtml += `</tbody></table>`;
|
||||
}
|
||||
} else {
|
||||
cardsHtml += `<div class="pipeline-section-label">${sec.label || ""}</div>`;
|
||||
cardsHtml += `<div class="pipeline-section-content">${sec.content}</div>`;
|
||||
}
|
||||
});
|
||||
|
||||
return `
|
||||
<div class="pipeline-stage-card">
|
||||
<div class="pipeline-stage-header" data-stage="${section.stage}">
|
||||
<span class="stage-title">${section.title}</span>
|
||||
<span><span class="stage-arrow" style="margin-right:6px;">▼</span><span class="stage-status ${section.status}">Complete</span></span>
|
||||
</div>
|
||||
<div class="stage-body">${cardsHtml}</div>
|
||||
</div>`;
|
||||
},
|
||||
|
||||
async loadPipelineSession(sessionId) {
|
||||
if (!sessionId) return;
|
||||
const pipelineEl = $("#pipeline-output");
|
||||
|
||||
try {
|
||||
const res = await API.get(`/api/research/pipeline/${sessionId}/render`);
|
||||
const sections = res.sections || [];
|
||||
|
||||
pipelineEl.innerHTML = sections.length > 0
|
||||
? sections.map(s => this.renderSectionHtml(s)).join('')
|
||||
: "<div class='empty-state'><div class='empty-icon'>ℹ️</div><div class='empty-text'>No pipeline data for this session</div></div>";
|
||||
|
||||
// Wire up collapse toggles
|
||||
pipelineEl.querySelectorAll(".pipeline-stage-header").forEach(header => {
|
||||
header.addEventListener("click", () => {
|
||||
const body = header.nextElementSibling;
|
||||
body.classList.toggle("collapsed");
|
||||
const arrow = header.querySelector(".stage-arrow");
|
||||
if (arrow) arrow.textContent = body.classList.contains("collapsed") ? "▶" : "▼";
|
||||
});
|
||||
});
|
||||
} catch (e) {
|
||||
pipelineEl.innerHTML = `<p class='typing' style="color:var(--accent-red)">Error loading pipeline: ${e.message}</p>`;
|
||||
}
|
||||
},
|
||||
};
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
// Store - Application state management
|
||||
const Store = {
|
||||
currentDoc: null,
|
||||
currentSession: null,
|
||||
documents: [],
|
||||
sessions: [],
|
||||
memories: [],
|
||||
findings: [],
|
||||
pageChunks: [],
|
||||
currentPage: 0,
|
||||
polygonsVisible: true,
|
||||
textVisible: true,
|
||||
chunkLabelsVisible: true,
|
||||
};
|
||||
|
||||
// ── API Helper ──
|
||||
async function api(endpoint, options = {}) {
|
||||
const defaultOpts = {
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
};
|
||||
const merged = { ...defaultOpts, ...options };
|
||||
if (options.body && typeof options.body === 'object' && !(options.body instanceof FormData)) {
|
||||
merged.body = JSON.stringify(options.body);
|
||||
}
|
||||
const url = endpoint.startsWith('http') ? endpoint : `/api${endpoint}`;
|
||||
const resp = await fetch(url, merged);
|
||||
if (!resp.ok) {
|
||||
const text = await resp.text();
|
||||
throw new Error(`${resp.status}: ${text}`);
|
||||
}
|
||||
const ct = resp.headers.get('content-type') || '';
|
||||
if (ct.includes('json')) return resp.json();
|
||||
return resp;
|
||||
}
|
||||
|
||||
// ── Document Store ──
|
||||
const DocStore = {
|
||||
async load() {
|
||||
const res = await api('/documents');
|
||||
Store.documents = res.documents || [];
|
||||
return Store.documents;
|
||||
},
|
||||
|
||||
async getChunks(docId, page) {
|
||||
const params = new URLSearchParams();
|
||||
if (page !== undefined) params.set('page', page);
|
||||
const res = await api(`/documents/${docId}/chunks?${params}`);
|
||||
Store.pageChunks = res.chunks || [];
|
||||
return Store.pageChunks;
|
||||
},
|
||||
|
||||
async upload(file) {
|
||||
const fd = new FormData();
|
||||
fd.append('file', file);
|
||||
const res = await api('/documents/upload', { method: 'POST', body: fd });
|
||||
return res;
|
||||
},
|
||||
|
||||
async uploadUrl(url, filename) {
|
||||
const res = await api('/documents/upload', {
|
||||
method: 'POST',
|
||||
body: { pdf_url: url, filename },
|
||||
});
|
||||
return res;
|
||||
},
|
||||
};
|
||||
|
||||
// ── Research Store ──
|
||||
const ResearchStore = {
|
||||
async run(query, skillNames) {
|
||||
const res = await api('/research/run', {
|
||||
method: 'POST',
|
||||
body: { query, skills: skillNames, doc_id: Store.currentDoc },
|
||||
});
|
||||
return res;
|
||||
},
|
||||
|
||||
async createSession(query) {
|
||||
const res = await api('/research/session', {
|
||||
method: 'POST',
|
||||
body: query,
|
||||
});
|
||||
return res;
|
||||
},
|
||||
|
||||
async semanticSearch(query, limit = 20) {
|
||||
const res = await api('/research/semantic', {
|
||||
method: 'POST',
|
||||
body: { query, doc_id: Store.currentDoc, limit },
|
||||
});
|
||||
return res;
|
||||
},
|
||||
|
||||
async textSearch(query, limit = 20) {
|
||||
const res = await api('/research/text-search', {
|
||||
method: 'POST',
|
||||
body: { query, doc_id: Store.currentDoc, limit },
|
||||
});
|
||||
return res;
|
||||
},
|
||||
|
||||
async getFindings(sessionId) {
|
||||
const res = await api(`/research/findings?session_id=${sessionId || ''}`);
|
||||
Store.findings = res.findings || [];
|
||||
return Store.findings;
|
||||
},
|
||||
|
||||
async getMemories(sessionId) {
|
||||
const res = await api(`/research/memories?session_id=${sessionId || ''}`);
|
||||
Store.memories = res.memories || [];
|
||||
return Store.memories;
|
||||
},
|
||||
|
||||
async saveFinding(sessionId, query, response) {
|
||||
await api('/research/run', {
|
||||
method: 'POST',
|
||||
body: { query, doc_id: sessionId },
|
||||
});
|
||||
},
|
||||
};
|
||||
|
||||
// ── Memory Store ──
|
||||
const MemoryStore = {
|
||||
async save(content, type = 'fact', importance = 3, sourceDocId) {
|
||||
const res = await api('/memories/save', {
|
||||
method: 'POST',
|
||||
body: {
|
||||
session_id: Store.currentSession,
|
||||
content,
|
||||
memory_type: type,
|
||||
importance,
|
||||
source_doc_id: sourceDocId,
|
||||
},
|
||||
});
|
||||
return res;
|
||||
},
|
||||
|
||||
async search(query, limit = 10) {
|
||||
const res = await api('/memories/search', {
|
||||
method: 'POST',
|
||||
body: { query, limit },
|
||||
});
|
||||
return res;
|
||||
},
|
||||
};
|
||||
|
||||
// ── Agent Store ──
|
||||
const AgentStore = {
|
||||
async getSkills() {
|
||||
const res = await api('/agents/skills');
|
||||
return res.skills;
|
||||
},
|
||||
|
||||
async chat(message, model) {
|
||||
const res = await api('/ollama/chat', {
|
||||
method: 'POST',
|
||||
body: {
|
||||
model: model || 'gpt-oss:20b',
|
||||
messages: [{ role: 'user', content: message }],
|
||||
},
|
||||
});
|
||||
return res;
|
||||
},
|
||||
};
|
||||
|
||||
// ── Viewer Store ──
|
||||
const ViewerStore = {
|
||||
async getPolygonView(docId, page) {
|
||||
const res = await api(`/documents/${docId}/polygon-view?page=${page}`);
|
||||
return res;
|
||||
},
|
||||
|
||||
async getPageText(docId, page) {
|
||||
const res = await api(`/documents/${docId}/page-text?page=${page}`);
|
||||
return res;
|
||||
},
|
||||
};
|
||||
|
||||
// ── Health Store ──
|
||||
const HealthStore = {
|
||||
async check() {
|
||||
const res = await api('/health');
|
||||
return res;
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,198 @@
|
||||
// Main UI management
|
||||
const UI = {
|
||||
async init() {
|
||||
// Tab switching
|
||||
$$(".tab").forEach(tab => {
|
||||
tab.addEventListener("click", () => {
|
||||
const target = tab.dataset.tab;
|
||||
this.switchTab(target);
|
||||
|
||||
// Load tab data
|
||||
if (target === "search") this.initSearch();
|
||||
if (target === "memories") this.loadMemories();
|
||||
if (target === "findings") this.loadFindings();
|
||||
if (target === "pipeline" && STORE.currentSessionId) this.loadPipelineSession(STORE.currentSessionId);
|
||||
});
|
||||
});
|
||||
|
||||
// Pipeline mode toggle
|
||||
const pipelineCheck = $("#pipeline-mode");
|
||||
const pipelineModeSelect = $("#pipeline-output-mode");
|
||||
if (pipelineCheck) {
|
||||
pipelineCheck.addEventListener("change", () => {
|
||||
const on = pipelineCheck.checked;
|
||||
pipelineModeSelect.style.display = on ? "inline-block" : "none";
|
||||
const toggles = document.querySelectorAll("#skill-toggles .skill-toggle");
|
||||
toggles.forEach(t => {
|
||||
const cb = t.querySelector("input");
|
||||
if (cb) cb.disabled = on;
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
// Rebuild pipeline button
|
||||
const rebuildBtn = $("#rebuild-pipeline");
|
||||
if (rebuildBtn && STORE.currentSessionId) {
|
||||
rebuildBtn.addEventListener("click", () => UI.loadPipelineSession(STORE.currentSessionId));
|
||||
}
|
||||
|
||||
// Search
|
||||
$("#execute-search").addEventListener("click", () => this.executeSearch());
|
||||
$("#search-query").addEventListener("keydown", (e) => {
|
||||
if (e.key === "Enter") this.executeSearch();
|
||||
});
|
||||
|
||||
// Check models
|
||||
$("#check-models").addEventListener("click", () => this.checkModels());
|
||||
|
||||
// Load state
|
||||
await loadInitialState();
|
||||
|
||||
// Init sub-modules
|
||||
DOC_VIEWER.init();
|
||||
RESEARCH_UI.init();
|
||||
DOC_UI.init();
|
||||
},
|
||||
|
||||
switchTab(tabName) {
|
||||
$$(".tab").forEach(t => t.classList.remove("active"));
|
||||
$$(".tab-pane").forEach(p => p.classList.remove("active"));
|
||||
document.querySelector(`[data-tab="${tabName}"]`).classList.add("active");
|
||||
$(`#tab-${tabName}`).classList.add("active");
|
||||
},
|
||||
|
||||
initSearch() {
|
||||
// Search already bound in HTML
|
||||
},
|
||||
|
||||
async executeSearch() {
|
||||
const query = $("#search-query").value.trim();
|
||||
if (!query) return showToast("Enter a search query", "error");
|
||||
|
||||
const type = $("#search-type").value;
|
||||
const resultsEl = $("#search-results");
|
||||
resultsEl.innerHTML = '<div class="spinner"></div> <span class="typing">Searching...</span>';
|
||||
|
||||
try {
|
||||
let results;
|
||||
if (type === "semantic") {
|
||||
results = await RESEARCH_UI.runSemanticSearch(query, STORE.currentDocId);
|
||||
} else {
|
||||
results = await RESEARCH_UI.runTextSearch(query, STORE.currentDocId);
|
||||
}
|
||||
|
||||
if (!results.length) {
|
||||
resultsEl.innerHTML = '<div style="padding:20px;color:var(--text-muted);text-align:center">No results found</div>';
|
||||
return;
|
||||
}
|
||||
|
||||
resultsEl.innerHTML = results.map(r => {
|
||||
const score = r.similarity ?? r.rank ?? 0;
|
||||
const page = r.page_num != null ? `p${r.page_num}` : "";
|
||||
const truncated = r.content ? r.content.substring(0, 200) : "";
|
||||
|
||||
// Highlight matching text
|
||||
let content = truncated.replace(/&/g, '&')
|
||||
.replace(/</g, '<')
|
||||
.replace(/>/g, '>');
|
||||
if (query) {
|
||||
const regex = new RegExp(`(${query.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')})`, 'gi');
|
||||
content = content.replace(regex, '<mark>$1</mark>');
|
||||
}
|
||||
|
||||
return `
|
||||
<div class="search-result-item" data-chunk-id="${r.id}">
|
||||
<div class="result-score">Score: ${score.toFixed(3)}</div>
|
||||
<div class="result-content">${content}${truncated.length >= 200 ? '...' : ''}</div>
|
||||
<div class="result-meta">${page} · ${r.chunk_type || 'text'}</div>
|
||||
</div>`;
|
||||
}).join('');
|
||||
|
||||
// Click to show chunk in viewer
|
||||
resultsEl.querySelectorAll(".search-result-item").forEach(item => {
|
||||
item.addEventListener("click", () => {
|
||||
const chunkId = item.dataset.chunkId;
|
||||
if (STORE.currentDocId) {
|
||||
DOC_VIEWER.showChunkInfo(chunkId);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
} catch (e) {
|
||||
resultsEl.innerHTML = `<p style="color:var(--accent-red);padding:20px">Search error: ${e.message}</p>`;
|
||||
}
|
||||
},
|
||||
|
||||
async loadMemories() {
|
||||
const list = $("#memories-list");
|
||||
list.innerHTML = '<div class="spinner"></div>';
|
||||
|
||||
try {
|
||||
if (STORE.currentSessionId) {
|
||||
const res = await API.get(`/api/research/memories?session_id=${STORE.currentSessionId}`);
|
||||
list.innerHTML = (res.memories || []).map(m => `
|
||||
<div class="memory-item">
|
||||
<div class="memory-type">${m.memory_type || "fact"} (imp: ${m.importance || "?"})</div>
|
||||
<div class="memory-content">${m.content ? m.content.substring(0, 300) : "(empty)"}</div>
|
||||
</div>`).join('') || '<div style="color:var(--text-muted);padding:12px">No memories</div>';
|
||||
} else {
|
||||
list.innerHTML = '<div style="color:var(--text-muted);padding:12px">No active session</div>';
|
||||
}
|
||||
} catch (e) {
|
||||
list.innerHTML = `<div style="color:var(--accent-red);padding:12px">${e.message}</div>`;
|
||||
}
|
||||
},
|
||||
|
||||
async loadFindings() {
|
||||
const list = $("#findings-list");
|
||||
list.innerHTML = '<div class="spinner"></div>';
|
||||
|
||||
try {
|
||||
const findings = await RESEARCH_UI.getFindings(STORE.currentSessionId);
|
||||
list.innerHTML = findings.length ? findings.map(f => `
|
||||
<div class="finding-item">
|
||||
<div class="finding-agent">${f.agent_name || "unknown"}</div>
|
||||
<div class="finding-query">${f.question || "N/A"}</div>
|
||||
<div class="finding-answer">${(f.answer || f.summary || "").substring(0, 400)}</div>
|
||||
</div>`).join('') : '<div style="color:var(--text-muted);padding:12px">No findings yet</div>';
|
||||
} catch (e) {
|
||||
list.innerHTML = `<div style="color:var(--accent-red);padding:12px">${e.message}</div>`;
|
||||
}
|
||||
},
|
||||
|
||||
async checkModels() {
|
||||
const list = $("#model-list");
|
||||
list.innerHTML = '<div class="spinner"></div>';
|
||||
|
||||
try {
|
||||
const res = await API.get("/api/ollama/models");
|
||||
const models = res.models || [];
|
||||
list.innerHTML = models.length ? models.map(m => `
|
||||
<div style="font-size:11px;padding:3px 8px;color:var(--accent-green)">
|
||||
${m.name || m.model}
|
||||
</div>`).join('') : '<div style="font-size:11px;padding:3px 8px;color:var(--text-muted)">None found</div>';
|
||||
} catch (e) {
|
||||
list.innerHTML = `<div style="font-size:11px;padding:3px 8px;color:var(--accent-red)">Off: ${e.message}</div>`;
|
||||
}
|
||||
},
|
||||
|
||||
updateHealth(health) {
|
||||
const dot = $("#health-indicator .health-dot");
|
||||
const text = $("#health-indicator");
|
||||
if (dot) {
|
||||
dot.className = health.status === "ok" ? "health-dot healthy" : "health-dot";
|
||||
text.textContent = health.status === "ok" ? "All systems operational" : "Systems degraded";
|
||||
}
|
||||
},
|
||||
|
||||
renderModels(models) {
|
||||
if (!models?.models?.length) return;
|
||||
const list = $("#model-list");
|
||||
if (list) {
|
||||
list.innerHTML = models.models.map(m => `
|
||||
<div style="font-size:11px;padding:3px 8px;color:var(--accent-green)">
|
||||
${m.name || m.model}
|
||||
</div>`).join('');
|
||||
}
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,156 @@
|
||||
// App state store
|
||||
const STORE = {
|
||||
currentDocId: null,
|
||||
currentSessionId: null,
|
||||
currentPage: 0,
|
||||
totalPages: 0,
|
||||
documents: [],
|
||||
sessions: [],
|
||||
polygons: [],
|
||||
viewerDoc: null,
|
||||
chunkCache: {},
|
||||
chatMessages: [],
|
||||
};
|
||||
|
||||
// API helpers
|
||||
const API = {
|
||||
async get(url) {
|
||||
const r = await fetch(url);
|
||||
if (!r.ok) throw new Error(`API ${r.status}: ${r.statusText}`);
|
||||
return r.json();
|
||||
},
|
||||
async post(url, body) {
|
||||
const r = await fetch(url, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
if (!r.ok) throw new Error(`API ${r.status}: ${r.statusText}`);
|
||||
return r.json();
|
||||
},
|
||||
};
|
||||
|
||||
// DOM helpers
|
||||
const $ = (sel) => document.querySelector(sel);
|
||||
const $$ = (sel) => document.querySelectorAll(sel);
|
||||
|
||||
// Render markdown-style content
|
||||
function renderContent(text) {
|
||||
if (!text) return "<p class='typing'>No content</p>";
|
||||
|
||||
let html = text
|
||||
.replace(/&/g, "&")
|
||||
.replace(/</g, "<")
|
||||
.replace(/>/g, ">");
|
||||
|
||||
// Headers
|
||||
html = html.replace(/^### (.+)$/gm, '<h4>$1</h4>');
|
||||
html = html.replace(/^## (.+)$/gm, '<h3>$1</h3>');
|
||||
html = html.replace(/^# (.+)$/gm, '<h3 class="research-answer" style="color:var(--accent-blue)">$1</h3>');
|
||||
|
||||
// Bold
|
||||
html = html.replace(/\*\*(.+?)\*\*/g, '<strong>$1</strong>');
|
||||
|
||||
// Italic
|
||||
html = html.replace(/\*(.+?)\*/g, '<em>$1</em>');
|
||||
|
||||
// Code blocks
|
||||
html = html.replace(/```([\s\S]*?)```/g, '<div class="code-block" style="background:var(--bg-tertiary);padding:10px;border-radius:6px;font-family:monospace;font-size:12px;margin:8px 0;overflow-x:auto">$1</div>');
|
||||
|
||||
// Inline code
|
||||
html = html.replace(/`(.+?)`/g, '<code style="background:var(--bg-tertiary);padding:2px 5px;border-radius:3px;font-size:12px">$1</code>');
|
||||
|
||||
// Unordered lists
|
||||
html = html.replace(/^- (.+)$/gm, '<li>$1</li>');
|
||||
html = html.replace(/(<li>.*<\/li>\n?)+/g, '<ul>$&</ul>');
|
||||
|
||||
// Ordered lists
|
||||
html = html.replace(/^\d+\. (.+)$/gm, '<li><strong>$1</strong></li>');
|
||||
|
||||
// Horizontal rules
|
||||
html = html.replace(/^---$/gm, '<hr style="border:none;border-top:1px solid var(--border-color);margin:12px 0">');
|
||||
|
||||
// Blockquotes
|
||||
html = html.replace(/^> (.+)$/gm, '<div class="block" style="border-left:3px solid var(--accent-purple);padding:8px 12px;margin:6px 0;font-style:italic;color:var(--text-secondary)">$1</div>');
|
||||
|
||||
// Tables
|
||||
html = html.replace(/^\|(.+)\|$/gm, (match, content) => {
|
||||
const cells = content.split('|').map(c => c.trim());
|
||||
if (cells.every(c => /^[-:]+$/.test(c))) {
|
||||
return '<!--table-sep-->';
|
||||
}
|
||||
if (cells.every(c => c.length < 30 && !c.includes(' '))) {
|
||||
return '<tr>' + cells.map(c => `<td>${c}</td>`).join('') + '</tr>';
|
||||
}
|
||||
return `<tr><td colspan="${cells.length}">${cells.join('</td><td>')}</td></tr>`;
|
||||
});
|
||||
|
||||
// Wrap rows in table
|
||||
let inTable = false;
|
||||
const lines = html.split('\n');
|
||||
let result = [];
|
||||
let tableBuffer = [];
|
||||
|
||||
for (let line of lines) {
|
||||
if (line.includes('<!--table-sep-->')) {
|
||||
if (tableBuffer.length > 0) {
|
||||
result.push('<table>' + tableBuffer.join('') + '</table>');
|
||||
tableBuffer = [];
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (line.startsWith('<tr>')) {
|
||||
tableBuffer.push(line);
|
||||
} else {
|
||||
if (tableBuffer.length > 0) {
|
||||
result.push('<table>' + tableBuffer.join('') + '</table>');
|
||||
tableBuffer = [];
|
||||
}
|
||||
result.push(line);
|
||||
}
|
||||
}
|
||||
if (tableBuffer.length > 0) {
|
||||
result.push('<table>' + tableBuffer.join('') + '</table>');
|
||||
}
|
||||
html = result.join('\n');
|
||||
|
||||
// Paragraphs
|
||||
html = html.replace(/^(?!<[huldtbr]|<!--)/gm, '<p>$&</p>');
|
||||
|
||||
return html;
|
||||
}
|
||||
|
||||
function showSpinner(el) {
|
||||
el.innerHTML = '<div class="spinner"></div> <span class="typing">Processing...</span>';
|
||||
}
|
||||
|
||||
function showToast(msg, type = "info") {
|
||||
const toast = document.createElement("div");
|
||||
toast.textContent = msg;
|
||||
toast.style.cssText = `
|
||||
position: fixed; bottom: 20px; right: 20px; padding: 10px 16px;
|
||||
background: ${type === "error" ? "var(--accent-red)" : "var(--accent-blue)"};
|
||||
color: #fff; border-radius: 6px; font-size: 13px; z-index: 999;
|
||||
box-shadow: 0 4px 12px rgba(0,0,0,0.3); animation: fadein 0.3s;
|
||||
`;
|
||||
document.body.appendChild(toast);
|
||||
setTimeout(() => toast.remove(), 3000);
|
||||
}
|
||||
|
||||
// Load initial state
|
||||
async function loadInitialState() {
|
||||
try {
|
||||
const docs = await API.get("/api/documents");
|
||||
STORE.documents = docs.documents || [];
|
||||
renderDocList();
|
||||
|
||||
const health = await API.get("/health");
|
||||
updateHealth(health);
|
||||
|
||||
const models = await API.get("/api/ollama/models");
|
||||
renderModels(models);
|
||||
} catch (e) {
|
||||
console.warn("Initial load failed:", e);
|
||||
updateHealth({ status: "err" });
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,377 @@
|
||||
"""FastAPI web routes for agentic research app."""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import json
|
||||
import uuid
|
||||
import asyncio
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from fastapi import FastAPI, File, UploadFile, HTTPException, Query
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import HTMLResponse, FileResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.config import get_settings
|
||||
from app.db.database import Database
|
||||
from app.agents.engine import Researcher, ResearchPipeline
|
||||
from app.agents.skills import SKILLS
|
||||
from app.agents.tools import TOOLS
|
||||
from app.core.processor import MarkerProcessor
|
||||
from app.core.embedding_engine import get_embedding_sync
|
||||
|
||||
settings = get_settings()
|
||||
app = FastAPI(title="Agentic Research", version="0.1.0")
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
db_conn: Database | None = None
|
||||
researcher = Researcher()
|
||||
processor = MarkerProcessor()
|
||||
|
||||
|
||||
# ── Models ──────────────────────────────────────────────
|
||||
|
||||
class ResearchRequest(BaseModel):
|
||||
query: str
|
||||
doc_id: str | None = None
|
||||
skills: list[str] | None = None
|
||||
|
||||
class SessionList(BaseModel):
|
||||
sessions: list
|
||||
|
||||
|
||||
# ── Lifecycle ───────────────────────────────────────────
|
||||
|
||||
@app.on_event("startup")
|
||||
async def on_startup():
|
||||
global db_conn
|
||||
pool = await Database.create_pool()
|
||||
db_conn = Database(pool)
|
||||
from app.db import database as _db_mod
|
||||
_db_mod.db = db_conn
|
||||
try:
|
||||
await db_conn.init_schema()
|
||||
except Exception as e:
|
||||
print(f"Schema init (non-fatal): {e}")
|
||||
|
||||
# Create directories
|
||||
for d in [settings.workspace_dir, settings.documents_dir]:
|
||||
os.makedirs(d, exist_ok=True)
|
||||
|
||||
|
||||
# ── Frontend ───────────────────────────────────────────
|
||||
|
||||
@app.get("/", response_class=HTMLResponse)
|
||||
async def index():
|
||||
return FileResponse("frontend/index.html")
|
||||
|
||||
app.mount("/static", StaticFiles(directory="frontend"), name="static")
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
return {
|
||||
"status": "ok",
|
||||
"ollama": settings.ollama_url,
|
||||
"marker_api": settings.marker_api_url,
|
||||
"db": "connected" if db_conn else "disconnected",
|
||||
}
|
||||
|
||||
|
||||
# ── Documents ──────────────────────────────────────────
|
||||
|
||||
@app.post("/api/documents/upload")
|
||||
async def upload_document(
|
||||
file: UploadFile = File(None),
|
||||
pdf_url: str | None = None,
|
||||
):
|
||||
if not file and not pdf_url:
|
||||
raise HTTPException(400, "Provide file or pdf_url")
|
||||
|
||||
if db_conn is None:
|
||||
raise HTTPException(500, "Database not initialized")
|
||||
|
||||
doc_id = str(uuid.uuid4())
|
||||
filename = file.filename if file else pdf_url.split("/")[-1] if pdf_url else "unknown"
|
||||
|
||||
# Read content
|
||||
text_content = ""
|
||||
file_data = b""
|
||||
|
||||
if file:
|
||||
file_data = await file.read()
|
||||
ext = Path(filename).suffix.lower()
|
||||
if ext == ".pdf":
|
||||
pass # PDF processing below
|
||||
else:
|
||||
# Store plain text directly
|
||||
text_content = file_data.decode("utf-8", errors="replace")
|
||||
file_path = os.path.join(settings.documents_dir, filename)
|
||||
os.makedirs(settings.documents_dir, exist_ok=True)
|
||||
with open(file_path, "wb") as f:
|
||||
f.write(file_data)
|
||||
|
||||
# Chunk the text
|
||||
import re
|
||||
paragraphs = re.split(r'\n\s*\n', text_content)
|
||||
chunks = []
|
||||
for i, para in enumerate(paragraphs):
|
||||
if len(para.strip()) > 20:
|
||||
chunks.append({
|
||||
"content": para.strip(), "page_num": 0,
|
||||
"block_index": i, "polygon": None, "chunk_type": "text",
|
||||
})
|
||||
if chunks:
|
||||
await db_conn.batch_chunk(doc_id, chunks)
|
||||
|
||||
await db_conn.upsert_document(
|
||||
filename, None, file.content_type or "text/plain",
|
||||
file_path, "indexed", 1, text_content[:50000], {"uploaded": True}
|
||||
)
|
||||
return {"doc_id": doc_id, "filename": filename, "chunks": len(chunks), "strategy": "direct_text"}
|
||||
|
||||
# PDF processing
|
||||
try:
|
||||
file_path = os.path.join(settings.documents_dir, filename)
|
||||
os.makedirs(settings.documents_dir, exist_ok=True)
|
||||
with open(file_path, "wb") as f:
|
||||
f.write(file_data)
|
||||
|
||||
# Create document record first (so FK constraint is satisfied)
|
||||
doc_uuid = str(uuid.uuid4())
|
||||
await db_conn.upsert_document(
|
||||
filename, doc_uuid, "application/pdf",
|
||||
file_path, "processing", 0,
|
||||
"", {"status": "processing"}
|
||||
)
|
||||
|
||||
# Process through marker, using the real PK as doc_id
|
||||
result = await processor.process_document_file(file_data, filename, doc_uuid)
|
||||
|
||||
# Update document with actual page count and result
|
||||
await db_conn.upsert_document(
|
||||
filename, doc_uuid, "application/pdf",
|
||||
file_path, "indexed", result.get("page_count", 0),
|
||||
json.dumps({"marker_result": result})[:10000], {"uploaded": True}
|
||||
)
|
||||
return {**result, "doc_id": doc_uuid, "filename": filename, "strategy": "marker_ocr"}
|
||||
except Exception as e:
|
||||
raise HTTPException(500, f"OCR failed: {str(e)}")
|
||||
|
||||
|
||||
@app.get("/api/documents")
|
||||
async def list_documents():
|
||||
if db_conn is None:
|
||||
return {"documents": []}
|
||||
docs = await db_conn.list_documents()
|
||||
return {"documents": docs}
|
||||
|
||||
|
||||
@app.get("/api/documents/{doc_id}/chunks")
|
||||
async def get_document_chunks(doc_id: str, page: int | None = None):
|
||||
if db_conn is None:
|
||||
return {"chunks": []}
|
||||
chunks = await db_conn.get_doc_chunks(doc_id)
|
||||
if page is not None:
|
||||
chunks = [c for c in chunks if c.get("page_num") == page]
|
||||
return {"chunks": chunks}
|
||||
|
||||
|
||||
@app.get("/api/documents/{doc_id}")
|
||||
async def get_document(doc_id: str):
|
||||
if db_conn is None:
|
||||
return {"document": None}
|
||||
doc = await db_conn.get_document(doc_id)
|
||||
return {"document": doc}
|
||||
|
||||
|
||||
# ── Polygons / View ───────────────────────────────────
|
||||
|
||||
@app.get("/api/documents/{doc_id}/polygon-view")
|
||||
async def get_polygon_view(doc_id: str, page: int = 0):
|
||||
view = await TOOLS["get_polygon_view"](doc_id, page)
|
||||
return view
|
||||
|
||||
|
||||
@app.get("/api/documents/{doc_id}/page-text")
|
||||
async def get_page_text(doc_id: str, page: int = 0):
|
||||
text = await TOOLS["get_page_text"](doc_id, page)
|
||||
return {"page": page, "text": text}
|
||||
|
||||
|
||||
# ── Research ───────────────────────────────────────────
|
||||
|
||||
@app.post("/api/research/session")
|
||||
async def create_session(body: dict):
|
||||
if db_conn is None:
|
||||
raise HTTPException(500, "DB not ready")
|
||||
query = body.get("query", "research query") if isinstance(body, dict) else str(body)
|
||||
session_id = await db_conn.create_session(query)
|
||||
return {"session_id": session_id, "query": query}
|
||||
|
||||
|
||||
@app.post("/api/research/run")
|
||||
async def run_research(req: ResearchRequest):
|
||||
if db_conn is None:
|
||||
raise HTTPException(500, "DB not ready")
|
||||
|
||||
researcher = Researcher()
|
||||
results = await researcher.run_research_session(
|
||||
req.query, req.doc_id or "temp", skill_names=req.skills
|
||||
)
|
||||
|
||||
# Save findings
|
||||
overall_answer = ""
|
||||
for skill, response in results.items():
|
||||
await db_conn.store_finding(
|
||||
req.doc_id or "temp", req.query, response, "Research complete",
|
||||
"researcher", 0.8
|
||||
)
|
||||
if response:
|
||||
overall_answer += f"### {skill}:\n{response}\n\n"
|
||||
|
||||
return {"results": results, "doc_id": req.doc_id, "session_id": req.doc_id or "temp"}
|
||||
|
||||
|
||||
@app.post("/api/research/semantic")
|
||||
async def semantic_search(query: str, doc_id: str | None = None, limit: int = 20):
|
||||
vec = get_embedding_sync(query, settings.ollama_url)
|
||||
results = await db_conn.vector_search(vec, doc_id=doc_id, limit=limit)
|
||||
return {"results": results, "query": query}
|
||||
|
||||
|
||||
@app.post("/api/research/text-search")
|
||||
async def text_search(query: str, doc_id: str | None = None, limit: int = 20):
|
||||
results = await db_conn.search_chunks_text(query, doc_id=doc_id, limit=limit)
|
||||
return {"results": results, "query": query}
|
||||
|
||||
|
||||
@app.get("/api/research/findings")
|
||||
async def get_findings(session_id: str):
|
||||
if db_conn:
|
||||
findings = await db_conn.get_findings(session_id)
|
||||
return {"findings": findings}
|
||||
return {"findings": []}
|
||||
|
||||
|
||||
@app.get("/api/research/memories")
|
||||
async def get_memories(session_id: str):
|
||||
if db_conn:
|
||||
mems = await db_conn.get_memories(session_id)
|
||||
return {"memories": mems}
|
||||
return {"memories": []}
|
||||
|
||||
|
||||
# ── Memories ───────────────────────────────────────────
|
||||
|
||||
@app.post("/api/memories/save")
|
||||
async def save_memory(
|
||||
session_id: str,
|
||||
content: str,
|
||||
memory_type: str = "fact",
|
||||
importance: int = 3,
|
||||
source_doc_id: str | None = None,
|
||||
):
|
||||
if not db_conn:
|
||||
raise HTTPException(500, "DB not ready")
|
||||
mem_id = await db_conn.store_memory(session_id, content, memory_type, importance, source_doc_id)
|
||||
return {"memory_id": mem_id}
|
||||
|
||||
|
||||
@app.post("/api/memories/search")
|
||||
async def search_memories(query: str, limit: int = 10):
|
||||
results = await db_conn.memory_similarity_search(query, limit)
|
||||
return {"memories": results}
|
||||
|
||||
|
||||
# ── Ollama / Models ──────────────────────────────────
|
||||
|
||||
@app.get("/api/ollama/models")
|
||||
async def check_ollama():
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=10) as client:
|
||||
resp = await client.get(f"{settings.ollama_url}/api/tags")
|
||||
return {"models": resp.json().get("models", [])}
|
||||
except Exception as e:
|
||||
return {"error": str(e), "ollama_url": settings.ollama_url}
|
||||
|
||||
|
||||
@app.post("/api/ollama/chat")
|
||||
async def ollama_chat(body: dict):
|
||||
model = body.get("model", settings.gpt_oss_model)
|
||||
messages = body.get("messages", [])
|
||||
|
||||
async with httpx.AsyncClient(timeout=600) as client:
|
||||
resp = await client.post(
|
||||
f"{settings.ollama_url}/api/chat",
|
||||
json={"model": model, "messages": messages, "stream": False},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
|
||||
# ── Agents ────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/agents/skills")
|
||||
async def get_skills():
|
||||
return {"skills": [{"name": n, "description": s.description} for n, s in SKILLS.items()]}
|
||||
|
||||
|
||||
# ── Pipeline ────────────────────────────────────
|
||||
|
||||
class PipelineRequest(BaseModel):
|
||||
query: str
|
||||
doc_ids: list[str] | None = None
|
||||
output_mode: str | None = None
|
||||
|
||||
|
||||
@app.post("/api/research/pipeline")
|
||||
async def run_pipeline(req: PipelineRequest):
|
||||
"""Run the full document-triage → evidence-extraction → research-synthesis pipeline."""
|
||||
if not db_conn:
|
||||
raise HTTPException(500, "DB not ready")
|
||||
|
||||
pipeline = ResearchPipeline()
|
||||
results = await pipeline.run(
|
||||
query=req.query,
|
||||
session_id=req.doc_ids[0] if req.doc_ids else "temp",
|
||||
db=db_conn,
|
||||
doc_ids=req.doc_ids,
|
||||
output_mode=req.output_mode,
|
||||
)
|
||||
return results
|
||||
|
||||
|
||||
@app.get("/api/research/pipeline/{session_id}/stages")
|
||||
async def get_pipeline_stages(session_id: str):
|
||||
if db_conn:
|
||||
stages = await db_conn.get_pipeline_stages(session_id)
|
||||
return {"stages": stages}
|
||||
return {"stages": []}
|
||||
|
||||
|
||||
@app.post("/api/research/pipeline/{session_id}/render")
|
||||
async def render_pipeline_results(session_id: str):
|
||||
"""Return rendered pipeline sections for display."""
|
||||
if not db_conn:
|
||||
raise HTTPException(500, "DB not ready")
|
||||
stages = await db_conn.get_pipeline_stages(session_id)
|
||||
pipeline = ResearchPipeline()
|
||||
sections = await pipeline.render_pipeline_results({s["stage"]: s["output"] for s in stages})
|
||||
return {"sections": sections, "session_id": session_id}
|
||||
|
||||
|
||||
# ── Docs ──────────────────────────────────────────────
|
||||
|
||||
@app.get("/docs", include_in_schema=False)
|
||||
async def docs_redirect():
|
||||
return FileResponse("/api/docs")
|
||||
Submodule
+1
Submodule marker-api added at d3739db19b
@@ -0,0 +1,118 @@
|
||||
-- Initialize pgvector extension and research tables
|
||||
CREATE EXTENSION IF NOT EXISTS vector;
|
||||
CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
||||
|
||||
-- Document storage
|
||||
CREATE TABLE IF NOT EXISTS documents (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
filename TEXT NOT NULL,
|
||||
doc_id TEXT,
|
||||
mime_type TEXT,
|
||||
file_path TEXT,
|
||||
status TEXT DEFAULT 'pending',
|
||||
page_count INTEGER DEFAULT 0,
|
||||
full_text TEXT,
|
||||
metadata JSONB DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Vector embeddings for similarity search
|
||||
CREATE TABLE IF NOT EXISTS chunks (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
doc_id UUID REFERENCES documents(id) ON DELETE CASCADE,
|
||||
content TEXT NOT NULL,
|
||||
vector VECTOR(4096),
|
||||
page_num INTEGER,
|
||||
block_index INTEGER,
|
||||
polygon JSONB,
|
||||
chunk_type TEXT DEFAULT 'text',
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Research memories/knowledge store
|
||||
CREATE TABLE IF NOT EXISTS memories (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
session_id UUID REFERENCES documents(id),
|
||||
content TEXT NOT NULL,
|
||||
vector VECTOR(4096),
|
||||
memory_type TEXT DEFAULT 'fact',
|
||||
importance INTEGER DEFAULT 3,
|
||||
source_doc_id UUID REFERENCES documents(id),
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Research findings
|
||||
CREATE TABLE IF NOT EXISTS findings (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
session_id UUID REFERENCES documents(id),
|
||||
question TEXT,
|
||||
answer TEXT,
|
||||
summary TEXT,
|
||||
vector VECTOR(4096),
|
||||
relevant_chunks JSONB,
|
||||
agent_name TEXT,
|
||||
confidence FLOAT,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Research sessions
|
||||
CREATE TABLE IF NOT EXISTS research_sessions (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
query TEXT NOT NULL,
|
||||
status TEXT DEFAULT 'running',
|
||||
documents JSONB DEFAULT '[]',
|
||||
findings JSONB DEFAULT '[]',
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
completed_at TIMESTAMPTZ
|
||||
);
|
||||
|
||||
-- Indexes
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_vector ON chunks USING ivfflat (vector vector_ip_opclass) WITH (lists = 100);
|
||||
CREATE INDEX IF NOT EXISTS idx_memories_vector ON memories USING ivfflat (vector vector_ip_opclass) WITH (lists = 100);
|
||||
CREATE INDEX IF NOT EXISTS idx_findings_vector ON findings USING ivfflat (vector vector_ip_opclass) WITH (lists = 100);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_trgm ON chunks USING gin (content gin_trgm_ops);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_doc ON chunks(doc_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_doc ON chunks(doc_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_memories_session ON memories(session_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_findings_session ON findings(session_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_documents_status ON documents(status);
|
||||
|
||||
-- Pipeline intermediate stage storage
|
||||
CREATE TABLE IF NOT EXISTS pipeline_stages (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
session_id UUID REFERENCES research_sessions(id) ON DELETE CASCADE,
|
||||
stage TEXT NOT NULL CHECK (stage IN ('triage','evidence','synthesis')),
|
||||
output TEXT,
|
||||
state JSONB DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_pipeline_session ON pipeline_stages(session_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_pipeline_stage ON pipeline_stages(stage, session_id);
|
||||
|
||||
-- Create function for cosine similarity
|
||||
CREATE OR REPLACE FUNCTION cosine_similarity(v1 VECTOR, v2 VECTOR)
|
||||
RETURNS FLOAT AS $$
|
||||
SELECT 1 - (v1 <-> v2) / 2;
|
||||
$$ LANGUAGE SQL;
|
||||
|
||||
-- Create function for vector similarity search
|
||||
CREATE OR REPLACE FUNCTION match_vectors(query_embedding VECTOR(4096), match_count INTEGER DEFAULT 10, min_score FLOAT DEFAULT 0)
|
||||
RETURNS TABLE(id UUID, chunk_id UUID, content TEXT, doc_id UUID, page_num INT, similarity FLOAT)
|
||||
LANGUAGE plpgsql
|
||||
AS $$
|
||||
BEGIN
|
||||
RETURN QUERY
|
||||
SELECT
|
||||
m.id::TEXT,
|
||||
m.id::TEXT,
|
||||
m.content::TEXT,
|
||||
m.doc_id::TEXT,
|
||||
m.page_num::INT,
|
||||
cosine_similarity(m.vector, query_embedding)::FLOAT
|
||||
FROM chunks m
|
||||
WHERE cosine_similarity(m.vector, query_embedding) >= min_score
|
||||
ORDER BY m.vector <-> query_embedding
|
||||
LIMIT match_count;
|
||||
END;
|
||||
$$;
|
||||
@@ -0,0 +1,68 @@
|
||||
services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg16
|
||||
container_name: agentic-research-pg
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "5432:5432"
|
||||
environment:
|
||||
POSTGRES_DB: research
|
||||
POSTGRES_USER: research
|
||||
POSTGRES_PASSWORD: research123
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
- ./migrations/init.sql:/docker-entrypoint-initdb.d/01-init.sql
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U research -d research"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
marker-api:
|
||||
build:
|
||||
context: ./marker-api
|
||||
dockerfile: Dockerfile
|
||||
container_name: agentic-research-marker
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8001:8001"
|
||||
environment:
|
||||
OLLAMA_URL: http://10.0.1.127:11434
|
||||
OCR_URL: http://10.0.1.127:11434
|
||||
DEEPSEEK_OCR_URL: http://10.0.1.127:11434
|
||||
DEEPSEEK_OCR_MODEL: deepseek-ocr
|
||||
LLM_BACKEND: ollama
|
||||
LLM_URL: http://10.0.1.127:11434
|
||||
MARKER_WORKSPACE: /workspace/marker_output
|
||||
volumes:
|
||||
- marker_ws:/workspace
|
||||
|
||||
app:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
container_name: agentic-research-app
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8000:8000"
|
||||
environment:
|
||||
OLLAMA_URL: http://10.0.1.127:11434
|
||||
GPT_OSS_MODEL: gpt-oss:20b
|
||||
MARKER_API_URL: http://marker-api:8001
|
||||
DB_HOST: postgres
|
||||
DB_PORT: 5432
|
||||
DB_NAME: research
|
||||
DB_USER: research
|
||||
DB_PASSWORD: research123
|
||||
DOC_STORE_DIR: /workspace/documents
|
||||
VECTOR_DIM: 4096
|
||||
volumes:
|
||||
- app_ws:/workspace
|
||||
depends_on:
|
||||
- postgres
|
||||
- marker-api
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
marker_ws:
|
||||
app_ws:
|
||||
@@ -0,0 +1,26 @@
|
||||
[project]
|
||||
name = "agentic-research"
|
||||
version = "0.1.0"
|
||||
description = "AI-powered agentic document research tool with polygon-aware viewer"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"fastapi>=0.115.0",
|
||||
"uvicorn[standard]>=0.32.0",
|
||||
"python-multipart>=0.0.9",
|
||||
"pydantic>=2.9.0",
|
||||
"pydantic-settings>=2.5.0",
|
||||
"httpx>=0.27.0",
|
||||
"asyncpg>=0.30.0",
|
||||
"psycopg2-binary>=2.9.9",
|
||||
"langchain>=0.3.0",
|
||||
"langchain-community>=0.3.0",
|
||||
"langchain-openai>=0.2.0",
|
||||
"langgraph>=0.2.0",
|
||||
"sentence-transformers>=3.0.0",
|
||||
"numpy>=1.26.0",
|
||||
"aiofiles>=24.1.0",
|
||||
"rich>=13.9.0",
|
||||
"python-dotenv>=1.0.1",
|
||||
"markdown>=3.7.0",
|
||||
"pdfplumber>=0.11.0",
|
||||
]
|
||||
+4488
File diff suppressed because one or more lines are too long
@@ -0,0 +1,71 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$SCRIPT_DIR"
|
||||
|
||||
echo "=== Agentic Research App ==="
|
||||
echo ""
|
||||
|
||||
# ── Check pre-requisites ─────────────────────────────
|
||||
check_cmd() {
|
||||
if ! command -v "$1" &>/dev/null; then
|
||||
echo "ERROR: $1 not found. Please install first." >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
check_cmd python3
|
||||
check_cmd podman
|
||||
check_cmd podman-compose
|
||||
|
||||
# ── Python venv ─────────────────────────
|
||||
|
||||
if [ ! -d "venv" ]; then
|
||||
echo "[1/6] Creating Python virtual environment..."
|
||||
python3 -m venv venv
|
||||
fi
|
||||
source venv/bin/activate
|
||||
|
||||
echo "[2/6] Installing Python dependencies..."
|
||||
pip install -q -e . 2>&1 | tail -1
|
||||
|
||||
# ── Environment ─────────────────────────
|
||||
|
||||
if [ ! -f ".env" ]; then
|
||||
echo "[3/6] Creating .env from .env.example..."
|
||||
cp .env.example .env
|
||||
fi
|
||||
|
||||
# ── Start PostgreSQL + pgvector via Podman ──────────────
|
||||
|
||||
echo "[4/6] Starting PostgreSQL + pgvector..."
|
||||
podman-compose -f podman-compose.yml up -d postgres 2>&1 | tail -1
|
||||
|
||||
# Wait for PostgreSQL to be ready
|
||||
echo " Waiting for PostgreSQL..."
|
||||
for i in $(seq 1 30); do
|
||||
if podman exec agentic-research-pg pg_isready -U research -d research &>/dev/null; then
|
||||
echo " PostgreSQL ready!"
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
# ── Start Marker API ───────────────────────
|
||||
|
||||
echo "[5/6] Starting Marker API with deepseek-ocr..."
|
||||
podman-compose -f podman-compose.yml up -d marker-api 2>&1 | tail -1
|
||||
|
||||
# ── Start App ─────────────────────────
|
||||
|
||||
echo "[6/6] Starting Agentic Research App..."
|
||||
podman-compose -f podman-compose.yml up -d app 2>&1 | tail -1
|
||||
|
||||
echo ""
|
||||
echo "=== Agentic Research App is running ==="
|
||||
echo " UI: http://localhost:8000"
|
||||
echo " Docs: http://localhost:8000/docs"
|
||||
echo " Marker: http://localhost:8001/docs"
|
||||
echo ""
|
||||
echo "To stop: podman-compose -f podman-compose.yml down"
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Reference in New Issue
Block a user