Probably functional. Testing
This commit is contained in:
@@ -1,3 +1,12 @@
|
|||||||
|
---
|
||||||
|
title: Add-Only Mode Examples
|
||||||
|
created: '2025-10-23T18:33:48+02:00'
|
||||||
|
changed: '2025-10-23T18:33:48+02:00'
|
||||||
|
authors:
|
||||||
|
- Test User
|
||||||
|
version: 8c4f612
|
||||||
|
tags: []
|
||||||
|
---
|
||||||
# Add-Only Mode Examples
|
# Add-Only Mode Examples
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|||||||
@@ -0,0 +1,287 @@
|
|||||||
|
# Configuration Management
|
||||||
|
|
||||||
|
## Configuration Directory
|
||||||
|
|
||||||
|
Madomeda stores its configuration in a platform-specific location:
|
||||||
|
|
||||||
|
- **Windows**: `%USERPROFILE%\.config\madomeda`
|
||||||
|
- **macOS**: `~/.config/madomeda`
|
||||||
|
- **Linux**: `~/.config/madomeda`
|
||||||
|
|
||||||
|
## First Run
|
||||||
|
|
||||||
|
On first run, Madomeda automatically creates:
|
||||||
|
|
||||||
|
```
|
||||||
|
~/.config/madomeda/
|
||||||
|
├── .env # API configuration
|
||||||
|
├── templates/
|
||||||
|
│ └── default.json # Default frontmatter template
|
||||||
|
├── rules/
|
||||||
|
│ ├── lowercase_keys.json
|
||||||
|
│ ├── summary_to_description.json
|
||||||
|
│ ├── tag_to_tags.json
|
||||||
|
│ └── tags_normalization.json
|
||||||
|
└── prompts/
|
||||||
|
└── default.txt # Default LLM prompt
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration Display
|
||||||
|
|
||||||
|
When running in an **interactive shell**, Madomeda displays configuration info:
|
||||||
|
|
||||||
|
```
|
||||||
|
Configuration directory: C:\Users\YourName\.config\madomeda
|
||||||
|
.env file: C:\Users\YourName\.config\madomeda\.env
|
||||||
|
Templates: C:\Users\YourName\.config\madomeda\templates
|
||||||
|
Rules: C:\Users\YourName\.config\madomeda\rules
|
||||||
|
Prompts: C:\Users\YourName\.config\madomeda\prompts
|
||||||
|
API Key: Not configured (LLM features disabled)
|
||||||
|
```
|
||||||
|
|
||||||
|
In **non-interactive mode** (pipes, scripts), this info is suppressed.
|
||||||
|
|
||||||
|
## API Key Configuration
|
||||||
|
|
||||||
|
The `.env` file supports three modes for `OPENAI_API_KEY`:
|
||||||
|
|
||||||
|
### 1. No API Key (Default)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
OPENAI_API_KEY=""
|
||||||
|
```
|
||||||
|
|
||||||
|
**Output:**
|
||||||
|
```
|
||||||
|
API Key: Not configured (LLM features disabled)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Behavior:** LLM features are disabled, only heuristics and original values used.
|
||||||
|
|
||||||
|
### 2. Environment Variable Reference
|
||||||
|
|
||||||
|
```bash
|
||||||
|
OPENAI_API_KEY=$MY_API_KEY
|
||||||
|
```
|
||||||
|
|
||||||
|
**Output:**
|
||||||
|
```
|
||||||
|
API Key: From environment variable $MY_API_KEY
|
||||||
|
```
|
||||||
|
|
||||||
|
**Behavior:**
|
||||||
|
- Reads key from environment variable `MY_API_KEY`
|
||||||
|
- **Error if not found:**
|
||||||
|
```
|
||||||
|
ValueError: Environment variable 'MY_API_KEY' not found.
|
||||||
|
Referenced in .env as OPENAI_API_KEY=$MY_API_KEY
|
||||||
|
```
|
||||||
|
|
||||||
|
**Setup:**
|
||||||
|
```bash
|
||||||
|
# Linux/macOS
|
||||||
|
export MY_API_KEY="sk-your-actual-key"
|
||||||
|
|
||||||
|
# Windows (PowerShell)
|
||||||
|
$env:MY_API_KEY = "sk-your-actual-key"
|
||||||
|
|
||||||
|
# Windows (CMD)
|
||||||
|
set MY_API_KEY=sk-your-actual-key
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Plain Text Key
|
||||||
|
|
||||||
|
```bash
|
||||||
|
OPENAI_API_KEY=sk-1234567890abcdef
|
||||||
|
```
|
||||||
|
|
||||||
|
**Output:**
|
||||||
|
```
|
||||||
|
API Key: Configured (plain text)
|
||||||
|
WARNING: Storing API keys in plain text is a security risk!
|
||||||
|
Consider using environment variable: OPENAI_API_KEY=$YOUR_ENV_VAR
|
||||||
|
```
|
||||||
|
|
||||||
|
**Behavior:** Works but shows security warning every run.
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
### ✅ Recommended: Environment Variable
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# In .env file
|
||||||
|
OPENAI_API_KEY=$OPENAI_API_KEY
|
||||||
|
|
||||||
|
# In your shell profile (~/.bashrc, ~/.zshrc, etc.)
|
||||||
|
export OPENAI_API_KEY="sk-your-actual-key"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Benefits:**
|
||||||
|
- Key not stored in config file
|
||||||
|
- Can be different per user/machine
|
||||||
|
- Easy to rotate without editing files
|
||||||
|
- Can use system keyring tools
|
||||||
|
|
||||||
|
### ⚠️ Acceptable: Empty String
|
||||||
|
|
||||||
|
```bash
|
||||||
|
OPENAI_API_KEY=""
|
||||||
|
```
|
||||||
|
|
||||||
|
**Use when:**
|
||||||
|
- Don't have LLM access
|
||||||
|
- Only using heuristics
|
||||||
|
- Testing/development
|
||||||
|
|
||||||
|
### ❌ Not Recommended: Plain Text
|
||||||
|
|
||||||
|
```bash
|
||||||
|
OPENAI_API_KEY=sk-1234567890abcdef
|
||||||
|
```
|
||||||
|
|
||||||
|
**Risks:**
|
||||||
|
- Key visible to anyone with file access
|
||||||
|
- Accidentally committed to version control
|
||||||
|
- Hard to rotate across systems
|
||||||
|
|
||||||
|
## Customizing Configuration
|
||||||
|
|
||||||
|
### Custom Templates
|
||||||
|
|
||||||
|
Create in `~/.config/madomeda/templates/`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create minimal template
|
||||||
|
cat > ~/.config/madomeda/templates/minimal.json << 'EOF'
|
||||||
|
{
|
||||||
|
"title": "heur|orig",
|
||||||
|
"date": "heur|orig",
|
||||||
|
"tags": "orig"
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Use it
|
||||||
|
madomeda --template minimal
|
||||||
|
```
|
||||||
|
|
||||||
|
### Custom Rules
|
||||||
|
|
||||||
|
Create in `~/.config/madomeda/rules/`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cat > ~/.config/madomeda/rules/my_rule.json << 'EOF'
|
||||||
|
{
|
||||||
|
"name": "my_custom_rule",
|
||||||
|
"description": "My custom rule description",
|
||||||
|
"type": "custom"
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
```
|
||||||
|
|
||||||
|
Note: Custom rules require code changes in `core/rules.py` to implement logic.
|
||||||
|
|
||||||
|
### Custom Prompts
|
||||||
|
|
||||||
|
Create in `~/.config/madomeda/prompts/`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cat > ~/.config/madomeda/prompts/technical.txt << 'EOF'
|
||||||
|
You are a technical documentation expert.
|
||||||
|
Focus on API documentation and code examples.
|
||||||
|
...
|
||||||
|
EOF
|
||||||
|
```
|
||||||
|
|
||||||
|
Reference in template:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"summary": "ai technical"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Multiple API Providers
|
||||||
|
|
||||||
|
Edit `OPENAI_API_URL` in `.env`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# OpenAI
|
||||||
|
OPENAI_API_URL=https://api.openai.com/v1
|
||||||
|
|
||||||
|
# Azure OpenAI
|
||||||
|
OPENAI_API_URL=https://your-resource.openai.azure.com/
|
||||||
|
|
||||||
|
# Local LLM (Ollama)
|
||||||
|
OPENAI_API_URL=http://localhost:11434/v1
|
||||||
|
|
||||||
|
# Other OpenAI-compatible APIs
|
||||||
|
OPENAI_API_URL=https://api.your-provider.com/v1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Config Not Created
|
||||||
|
|
||||||
|
Ensure write permissions:
|
||||||
|
```bash
|
||||||
|
mkdir -p ~/.config/madomeda
|
||||||
|
chmod 755 ~/.config/madomeda
|
||||||
|
```
|
||||||
|
|
||||||
|
### Environment Variable Not Found
|
||||||
|
|
||||||
|
Check it's set:
|
||||||
|
```bash
|
||||||
|
# Linux/macOS
|
||||||
|
echo $MY_API_KEY
|
||||||
|
|
||||||
|
# Windows (PowerShell)
|
||||||
|
echo $env:MY_API_KEY
|
||||||
|
```
|
||||||
|
|
||||||
|
If not set, add to shell profile or set before running:
|
||||||
|
```bash
|
||||||
|
MY_API_KEY="sk-key" madomeda
|
||||||
|
```
|
||||||
|
|
||||||
|
### Wrong Config Location
|
||||||
|
|
||||||
|
Madomeda uses `$USERPROFILE` (Windows) or `$HOME` (Unix).
|
||||||
|
|
||||||
|
Check:
|
||||||
|
```bash
|
||||||
|
# Linux/macOS
|
||||||
|
echo ~/.config/madomeda
|
||||||
|
|
||||||
|
# Windows (PowerShell)
|
||||||
|
echo $env:USERPROFILE\.config\madomeda
|
||||||
|
```
|
||||||
|
|
||||||
|
### API Key Warning on Every Run
|
||||||
|
|
||||||
|
This is intentional for plain text keys. Switch to environment variable to suppress.
|
||||||
|
|
||||||
|
## Migration from Old Version
|
||||||
|
|
||||||
|
If you had local `.env`, `templates/`, `rules/`, `prompts/` in the repo:
|
||||||
|
|
||||||
|
1. Copy to new location:
|
||||||
|
```bash
|
||||||
|
cp -r templates ~/.config/madomeda/
|
||||||
|
cp -r rules ~/.config/madomeda/
|
||||||
|
cp -r prompts ~/.config/madomeda/
|
||||||
|
cp .env ~/.config/madomeda/
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Update `.gitignore` (no longer need to ignore these locally)
|
||||||
|
|
||||||
|
3. Old local files are now ignored by the program
|
||||||
|
|
||||||
|
## Security Notes
|
||||||
|
|
||||||
|
- Never commit `.env` with plain text keys
|
||||||
|
- Use environment variables or secret management tools
|
||||||
|
- Config directory is user-specific (not shared)
|
||||||
|
- On shared systems, ensure `~/.config/madomeda/` has proper permissions:
|
||||||
|
```bash
|
||||||
|
chmod 700 ~/.config/madomeda
|
||||||
|
```
|
||||||
@@ -1,3 +1,12 @@
|
|||||||
|
---
|
||||||
|
title: Getting Started with Madomeda
|
||||||
|
created: '2025-10-23T18:33:48+02:00'
|
||||||
|
changed: '2025-10-23T18:33:48+02:00'
|
||||||
|
authors:
|
||||||
|
- Test User
|
||||||
|
version: 8c4f612
|
||||||
|
tags: []
|
||||||
|
---
|
||||||
# Getting Started with Madomeda
|
# Getting Started with Madomeda
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|||||||
@@ -1,3 +1,12 @@
|
|||||||
|
---
|
||||||
|
title: Madomeda - Implementation Summary
|
||||||
|
created: '2025-10-23T18:33:48+02:00'
|
||||||
|
changed: '2025-10-23T18:33:48+02:00'
|
||||||
|
authors:
|
||||||
|
- Test User
|
||||||
|
version: 8c4f612
|
||||||
|
tags: []
|
||||||
|
---
|
||||||
# Madomeda - Implementation Summary
|
# Madomeda - Implementation Summary
|
||||||
|
|
||||||
## What Was Created
|
## What Was Created
|
||||||
|
|||||||
@@ -1,3 +1,12 @@
|
|||||||
|
---
|
||||||
|
title: Madomeda - Project Complete
|
||||||
|
created: '2025-10-23T18:33:48+02:00'
|
||||||
|
changed: '2025-10-23T18:33:48+02:00'
|
||||||
|
authors:
|
||||||
|
- Test User
|
||||||
|
version: 8c4f612
|
||||||
|
tags: []
|
||||||
|
---
|
||||||
# Madomeda - Project Complete
|
# Madomeda - Project Complete
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|||||||
@@ -1,3 +1,12 @@
|
|||||||
|
---
|
||||||
|
title: Madomeda - Markdown Document Metadata Manager
|
||||||
|
created: '2025-10-23T18:33:48+02:00'
|
||||||
|
changed: '2025-10-23T18:33:48+02:00'
|
||||||
|
authors:
|
||||||
|
- Test User
|
||||||
|
version: 8c4f612
|
||||||
|
tags: []
|
||||||
|
---
|
||||||
# Madomeda - Markdown Document Metadata Manager
|
# Madomeda - Markdown Document Metadata Manager
|
||||||
|
|
||||||
A Python tool for managing and normalizing frontmatter in markdown files within git repositories.
|
A Python tool for managing and normalizing frontmatter in markdown files within git repositories.
|
||||||
@@ -64,15 +73,25 @@ SQLite database stores:
|
|||||||
pip install -r requirements.txt
|
pip install -r requirements.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The first time you run Madomeda, it will automatically create a configuration directory at `~/.config/madomeda` (all platforms) with default settings.
|
||||||
|
|
||||||
### Optional: LLM Configuration
|
### Optional: LLM Configuration
|
||||||
|
|
||||||
Edit `.env`:
|
Edit `~/.config/madomeda/.env`:
|
||||||
```
|
|
||||||
OPENAI_API_URL=https://api.openai.com/v1
|
```bash
|
||||||
OPENAI_MODEL=gpt-4
|
# Option 1: Use environment variable (recommended)
|
||||||
OPENAI_API_KEY=your-api-key-here
|
OPENAI_API_KEY=$OPENAI_API_KEY
|
||||||
|
|
||||||
|
# Option 2: No LLM (empty string)
|
||||||
|
OPENAI_API_KEY=""
|
||||||
|
|
||||||
|
# Option 3: Plain text (not recommended - security warning)
|
||||||
|
OPENAI_API_KEY=sk-your-key-here
|
||||||
```
|
```
|
||||||
|
|
||||||
|
For detailed configuration options, see [CONFIGURATION.md](CONFIGURATION.md).
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -131,9 +150,12 @@ tags:
|
|||||||
|
|
||||||
- **[GETTING_STARTED.md](GETTING_STARTED.md)** - Step-by-step guide
|
- **[GETTING_STARTED.md](GETTING_STARTED.md)** - Step-by-step guide
|
||||||
- **[USAGE.md](USAGE.md)** - Detailed usage examples
|
- **[USAGE.md](USAGE.md)** - Detailed usage examples
|
||||||
|
- **[CONFIGURATION.md](CONFIGURATION.md)** - Configuration management
|
||||||
- **[STRUCTURE.md](STRUCTURE.md)** - Project architecture
|
- **[STRUCTURE.md](STRUCTURE.md)** - Project architecture
|
||||||
- **[IMPLEMENTATION.md](IMPLEMENTATION.md)** - Technical details
|
- **[IMPLEMENTATION.md](IMPLEMENTATION.md)** - Technical details
|
||||||
- **[PROJECT_SUMMARY.md](PROJECT_SUMMARY.md)** - Complete overview
|
- **[PROJECT_SUMMARY.md](PROJECT_SUMMARY.md)** - Complete overview
|
||||||
|
- **[ADD_ONLY_EXAMPLES.md](ADD_ONLY_EXAMPLES.md)** - Add-only mode guide
|
||||||
|
- **[VERSION_FIELD.md](VERSION_FIELD.md)** - Version field behavior
|
||||||
|
|
||||||
## Project Structure
|
## Project Structure
|
||||||
|
|
||||||
@@ -142,9 +164,9 @@ madomeda/
|
|||||||
├── madomeda.py # Main entry point
|
├── madomeda.py # Main entry point
|
||||||
├── inspect_db.py # Database inspection tool
|
├── inspect_db.py # Database inspection tool
|
||||||
├── requirements.txt # Dependencies
|
├── requirements.txt # Dependencies
|
||||||
├── .env # LLM configuration
|
|
||||||
│
|
│
|
||||||
├── core/ # Core modules
|
├── core/ # Core modules
|
||||||
|
│ ├── config.py # Configuration management
|
||||||
│ ├── database.py # SQLite management
|
│ ├── database.py # SQLite management
|
||||||
│ ├── repository.py # Git operations
|
│ ├── repository.py # Git operations
|
||||||
│ ├── frontmatter.py # YAML parsing
|
│ ├── frontmatter.py # YAML parsing
|
||||||
@@ -154,12 +176,13 @@ madomeda/
|
|||||||
│ ├── changelog.py # Change logging
|
│ ├── changelog.py # Change logging
|
||||||
│ └── processor.py # Main orchestration
|
│ └── processor.py # Main orchestration
|
||||||
│
|
│
|
||||||
|
└── madomeda.db # Database (in repo, gitignored)
|
||||||
|
|
||||||
|
~/.config/madomeda/ # User configuration (auto-created)
|
||||||
|
├── .env # API configuration
|
||||||
├── templates/ # Frontmatter templates
|
├── templates/ # Frontmatter templates
|
||||||
├── rules/ # Validation rules
|
├── rules/ # Validation rules
|
||||||
├── prompts/ # LLM prompts
|
└── prompts/ # LLM prompts
|
||||||
│
|
|
||||||
├── madomeda.db # Database (gitignored)
|
|
||||||
└── madomeda_changelog.txt # Change log
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Templates
|
## Templates
|
||||||
|
|||||||
@@ -1,3 +1,12 @@
|
|||||||
|
---
|
||||||
|
title: Madomeda Project Structure
|
||||||
|
created: '2025-10-23T18:33:48+02:00'
|
||||||
|
changed: '2025-10-23T18:33:48+02:00'
|
||||||
|
authors:
|
||||||
|
- Test User
|
||||||
|
version: 8c4f612
|
||||||
|
tags: []
|
||||||
|
---
|
||||||
# Madomeda Project Structure
|
# Madomeda Project Structure
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|||||||
@@ -1,3 +1,12 @@
|
|||||||
|
---
|
||||||
|
title: Madomeda Usage Examples
|
||||||
|
created: '2025-10-23T18:33:48+02:00'
|
||||||
|
changed: '2025-10-23T18:33:48+02:00'
|
||||||
|
authors:
|
||||||
|
- Test User
|
||||||
|
version: 8c4f612
|
||||||
|
tags: []
|
||||||
|
---
|
||||||
# Madomeda Usage Examples
|
# Madomeda Usage Examples
|
||||||
|
|
||||||
## Basic Usage
|
## Basic Usage
|
||||||
|
|||||||
@@ -1,3 +1,12 @@
|
|||||||
|
---
|
||||||
|
title: Version Field Behavior
|
||||||
|
created: '2025-10-23T18:33:48+02:00'
|
||||||
|
changed: '2025-10-23T18:33:48+02:00'
|
||||||
|
authors:
|
||||||
|
- Test User
|
||||||
|
version: 8c4f612
|
||||||
|
tags: []
|
||||||
|
---
|
||||||
# Version Field Behavior
|
# Version Field Behavior
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|||||||
+14
@@ -1,3 +1,17 @@
|
|||||||
|
---
|
||||||
|
title: AI Testing Document
|
||||||
|
created: '2025-10-23T19:22:20+02:00'
|
||||||
|
changed: '2025-10-23T19:22:20+02:00'
|
||||||
|
authors:
|
||||||
|
- Test User
|
||||||
|
version: ab4b95c
|
||||||
|
tags:
|
||||||
|
- aiml
|
||||||
|
- data_science
|
||||||
|
- deep_learning
|
||||||
|
- machine_learning
|
||||||
|
- neural_networks
|
||||||
|
---
|
||||||
# AI Testing Document
|
# AI Testing Document
|
||||||
|
|
||||||
This document is about machine learning, artificial intelligence, and neural networks. It covers deep learning techniques, natural language processing, and computer vision applications.
|
This document is about machine learning, artificial intelligence, and neural networks. It covers deep learning techniques, natural language processing, and computer vision applications.
|
||||||
|
|||||||
+220
@@ -0,0 +1,220 @@
|
|||||||
|
"""
|
||||||
|
Configuration management module
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import shutil
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
|
||||||
|
class ConfigManager:
|
||||||
|
def __init__(self):
|
||||||
|
self.config_dir = self._get_config_dir()
|
||||||
|
self.env_file = self.config_dir / '.env'
|
||||||
|
self.templates_dir = self.config_dir / 'templates'
|
||||||
|
self.rules_dir = self.config_dir / 'rules'
|
||||||
|
self.prompts_dir = self.config_dir / 'prompts'
|
||||||
|
|
||||||
|
def _get_config_dir(self) -> Path:
|
||||||
|
"""Get platform-specific config directory"""
|
||||||
|
if sys.platform == 'win32':
|
||||||
|
# Windows: %USERPROFILE%\.config\madomeda
|
||||||
|
base = Path(os.environ.get('USERPROFILE', os.path.expanduser('~')))
|
||||||
|
config_dir = base / '.config' / 'madomeda'
|
||||||
|
elif sys.platform == 'darwin':
|
||||||
|
# macOS: ~/.config/madomeda
|
||||||
|
config_dir = Path.home() / '.config' / 'madomeda'
|
||||||
|
else:
|
||||||
|
# Linux: ~/.config/madomeda
|
||||||
|
config_dir = Path.home() / '.config' / 'madomeda'
|
||||||
|
|
||||||
|
return config_dir
|
||||||
|
|
||||||
|
def ensure_config_exists(self):
|
||||||
|
"""Ensure configuration directory exists with default files"""
|
||||||
|
if not self.config_dir.exists():
|
||||||
|
self.config_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
self._create_default_config()
|
||||||
|
|
||||||
|
# Ensure subdirectories exist
|
||||||
|
self.templates_dir.mkdir(exist_ok=True)
|
||||||
|
self.rules_dir.mkdir(exist_ok=True)
|
||||||
|
self.prompts_dir.mkdir(exist_ok=True)
|
||||||
|
|
||||||
|
# Copy defaults if they don't exist
|
||||||
|
if not (self.templates_dir / 'default.json').exists():
|
||||||
|
self._create_default_template()
|
||||||
|
|
||||||
|
if not list(self.rules_dir.glob('*.json')):
|
||||||
|
self._create_default_rules()
|
||||||
|
|
||||||
|
if not (self.prompts_dir / 'default.txt').exists():
|
||||||
|
self._create_default_prompt()
|
||||||
|
|
||||||
|
def _create_default_config(self):
|
||||||
|
"""Create default .env file"""
|
||||||
|
default_env = """# Madomeda Configuration
|
||||||
|
# OpenAI-compatible API settings
|
||||||
|
|
||||||
|
# API endpoint URL
|
||||||
|
OPENAI_API_URL=https://api.openai.com/v1
|
||||||
|
|
||||||
|
# Model to use
|
||||||
|
OPENAI_MODEL=gpt-4
|
||||||
|
|
||||||
|
# API Key options:
|
||||||
|
# 1. Empty string "" = No API key needed (LLM features disabled)
|
||||||
|
# 2. $ENV_VAR_NAME = Read from environment variable
|
||||||
|
# 3. Direct key = Use the key (WARNING: security risk in plain text)
|
||||||
|
OPENAI_API_KEY=""
|
||||||
|
"""
|
||||||
|
with open(self.env_file, 'w', encoding='utf-8') as f:
|
||||||
|
f.write(default_env)
|
||||||
|
|
||||||
|
def _create_default_template(self):
|
||||||
|
"""Create default template"""
|
||||||
|
import json
|
||||||
|
default_template = {
|
||||||
|
"title": "heur|orig",
|
||||||
|
"created": "heur|orig",
|
||||||
|
"changed": "heur|orig",
|
||||||
|
"authors": "heur|orig",
|
||||||
|
"version": "heur|orig",
|
||||||
|
"tags": "heur|orig|ai default"
|
||||||
|
}
|
||||||
|
with open(self.templates_dir / 'default.json', 'w', encoding='utf-8') as f:
|
||||||
|
json.dump(default_template, f, indent=2)
|
||||||
|
|
||||||
|
def _create_default_rules(self):
|
||||||
|
"""Create default rules"""
|
||||||
|
import json
|
||||||
|
|
||||||
|
rules = {
|
||||||
|
'tags_normalization.json': {
|
||||||
|
"name": "tags_normalization",
|
||||||
|
"description": "Normalize tags to lowercase with underscores",
|
||||||
|
"type": "tags"
|
||||||
|
},
|
||||||
|
'tag_to_tags.json': {
|
||||||
|
"name": "tag_to_tags",
|
||||||
|
"description": "Rename 'tag' key to 'tags'",
|
||||||
|
"type": "rename"
|
||||||
|
},
|
||||||
|
'lowercase_keys.json': {
|
||||||
|
"name": "lowercase_keys",
|
||||||
|
"description": "All frontmatter keys must be lowercase",
|
||||||
|
"type": "keys"
|
||||||
|
},
|
||||||
|
'summary_to_description.json': {
|
||||||
|
"name": "summary_to_description",
|
||||||
|
"description": "Rename 'summary' key to 'description'",
|
||||||
|
"type": "rename"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for filename, rule_data in rules.items():
|
||||||
|
with open(self.rules_dir / filename, 'w', encoding='utf-8') as f:
|
||||||
|
json.dump(rule_data, f, indent=2)
|
||||||
|
|
||||||
|
def _create_default_prompt(self):
|
||||||
|
"""Create default prompt"""
|
||||||
|
default_prompt = """You are a metadata extraction assistant for markdown documents.
|
||||||
|
|
||||||
|
Your task is to analyze the document content and suggest appropriate frontmatter metadata.
|
||||||
|
|
||||||
|
Guidelines:
|
||||||
|
- For tags: prefer selecting from the provided existing tags list when appropriate
|
||||||
|
- You may create new tags if they better fit the content
|
||||||
|
- New tags must follow the rules: lowercase letters, numbers, and underscores only
|
||||||
|
- Be concise and accurate
|
||||||
|
- Preserve important existing metadata when present"""
|
||||||
|
|
||||||
|
with open(self.prompts_dir / 'default.txt', 'w', encoding='utf-8') as f:
|
||||||
|
f.write(default_prompt)
|
||||||
|
|
||||||
|
def get_api_config(self) -> dict:
|
||||||
|
"""Get API configuration with environment variable resolution"""
|
||||||
|
config = {
|
||||||
|
'url': 'https://api.openai.com/v1',
|
||||||
|
'model': 'gpt-4',
|
||||||
|
'api_key': '',
|
||||||
|
'api_key_source': 'none'
|
||||||
|
}
|
||||||
|
|
||||||
|
if not self.env_file.exists():
|
||||||
|
return config
|
||||||
|
|
||||||
|
# Parse .env file
|
||||||
|
env_vars = {}
|
||||||
|
with open(self.env_file, 'r', encoding='utf-8') as f:
|
||||||
|
for line in f:
|
||||||
|
line = line.strip()
|
||||||
|
if not line or line.startswith('#'):
|
||||||
|
continue
|
||||||
|
if '=' in line:
|
||||||
|
key, value = line.split('=', 1)
|
||||||
|
env_vars[key.strip()] = value.strip()
|
||||||
|
|
||||||
|
# Get URL
|
||||||
|
if 'OPENAI_API_URL' in env_vars:
|
||||||
|
config['url'] = env_vars['OPENAI_API_URL']
|
||||||
|
|
||||||
|
# Get model
|
||||||
|
if 'OPENAI_MODEL' in env_vars:
|
||||||
|
config['model'] = env_vars['OPENAI_MODEL']
|
||||||
|
|
||||||
|
# Get API key with special handling
|
||||||
|
if 'OPENAI_API_KEY' in env_vars:
|
||||||
|
api_key_value = env_vars['OPENAI_API_KEY']
|
||||||
|
|
||||||
|
# Remove quotes if present
|
||||||
|
api_key_value = api_key_value.strip('"').strip("'")
|
||||||
|
|
||||||
|
if not api_key_value:
|
||||||
|
# Empty string = no key needed
|
||||||
|
config['api_key'] = ''
|
||||||
|
config['api_key_source'] = 'none'
|
||||||
|
elif api_key_value.startswith('$'):
|
||||||
|
# Environment variable reference
|
||||||
|
env_var_name = api_key_value[1:]
|
||||||
|
env_value = os.environ.get(env_var_name)
|
||||||
|
if env_value is None:
|
||||||
|
raise ValueError(
|
||||||
|
f"Environment variable '{env_var_name}' not found. "
|
||||||
|
f"Referenced in .env as OPENAI_API_KEY={api_key_value}"
|
||||||
|
)
|
||||||
|
config['api_key'] = env_value
|
||||||
|
config['api_key_source'] = f'env:{env_var_name}'
|
||||||
|
else:
|
||||||
|
# Plain text key (security warning will be issued)
|
||||||
|
config['api_key'] = api_key_value
|
||||||
|
config['api_key_source'] = 'plaintext'
|
||||||
|
|
||||||
|
return config
|
||||||
|
|
||||||
|
def print_config_info(self):
|
||||||
|
"""Print configuration location and status"""
|
||||||
|
print(f"Configuration directory: {self.config_dir}")
|
||||||
|
print(f" .env file: {self.env_file}")
|
||||||
|
print(f" Templates: {self.templates_dir}")
|
||||||
|
print(f" Rules: {self.rules_dir}")
|
||||||
|
print(f" Prompts: {self.prompts_dir}")
|
||||||
|
|
||||||
|
# Get API config to show status
|
||||||
|
try:
|
||||||
|
api_config = self.get_api_config()
|
||||||
|
|
||||||
|
if api_config['api_key_source'] == 'none':
|
||||||
|
print(f" API Key: Not configured (LLM features disabled)")
|
||||||
|
elif api_config['api_key_source'] == 'plaintext':
|
||||||
|
print(f" API Key: Configured (plain text)")
|
||||||
|
print(f" WARNING: Storing API keys in plain text is a security risk!")
|
||||||
|
print(f" Consider using environment variable: OPENAI_API_KEY=$YOUR_ENV_VAR")
|
||||||
|
elif api_config['api_key_source'].startswith('env:'):
|
||||||
|
env_var = api_config['api_key_source'].split(':', 1)[1]
|
||||||
|
print(f" API Key: From environment variable ${env_var}")
|
||||||
|
except ValueError as e:
|
||||||
|
print(f" API Key: ERROR - {e}")
|
||||||
|
|
||||||
|
print()
|
||||||
+4
-26
@@ -5,17 +5,14 @@ import os
|
|||||||
import json
|
import json
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Optional
|
from typing import Optional
|
||||||
from dotenv import load_dotenv
|
|
||||||
from openai import OpenAI
|
from openai import OpenAI
|
||||||
|
|
||||||
|
|
||||||
class LLMClient:
|
class LLMClient:
|
||||||
def __init__(self, prompts_dir: Path):
|
def __init__(self, prompts_dir: Path, api_config: dict):
|
||||||
load_dotenv()
|
self.api_url = api_config.get('url', 'https://api.openai.com/v1')
|
||||||
|
self.model = api_config.get('model', 'gpt-4')
|
||||||
self.api_url = os.getenv('OPENAI_API_URL', 'https://api.openai.com/v1')
|
self.api_key = api_config.get('api_key', '')
|
||||||
self.model = os.getenv('OPENAI_MODEL', 'gpt-4')
|
|
||||||
self.api_key = os.getenv('OPENAI_API_KEY', '')
|
|
||||||
|
|
||||||
self.client = OpenAI(
|
self.client = OpenAI(
|
||||||
base_url=self.api_url,
|
base_url=self.api_url,
|
||||||
@@ -23,25 +20,6 @@ class LLMClient:
|
|||||||
) if self.api_key else None
|
) if self.api_key else None
|
||||||
|
|
||||||
self.prompts_dir = prompts_dir
|
self.prompts_dir = prompts_dir
|
||||||
if not prompts_dir.exists():
|
|
||||||
prompts_dir.mkdir(parents=True)
|
|
||||||
self._create_default_prompt()
|
|
||||||
|
|
||||||
def _create_default_prompt(self):
|
|
||||||
"""Create default system prompt"""
|
|
||||||
default_prompt = """You are a metadata extraction assistant for markdown documents.
|
|
||||||
|
|
||||||
Your task is to analyze the document content and suggest appropriate frontmatter metadata.
|
|
||||||
|
|
||||||
Guidelines:
|
|
||||||
- For tags: prefer selecting from the provided existing tags list when appropriate
|
|
||||||
- You may create new tags if they better fit the content
|
|
||||||
- New tags must follow the rules: lowercase letters, numbers, and underscores only
|
|
||||||
- Be concise and accurate
|
|
||||||
- Preserve important existing metadata when present"""
|
|
||||||
|
|
||||||
with open(self.prompts_dir / 'default.txt', 'w', encoding='utf-8') as f:
|
|
||||||
f.write(default_prompt)
|
|
||||||
|
|
||||||
def load_prompt(self, name: str = 'default') -> str:
|
def load_prompt(self, name: str = 'default') -> str:
|
||||||
"""Load prompt template"""
|
"""Load prompt template"""
|
||||||
|
|||||||
+18
-5
@@ -1,6 +1,7 @@
|
|||||||
"""
|
"""
|
||||||
Main frontmatter processor
|
Main frontmatter processor
|
||||||
"""
|
"""
|
||||||
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from core.database import Database
|
from core.database import Database
|
||||||
@@ -10,6 +11,7 @@ from core.rules import RulesEngine
|
|||||||
from core.template import TemplateManager
|
from core.template import TemplateManager
|
||||||
from core.llm import LLMClient
|
from core.llm import LLMClient
|
||||||
from core.changelog import ChangeLog
|
from core.changelog import ChangeLog
|
||||||
|
from core.config import ConfigManager
|
||||||
|
|
||||||
|
|
||||||
class FrontmatterProcessor:
|
class FrontmatterProcessor:
|
||||||
@@ -22,15 +24,26 @@ class FrontmatterProcessor:
|
|||||||
self.force = force
|
self.force = force
|
||||||
self.add_only = add_only
|
self.add_only = add_only
|
||||||
|
|
||||||
|
# Initialize configuration
|
||||||
|
self.config = ConfigManager()
|
||||||
|
self.config.ensure_config_exists()
|
||||||
|
|
||||||
|
# Get API configuration
|
||||||
|
api_config = self.config.get_api_config()
|
||||||
|
|
||||||
self.db = Database(repo.path / 'madomeda.db')
|
self.db = Database(repo.path / 'madomeda.db')
|
||||||
self.rules = RulesEngine(repo.path / 'rules')
|
self.rules = RulesEngine(self.config.rules_dir)
|
||||||
self.templates = TemplateManager(repo.path / 'templates')
|
self.templates = TemplateManager(self.config.templates_dir)
|
||||||
self.llm = LLMClient(repo.path / 'prompts')
|
self.llm = LLMClient(self.config.prompts_dir, api_config)
|
||||||
self.changelog = ChangeLog(repo.path)
|
self.changelog = ChangeLog(repo.path)
|
||||||
self.parser = FrontmatterParser()
|
self.parser = FrontmatterParser()
|
||||||
|
|
||||||
def process(self):
|
def process(self):
|
||||||
"""Main processing loop"""
|
"""Main processing loop"""
|
||||||
|
# Print configuration info if running in interactive shell
|
||||||
|
if sys.stdin.isatty():
|
||||||
|
self.config.print_config_info()
|
||||||
|
|
||||||
print(f"Processing repository: {self.repo.path}")
|
print(f"Processing repository: {self.repo.path}")
|
||||||
print(f"Template: {self.template_name}")
|
print(f"Template: {self.template_name}")
|
||||||
print(f"Mode: {'DRY RUN' if self.whatif else 'LIVE'}")
|
print(f"Mode: {'DRY RUN' if self.whatif else 'LIVE'}")
|
||||||
@@ -235,8 +248,8 @@ class FrontmatterProcessor:
|
|||||||
|
|
||||||
elif field == 'tags':
|
elif field == 'tags':
|
||||||
tags = normalized_fm.get('tags', [])
|
tags = normalized_fm.get('tags', [])
|
||||||
# Always return the list (even if empty) so it's preserved
|
# Return None if empty so AI inference is tried
|
||||||
return tags
|
return tags if tags else None
|
||||||
|
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|||||||
+4
-46
@@ -14,56 +14,14 @@ class RulesEngine:
|
|||||||
|
|
||||||
def _load_rules(self) -> List[dict]:
|
def _load_rules(self) -> List[dict]:
|
||||||
"""Load all rule files"""
|
"""Load all rule files"""
|
||||||
if not self.rules_dir.exists():
|
|
||||||
self.rules_dir.mkdir(parents=True)
|
|
||||||
self._create_default_rules()
|
|
||||||
|
|
||||||
rules = []
|
rules = []
|
||||||
for rule_file in self.rules_dir.glob('*.json'):
|
if self.rules_dir.exists():
|
||||||
with open(rule_file, 'r', encoding='utf-8') as f:
|
for rule_file in self.rules_dir.glob('*.json'):
|
||||||
rules.append(json.load(f))
|
with open(rule_file, 'r', encoding='utf-8') as f:
|
||||||
|
rules.append(json.load(f))
|
||||||
|
|
||||||
return rules
|
return rules
|
||||||
|
|
||||||
def _create_default_rules(self):
|
|
||||||
"""Create default rule files"""
|
|
||||||
# Tags normalization rule
|
|
||||||
tags_rule = {
|
|
||||||
"name": "tags_normalization",
|
|
||||||
"description": "Normalize tags to lowercase with underscores",
|
|
||||||
"type": "tags"
|
|
||||||
}
|
|
||||||
|
|
||||||
tag_to_tags_rule = {
|
|
||||||
"name": "tag_to_tags",
|
|
||||||
"description": "Rename 'tag' key to 'tags'",
|
|
||||||
"type": "rename"
|
|
||||||
}
|
|
||||||
|
|
||||||
lowercase_keys_rule = {
|
|
||||||
"name": "lowercase_keys",
|
|
||||||
"description": "All frontmatter keys must be lowercase",
|
|
||||||
"type": "keys"
|
|
||||||
}
|
|
||||||
|
|
||||||
summary_to_description_rule = {
|
|
||||||
"name": "summary_to_description",
|
|
||||||
"description": "Rename 'summary' key to 'description'",
|
|
||||||
"type": "rename"
|
|
||||||
}
|
|
||||||
|
|
||||||
with open(self.rules_dir / 'tags_normalization.json', 'w', encoding='utf-8') as f:
|
|
||||||
json.dump(tags_rule, f, indent=2)
|
|
||||||
|
|
||||||
with open(self.rules_dir / 'tag_to_tags.json', 'w', encoding='utf-8') as f:
|
|
||||||
json.dump(tag_to_tags_rule, f, indent=2)
|
|
||||||
|
|
||||||
with open(self.rules_dir / 'lowercase_keys.json', 'w', encoding='utf-8') as f:
|
|
||||||
json.dump(lowercase_keys_rule, f, indent=2)
|
|
||||||
|
|
||||||
with open(self.rules_dir / 'summary_to_description.json', 'w', encoding='utf-8') as f:
|
|
||||||
json.dump(summary_to_description_rule, f, indent=2)
|
|
||||||
|
|
||||||
def apply_rules(self, frontmatter: dict) -> Tuple[dict, bool]:
|
def apply_rules(self, frontmatter: dict) -> Tuple[dict, bool]:
|
||||||
"""Apply all rules to frontmatter, return (modified_fm, is_conformant)"""
|
"""Apply all rules to frontmatter, return (modified_fm, is_conformant)"""
|
||||||
if not frontmatter:
|
if not frontmatter:
|
||||||
|
|||||||
@@ -9,24 +9,6 @@ from typing import Optional
|
|||||||
class TemplateManager:
|
class TemplateManager:
|
||||||
def __init__(self, templates_dir: Path):
|
def __init__(self, templates_dir: Path):
|
||||||
self.templates_dir = templates_dir
|
self.templates_dir = templates_dir
|
||||||
|
|
||||||
if not templates_dir.exists():
|
|
||||||
templates_dir.mkdir(parents=True)
|
|
||||||
self._create_default_template()
|
|
||||||
|
|
||||||
def _create_default_template(self):
|
|
||||||
"""Create default template"""
|
|
||||||
default_template = {
|
|
||||||
"title": "heur|orig",
|
|
||||||
"created": "heur|orig",
|
|
||||||
"changed": "heur|orig",
|
|
||||||
"authors": "heur|orig",
|
|
||||||
"version": "heur|orig",
|
|
||||||
"tags": "orig|ai default"
|
|
||||||
}
|
|
||||||
|
|
||||||
with open(self.templates_dir / 'default.json', 'w', encoding='utf-8') as f:
|
|
||||||
json.dump(default_template, f, indent=2)
|
|
||||||
|
|
||||||
def load_template(self, name: str) -> dict:
|
def load_template(self, name: str) -> dict:
|
||||||
"""Load template by name"""
|
"""Load template by name"""
|
||||||
@@ -35,8 +17,6 @@ class TemplateManager:
|
|||||||
if not template_path.exists():
|
if not template_path.exists():
|
||||||
# Fallback to default
|
# Fallback to default
|
||||||
template_path = self.templates_dir / 'default.json'
|
template_path = self.templates_dir / 'default.json'
|
||||||
if not template_path.exists():
|
|
||||||
self._create_default_template()
|
|
||||||
|
|
||||||
with open(template_path, 'r', encoding='utf-8') as f:
|
with open(template_path, 'r', encoding='utf-8') as f:
|
||||||
return json.load(f)
|
return json.load(f)
|
||||||
|
|||||||
@@ -35,3 +35,17 @@
|
|||||||
2025-10-23 18:32:49 (commit: 0463d73ef927403e235bfd18ad9d202ae57ec618)
|
2025-10-23 18:32:49 (commit: 0463d73ef927403e235bfd18ad9d202ae57ec618)
|
||||||
notag.md - frontmatter updated
|
notag.md - frontmatter updated
|
||||||
|
|
||||||
|
2025-10-23 19:23:00 (commit: ab4b95c664427f6b08bd7510fd8c925403887ea5)
|
||||||
|
ADD_ONLY_EXAMPLES.md - frontmatter updated
|
||||||
|
GETTING_STARTED.md - frontmatter updated
|
||||||
|
IMPLEMENTATION.md - frontmatter updated
|
||||||
|
PROJECT_SUMMARY.md - frontmatter updated
|
||||||
|
README.md - frontmatter updated
|
||||||
|
STRUCTURE.md - frontmatter updated
|
||||||
|
USAGE.md - frontmatter updated
|
||||||
|
VERSION_FIELD.md - frontmatter updated
|
||||||
|
ai_test.md - frontmatter updated
|
||||||
|
|
||||||
|
2025-10-23 19:23:44 (commit: ab4b95c664427f6b08bd7510fd8c925403887ea5)
|
||||||
|
ai_test.md - frontmatter updated
|
||||||
|
|
||||||
|
|||||||
@@ -1,3 +1,2 @@
|
|||||||
PyYAML>=6.0
|
PyYAML>=6.0
|
||||||
python-dotenv>=1.0.0
|
|
||||||
openai>=1.0.0
|
openai>=1.0.0
|
||||||
|
|||||||
Reference in New Issue
Block a user