Probably functional. Testing
This commit is contained in:
Test User
@@ -1,3 +1,12 @@
|
||||
---
|
||||
title: Add-Only Mode Examples
|
||||
created: '2025-10-23T18:33:48+02:00'
|
||||
changed: '2025-10-23T18:33:48+02:00'
|
||||
authors:
|
||||
- Test User
|
||||
version: 8c4f612
|
||||
tags: []
|
||||
---
|
||||
# Add-Only Mode Examples
|
||||
|
||||
## Overview
|
||||
|
||||
@@ -0,0 +1,287 @@
|
||||
# Configuration Management
|
||||
|
||||
## Configuration Directory
|
||||
|
||||
Madomeda stores its configuration in a platform-specific location:
|
||||
|
||||
- **Windows**: `%USERPROFILE%\.config\madomeda`
|
||||
- **macOS**: `~/.config/madomeda`
|
||||
- **Linux**: `~/.config/madomeda`
|
||||
|
||||
## First Run
|
||||
|
||||
On first run, Madomeda automatically creates:
|
||||
|
||||
```
|
||||
~/.config/madomeda/
|
||||
├── .env # API configuration
|
||||
├── templates/
|
||||
│ └── default.json # Default frontmatter template
|
||||
├── rules/
|
||||
│ ├── lowercase_keys.json
|
||||
│ ├── summary_to_description.json
|
||||
│ ├── tag_to_tags.json
|
||||
│ └── tags_normalization.json
|
||||
└── prompts/
|
||||
└── default.txt # Default LLM prompt
|
||||
```
|
||||
|
||||
## Configuration Display
|
||||
|
||||
When running in an **interactive shell**, Madomeda displays configuration info:
|
||||
|
||||
```
|
||||
Configuration directory: C:\Users\YourName\.config\madomeda
|
||||
.env file: C:\Users\YourName\.config\madomeda\.env
|
||||
Templates: C:\Users\YourName\.config\madomeda\templates
|
||||
Rules: C:\Users\YourName\.config\madomeda\rules
|
||||
Prompts: C:\Users\YourName\.config\madomeda\prompts
|
||||
API Key: Not configured (LLM features disabled)
|
||||
```
|
||||
|
||||
In **non-interactive mode** (pipes, scripts), this info is suppressed.
|
||||
|
||||
## API Key Configuration
|
||||
|
||||
The `.env` file supports three modes for `OPENAI_API_KEY`:
|
||||
|
||||
### 1. No API Key (Default)
|
||||
|
||||
```bash
|
||||
OPENAI_API_KEY=""
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
API Key: Not configured (LLM features disabled)
|
||||
```
|
||||
|
||||
**Behavior:** LLM features are disabled, only heuristics and original values used.
|
||||
|
||||
### 2. Environment Variable Reference
|
||||
|
||||
```bash
|
||||
OPENAI_API_KEY=$MY_API_KEY
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
API Key: From environment variable $MY_API_KEY
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
- Reads key from environment variable `MY_API_KEY`
|
||||
- **Error if not found:**
|
||||
```
|
||||
ValueError: Environment variable 'MY_API_KEY' not found.
|
||||
Referenced in .env as OPENAI_API_KEY=$MY_API_KEY
|
||||
```
|
||||
|
||||
**Setup:**
|
||||
```bash
|
||||
# Linux/macOS
|
||||
export MY_API_KEY="sk-your-actual-key"
|
||||
|
||||
# Windows (PowerShell)
|
||||
$env:MY_API_KEY = "sk-your-actual-key"
|
||||
|
||||
# Windows (CMD)
|
||||
set MY_API_KEY=sk-your-actual-key
|
||||
```
|
||||
|
||||
### 3. Plain Text Key
|
||||
|
||||
```bash
|
||||
OPENAI_API_KEY=sk-1234567890abcdef
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
API Key: Configured (plain text)
|
||||
WARNING: Storing API keys in plain text is a security risk!
|
||||
Consider using environment variable: OPENAI_API_KEY=$YOUR_ENV_VAR
|
||||
```
|
||||
|
||||
**Behavior:** Works but shows security warning every run.
|
||||
|
||||
## Best Practices
|
||||
|
||||
### ✅ Recommended: Environment Variable
|
||||
|
||||
```bash
|
||||
# In .env file
|
||||
OPENAI_API_KEY=$OPENAI_API_KEY
|
||||
|
||||
# In your shell profile (~/.bashrc, ~/.zshrc, etc.)
|
||||
export OPENAI_API_KEY="sk-your-actual-key"
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Key not stored in config file
|
||||
- Can be different per user/machine
|
||||
- Easy to rotate without editing files
|
||||
- Can use system keyring tools
|
||||
|
||||
### ⚠️ Acceptable: Empty String
|
||||
|
||||
```bash
|
||||
OPENAI_API_KEY=""
|
||||
```
|
||||
|
||||
**Use when:**
|
||||
- Don't have LLM access
|
||||
- Only using heuristics
|
||||
- Testing/development
|
||||
|
||||
### ❌ Not Recommended: Plain Text
|
||||
|
||||
```bash
|
||||
OPENAI_API_KEY=sk-1234567890abcdef
|
||||
```
|
||||
|
||||
**Risks:**
|
||||
- Key visible to anyone with file access
|
||||
- Accidentally committed to version control
|
||||
- Hard to rotate across systems
|
||||
|
||||
## Customizing Configuration
|
||||
|
||||
### Custom Templates
|
||||
|
||||
Create in `~/.config/madomeda/templates/`:
|
||||
|
||||
```bash
|
||||
# Create minimal template
|
||||
cat > ~/.config/madomeda/templates/minimal.json << 'EOF'
|
||||
{
|
||||
"title": "heur|orig",
|
||||
"date": "heur|orig",
|
||||
"tags": "orig"
|
||||
}
|
||||
EOF
|
||||
|
||||
# Use it
|
||||
madomeda --template minimal
|
||||
```
|
||||
|
||||
### Custom Rules
|
||||
|
||||
Create in `~/.config/madomeda/rules/`:
|
||||
|
||||
```bash
|
||||
cat > ~/.config/madomeda/rules/my_rule.json << 'EOF'
|
||||
{
|
||||
"name": "my_custom_rule",
|
||||
"description": "My custom rule description",
|
||||
"type": "custom"
|
||||
}
|
||||
EOF
|
||||
```
|
||||
|
||||
Note: Custom rules require code changes in `core/rules.py` to implement logic.
|
||||
|
||||
### Custom Prompts
|
||||
|
||||
Create in `~/.config/madomeda/prompts/`:
|
||||
|
||||
```bash
|
||||
cat > ~/.config/madomeda/prompts/technical.txt << 'EOF'
|
||||
You are a technical documentation expert.
|
||||
Focus on API documentation and code examples.
|
||||
...
|
||||
EOF
|
||||
```
|
||||
|
||||
Reference in template:
|
||||
```json
|
||||
{
|
||||
"summary": "ai technical"
|
||||
}
|
||||
```
|
||||
|
||||
## Multiple API Providers
|
||||
|
||||
Edit `OPENAI_API_URL` in `.env`:
|
||||
|
||||
```bash
|
||||
# OpenAI
|
||||
OPENAI_API_URL=https://api.openai.com/v1
|
||||
|
||||
# Azure OpenAI
|
||||
OPENAI_API_URL=https://your-resource.openai.azure.com/
|
||||
|
||||
# Local LLM (Ollama)
|
||||
OPENAI_API_URL=http://localhost:11434/v1
|
||||
|
||||
# Other OpenAI-compatible APIs
|
||||
OPENAI_API_URL=https://api.your-provider.com/v1
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Config Not Created
|
||||
|
||||
Ensure write permissions:
|
||||
```bash
|
||||
mkdir -p ~/.config/madomeda
|
||||
chmod 755 ~/.config/madomeda
|
||||
```
|
||||
|
||||
### Environment Variable Not Found
|
||||
|
||||
Check it's set:
|
||||
```bash
|
||||
# Linux/macOS
|
||||
echo $MY_API_KEY
|
||||
|
||||
# Windows (PowerShell)
|
||||
echo $env:MY_API_KEY
|
||||
```
|
||||
|
||||
If not set, add to shell profile or set before running:
|
||||
```bash
|
||||
MY_API_KEY="sk-key" madomeda
|
||||
```
|
||||
|
||||
### Wrong Config Location
|
||||
|
||||
Madomeda uses `$USERPROFILE` (Windows) or `$HOME` (Unix).
|
||||
|
||||
Check:
|
||||
```bash
|
||||
# Linux/macOS
|
||||
echo ~/.config/madomeda
|
||||
|
||||
# Windows (PowerShell)
|
||||
echo $env:USERPROFILE\.config\madomeda
|
||||
```
|
||||
|
||||
### API Key Warning on Every Run
|
||||
|
||||
This is intentional for plain text keys. Switch to environment variable to suppress.
|
||||
|
||||
## Migration from Old Version
|
||||
|
||||
If you had local `.env`, `templates/`, `rules/`, `prompts/` in the repo:
|
||||
|
||||
1. Copy to new location:
|
||||
```bash
|
||||
cp -r templates ~/.config/madomeda/
|
||||
cp -r rules ~/.config/madomeda/
|
||||
cp -r prompts ~/.config/madomeda/
|
||||
cp .env ~/.config/madomeda/
|
||||
```
|
||||
|
||||
2. Update `.gitignore` (no longer need to ignore these locally)
|
||||
|
||||
3. Old local files are now ignored by the program
|
||||
|
||||
## Security Notes
|
||||
|
||||
- Never commit `.env` with plain text keys
|
||||
- Use environment variables or secret management tools
|
||||
- Config directory is user-specific (not shared)
|
||||
- On shared systems, ensure `~/.config/madomeda/` has proper permissions:
|
||||
```bash
|
||||
chmod 700 ~/.config/madomeda
|
||||
```
|
||||
@@ -1,3 +1,12 @@
|
||||
---
|
||||
title: Getting Started with Madomeda
|
||||
created: '2025-10-23T18:33:48+02:00'
|
||||
changed: '2025-10-23T18:33:48+02:00'
|
||||
authors:
|
||||
- Test User
|
||||
version: 8c4f612
|
||||
tags: []
|
||||
---
|
||||
# Getting Started with Madomeda
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -1,3 +1,12 @@
|
||||
---
|
||||
title: Madomeda - Implementation Summary
|
||||
created: '2025-10-23T18:33:48+02:00'
|
||||
changed: '2025-10-23T18:33:48+02:00'
|
||||
authors:
|
||||
- Test User
|
||||
version: 8c4f612
|
||||
tags: []
|
||||
---
|
||||
# Madomeda - Implementation Summary
|
||||
|
||||
## What Was Created
|
||||
|
||||
@@ -1,3 +1,12 @@
|
||||
---
|
||||
title: Madomeda - Project Complete
|
||||
created: '2025-10-23T18:33:48+02:00'
|
||||
changed: '2025-10-23T18:33:48+02:00'
|
||||
authors:
|
||||
- Test User
|
||||
version: 8c4f612
|
||||
tags: []
|
||||
---
|
||||
# Madomeda - Project Complete
|
||||
|
||||
## Overview
|
||||
|
||||
@@ -1,3 +1,12 @@
|
||||
---
|
||||
title: Madomeda - Markdown Document Metadata Manager
|
||||
created: '2025-10-23T18:33:48+02:00'
|
||||
changed: '2025-10-23T18:33:48+02:00'
|
||||
authors:
|
||||
- Test User
|
||||
version: 8c4f612
|
||||
tags: []
|
||||
---
|
||||
# Madomeda - Markdown Document Metadata Manager
|
||||
|
||||
A Python tool for managing and normalizing frontmatter in markdown files within git repositories.
|
||||
@@ -64,15 +73,25 @@ SQLite database stores:
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
The first time you run Madomeda, it will automatically create a configuration directory at `~/.config/madomeda` (all platforms) with default settings.
|
||||
|
||||
### Optional: LLM Configuration
|
||||
|
||||
Edit `.env`:
|
||||
```
|
||||
OPENAI_API_URL=https://api.openai.com/v1
|
||||
OPENAI_MODEL=gpt-4
|
||||
OPENAI_API_KEY=your-api-key-here
|
||||
Edit `~/.config/madomeda/.env`:
|
||||
|
||||
```bash
|
||||
# Option 1: Use environment variable (recommended)
|
||||
OPENAI_API_KEY=$OPENAI_API_KEY
|
||||
|
||||
# Option 2: No LLM (empty string)
|
||||
OPENAI_API_KEY=""
|
||||
|
||||
# Option 3: Plain text (not recommended - security warning)
|
||||
OPENAI_API_KEY=sk-your-key-here
|
||||
```
|
||||
|
||||
For detailed configuration options, see [CONFIGURATION.md](CONFIGURATION.md).
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
@@ -131,9 +150,12 @@ tags:
|
||||
|
||||
- **[GETTING_STARTED.md](GETTING_STARTED.md)** - Step-by-step guide
|
||||
- **[USAGE.md](USAGE.md)** - Detailed usage examples
|
||||
- **[CONFIGURATION.md](CONFIGURATION.md)** - Configuration management
|
||||
- **[STRUCTURE.md](STRUCTURE.md)** - Project architecture
|
||||
- **[IMPLEMENTATION.md](IMPLEMENTATION.md)** - Technical details
|
||||
- **[PROJECT_SUMMARY.md](PROJECT_SUMMARY.md)** - Complete overview
|
||||
- **[ADD_ONLY_EXAMPLES.md](ADD_ONLY_EXAMPLES.md)** - Add-only mode guide
|
||||
- **[VERSION_FIELD.md](VERSION_FIELD.md)** - Version field behavior
|
||||
|
||||
## Project Structure
|
||||
|
||||
@@ -142,9 +164,9 @@ madomeda/
|
||||
├── madomeda.py # Main entry point
|
||||
├── inspect_db.py # Database inspection tool
|
||||
├── requirements.txt # Dependencies
|
||||
├── .env # LLM configuration
|
||||
│
|
||||
├── core/ # Core modules
|
||||
│ ├── config.py # Configuration management
|
||||
│ ├── database.py # SQLite management
|
||||
│ ├── repository.py # Git operations
|
||||
│ ├── frontmatter.py # YAML parsing
|
||||
@@ -154,12 +176,13 @@ madomeda/
|
||||
│ ├── changelog.py # Change logging
|
||||
│ └── processor.py # Main orchestration
|
||||
│
|
||||
└── madomeda.db # Database (in repo, gitignored)
|
||||
|
||||
~/.config/madomeda/ # User configuration (auto-created)
|
||||
├── .env # API configuration
|
||||
├── templates/ # Frontmatter templates
|
||||
├── rules/ # Validation rules
|
||||
├── prompts/ # LLM prompts
|
||||
│
|
||||
├── madomeda.db # Database (gitignored)
|
||||
└── madomeda_changelog.txt # Change log
|
||||
└── prompts/ # LLM prompts
|
||||
```
|
||||
|
||||
## Templates
|
||||
|
||||
@@ -1,3 +1,12 @@
|
||||
---
|
||||
title: Madomeda Project Structure
|
||||
created: '2025-10-23T18:33:48+02:00'
|
||||
changed: '2025-10-23T18:33:48+02:00'
|
||||
authors:
|
||||
- Test User
|
||||
version: 8c4f612
|
||||
tags: []
|
||||
---
|
||||
# Madomeda Project Structure
|
||||
|
||||
## Overview
|
||||
|
||||
@@ -1,3 +1,12 @@
|
||||
---
|
||||
title: Madomeda Usage Examples
|
||||
created: '2025-10-23T18:33:48+02:00'
|
||||
changed: '2025-10-23T18:33:48+02:00'
|
||||
authors:
|
||||
- Test User
|
||||
version: 8c4f612
|
||||
tags: []
|
||||
---
|
||||
# Madomeda Usage Examples
|
||||
|
||||
## Basic Usage
|
||||
|
||||
@@ -1,3 +1,12 @@
|
||||
---
|
||||
title: Version Field Behavior
|
||||
created: '2025-10-23T18:33:48+02:00'
|
||||
changed: '2025-10-23T18:33:48+02:00'
|
||||
authors:
|
||||
- Test User
|
||||
version: 8c4f612
|
||||
tags: []
|
||||
---
|
||||
# Version Field Behavior
|
||||
|
||||
## Overview
|
||||
|
||||
+14
@@ -1,3 +1,17 @@
|
||||
---
|
||||
title: AI Testing Document
|
||||
created: '2025-10-23T19:22:20+02:00'
|
||||
changed: '2025-10-23T19:22:20+02:00'
|
||||
authors:
|
||||
- Test User
|
||||
version: ab4b95c
|
||||
tags:
|
||||
- aiml
|
||||
- data_science
|
||||
- deep_learning
|
||||
- machine_learning
|
||||
- neural_networks
|
||||
---
|
||||
# AI Testing Document
|
||||
|
||||
This document is about machine learning, artificial intelligence, and neural networks. It covers deep learning techniques, natural language processing, and computer vision applications.
|
||||
|
||||
+220
@@ -0,0 +1,220 @@
|
||||
"""
|
||||
Configuration management module
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
class ConfigManager:
|
||||
def __init__(self):
|
||||
self.config_dir = self._get_config_dir()
|
||||
self.env_file = self.config_dir / '.env'
|
||||
self.templates_dir = self.config_dir / 'templates'
|
||||
self.rules_dir = self.config_dir / 'rules'
|
||||
self.prompts_dir = self.config_dir / 'prompts'
|
||||
|
||||
def _get_config_dir(self) -> Path:
|
||||
"""Get platform-specific config directory"""
|
||||
if sys.platform == 'win32':
|
||||
# Windows: %USERPROFILE%\.config\madomeda
|
||||
base = Path(os.environ.get('USERPROFILE', os.path.expanduser('~')))
|
||||
config_dir = base / '.config' / 'madomeda'
|
||||
elif sys.platform == 'darwin':
|
||||
# macOS: ~/.config/madomeda
|
||||
config_dir = Path.home() / '.config' / 'madomeda'
|
||||
else:
|
||||
# Linux: ~/.config/madomeda
|
||||
config_dir = Path.home() / '.config' / 'madomeda'
|
||||
|
||||
return config_dir
|
||||
|
||||
def ensure_config_exists(self):
|
||||
"""Ensure configuration directory exists with default files"""
|
||||
if not self.config_dir.exists():
|
||||
self.config_dir.mkdir(parents=True, exist_ok=True)
|
||||
self._create_default_config()
|
||||
|
||||
# Ensure subdirectories exist
|
||||
self.templates_dir.mkdir(exist_ok=True)
|
||||
self.rules_dir.mkdir(exist_ok=True)
|
||||
self.prompts_dir.mkdir(exist_ok=True)
|
||||
|
||||
# Copy defaults if they don't exist
|
||||
if not (self.templates_dir / 'default.json').exists():
|
||||
self._create_default_template()
|
||||
|
||||
if not list(self.rules_dir.glob('*.json')):
|
||||
self._create_default_rules()
|
||||
|
||||
if not (self.prompts_dir / 'default.txt').exists():
|
||||
self._create_default_prompt()
|
||||
|
||||
def _create_default_config(self):
|
||||
"""Create default .env file"""
|
||||
default_env = """# Madomeda Configuration
|
||||
# OpenAI-compatible API settings
|
||||
|
||||
# API endpoint URL
|
||||
OPENAI_API_URL=https://api.openai.com/v1
|
||||
|
||||
# Model to use
|
||||
OPENAI_MODEL=gpt-4
|
||||
|
||||
# API Key options:
|
||||
# 1. Empty string "" = No API key needed (LLM features disabled)
|
||||
# 2. $ENV_VAR_NAME = Read from environment variable
|
||||
# 3. Direct key = Use the key (WARNING: security risk in plain text)
|
||||
OPENAI_API_KEY=""
|
||||
"""
|
||||
with open(self.env_file, 'w', encoding='utf-8') as f:
|
||||
f.write(default_env)
|
||||
|
||||
def _create_default_template(self):
|
||||
"""Create default template"""
|
||||
import json
|
||||
default_template = {
|
||||
"title": "heur|orig",
|
||||
"created": "heur|orig",
|
||||
"changed": "heur|orig",
|
||||
"authors": "heur|orig",
|
||||
"version": "heur|orig",
|
||||
"tags": "heur|orig|ai default"
|
||||
}
|
||||
with open(self.templates_dir / 'default.json', 'w', encoding='utf-8') as f:
|
||||
json.dump(default_template, f, indent=2)
|
||||
|
||||
def _create_default_rules(self):
|
||||
"""Create default rules"""
|
||||
import json
|
||||
|
||||
rules = {
|
||||
'tags_normalization.json': {
|
||||
"name": "tags_normalization",
|
||||
"description": "Normalize tags to lowercase with underscores",
|
||||
"type": "tags"
|
||||
},
|
||||
'tag_to_tags.json': {
|
||||
"name": "tag_to_tags",
|
||||
"description": "Rename 'tag' key to 'tags'",
|
||||
"type": "rename"
|
||||
},
|
||||
'lowercase_keys.json': {
|
||||
"name": "lowercase_keys",
|
||||
"description": "All frontmatter keys must be lowercase",
|
||||
"type": "keys"
|
||||
},
|
||||
'summary_to_description.json': {
|
||||
"name": "summary_to_description",
|
||||
"description": "Rename 'summary' key to 'description'",
|
||||
"type": "rename"
|
||||
}
|
||||
}
|
||||
|
||||
for filename, rule_data in rules.items():
|
||||
with open(self.rules_dir / filename, 'w', encoding='utf-8') as f:
|
||||
json.dump(rule_data, f, indent=2)
|
||||
|
||||
def _create_default_prompt(self):
|
||||
"""Create default prompt"""
|
||||
default_prompt = """You are a metadata extraction assistant for markdown documents.
|
||||
|
||||
Your task is to analyze the document content and suggest appropriate frontmatter metadata.
|
||||
|
||||
Guidelines:
|
||||
- For tags: prefer selecting from the provided existing tags list when appropriate
|
||||
- You may create new tags if they better fit the content
|
||||
- New tags must follow the rules: lowercase letters, numbers, and underscores only
|
||||
- Be concise and accurate
|
||||
- Preserve important existing metadata when present"""
|
||||
|
||||
with open(self.prompts_dir / 'default.txt', 'w', encoding='utf-8') as f:
|
||||
f.write(default_prompt)
|
||||
|
||||
def get_api_config(self) -> dict:
|
||||
"""Get API configuration with environment variable resolution"""
|
||||
config = {
|
||||
'url': 'https://api.openai.com/v1',
|
||||
'model': 'gpt-4',
|
||||
'api_key': '',
|
||||
'api_key_source': 'none'
|
||||
}
|
||||
|
||||
if not self.env_file.exists():
|
||||
return config
|
||||
|
||||
# Parse .env file
|
||||
env_vars = {}
|
||||
with open(self.env_file, 'r', encoding='utf-8') as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line or line.startswith('#'):
|
||||
continue
|
||||
if '=' in line:
|
||||
key, value = line.split('=', 1)
|
||||
env_vars[key.strip()] = value.strip()
|
||||
|
||||
# Get URL
|
||||
if 'OPENAI_API_URL' in env_vars:
|
||||
config['url'] = env_vars['OPENAI_API_URL']
|
||||
|
||||
# Get model
|
||||
if 'OPENAI_MODEL' in env_vars:
|
||||
config['model'] = env_vars['OPENAI_MODEL']
|
||||
|
||||
# Get API key with special handling
|
||||
if 'OPENAI_API_KEY' in env_vars:
|
||||
api_key_value = env_vars['OPENAI_API_KEY']
|
||||
|
||||
# Remove quotes if present
|
||||
api_key_value = api_key_value.strip('"').strip("'")
|
||||
|
||||
if not api_key_value:
|
||||
# Empty string = no key needed
|
||||
config['api_key'] = ''
|
||||
config['api_key_source'] = 'none'
|
||||
elif api_key_value.startswith('$'):
|
||||
# Environment variable reference
|
||||
env_var_name = api_key_value[1:]
|
||||
env_value = os.environ.get(env_var_name)
|
||||
if env_value is None:
|
||||
raise ValueError(
|
||||
f"Environment variable '{env_var_name}' not found. "
|
||||
f"Referenced in .env as OPENAI_API_KEY={api_key_value}"
|
||||
)
|
||||
config['api_key'] = env_value
|
||||
config['api_key_source'] = f'env:{env_var_name}'
|
||||
else:
|
||||
# Plain text key (security warning will be issued)
|
||||
config['api_key'] = api_key_value
|
||||
config['api_key_source'] = 'plaintext'
|
||||
|
||||
return config
|
||||
|
||||
def print_config_info(self):
|
||||
"""Print configuration location and status"""
|
||||
print(f"Configuration directory: {self.config_dir}")
|
||||
print(f" .env file: {self.env_file}")
|
||||
print(f" Templates: {self.templates_dir}")
|
||||
print(f" Rules: {self.rules_dir}")
|
||||
print(f" Prompts: {self.prompts_dir}")
|
||||
|
||||
# Get API config to show status
|
||||
try:
|
||||
api_config = self.get_api_config()
|
||||
|
||||
if api_config['api_key_source'] == 'none':
|
||||
print(f" API Key: Not configured (LLM features disabled)")
|
||||
elif api_config['api_key_source'] == 'plaintext':
|
||||
print(f" API Key: Configured (plain text)")
|
||||
print(f" WARNING: Storing API keys in plain text is a security risk!")
|
||||
print(f" Consider using environment variable: OPENAI_API_KEY=$YOUR_ENV_VAR")
|
||||
elif api_config['api_key_source'].startswith('env:'):
|
||||
env_var = api_config['api_key_source'].split(':', 1)[1]
|
||||
print(f" API Key: From environment variable ${env_var}")
|
||||
except ValueError as e:
|
||||
print(f" API Key: ERROR - {e}")
|
||||
|
||||
print()
|
||||
+4
-26
@@ -5,17 +5,14 @@ import os
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from dotenv import load_dotenv
|
||||
from openai import OpenAI
|
||||
|
||||
|
||||
class LLMClient:
|
||||
def __init__(self, prompts_dir: Path):
|
||||
load_dotenv()
|
||||
|
||||
self.api_url = os.getenv('OPENAI_API_URL', 'https://api.openai.com/v1')
|
||||
self.model = os.getenv('OPENAI_MODEL', 'gpt-4')
|
||||
self.api_key = os.getenv('OPENAI_API_KEY', '')
|
||||
def __init__(self, prompts_dir: Path, api_config: dict):
|
||||
self.api_url = api_config.get('url', 'https://api.openai.com/v1')
|
||||
self.model = api_config.get('model', 'gpt-4')
|
||||
self.api_key = api_config.get('api_key', '')
|
||||
|
||||
self.client = OpenAI(
|
||||
base_url=self.api_url,
|
||||
@@ -23,25 +20,6 @@ class LLMClient:
|
||||
) if self.api_key else None
|
||||
|
||||
self.prompts_dir = prompts_dir
|
||||
if not prompts_dir.exists():
|
||||
prompts_dir.mkdir(parents=True)
|
||||
self._create_default_prompt()
|
||||
|
||||
def _create_default_prompt(self):
|
||||
"""Create default system prompt"""
|
||||
default_prompt = """You are a metadata extraction assistant for markdown documents.
|
||||
|
||||
Your task is to analyze the document content and suggest appropriate frontmatter metadata.
|
||||
|
||||
Guidelines:
|
||||
- For tags: prefer selecting from the provided existing tags list when appropriate
|
||||
- You may create new tags if they better fit the content
|
||||
- New tags must follow the rules: lowercase letters, numbers, and underscores only
|
||||
- Be concise and accurate
|
||||
- Preserve important existing metadata when present"""
|
||||
|
||||
with open(self.prompts_dir / 'default.txt', 'w', encoding='utf-8') as f:
|
||||
f.write(default_prompt)
|
||||
|
||||
def load_prompt(self, name: str = 'default') -> str:
|
||||
"""Load prompt template"""
|
||||
|
||||
+18
-5
@@ -1,6 +1,7 @@
|
||||
"""
|
||||
Main frontmatter processor
|
||||
"""
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from core.database import Database
|
||||
@@ -10,6 +11,7 @@ from core.rules import RulesEngine
|
||||
from core.template import TemplateManager
|
||||
from core.llm import LLMClient
|
||||
from core.changelog import ChangeLog
|
||||
from core.config import ConfigManager
|
||||
|
||||
|
||||
class FrontmatterProcessor:
|
||||
@@ -22,15 +24,26 @@ class FrontmatterProcessor:
|
||||
self.force = force
|
||||
self.add_only = add_only
|
||||
|
||||
# Initialize configuration
|
||||
self.config = ConfigManager()
|
||||
self.config.ensure_config_exists()
|
||||
|
||||
# Get API configuration
|
||||
api_config = self.config.get_api_config()
|
||||
|
||||
self.db = Database(repo.path / 'madomeda.db')
|
||||
self.rules = RulesEngine(repo.path / 'rules')
|
||||
self.templates = TemplateManager(repo.path / 'templates')
|
||||
self.llm = LLMClient(repo.path / 'prompts')
|
||||
self.rules = RulesEngine(self.config.rules_dir)
|
||||
self.templates = TemplateManager(self.config.templates_dir)
|
||||
self.llm = LLMClient(self.config.prompts_dir, api_config)
|
||||
self.changelog = ChangeLog(repo.path)
|
||||
self.parser = FrontmatterParser()
|
||||
|
||||
def process(self):
|
||||
"""Main processing loop"""
|
||||
# Print configuration info if running in interactive shell
|
||||
if sys.stdin.isatty():
|
||||
self.config.print_config_info()
|
||||
|
||||
print(f"Processing repository: {self.repo.path}")
|
||||
print(f"Template: {self.template_name}")
|
||||
print(f"Mode: {'DRY RUN' if self.whatif else 'LIVE'}")
|
||||
@@ -235,8 +248,8 @@ class FrontmatterProcessor:
|
||||
|
||||
elif field == 'tags':
|
||||
tags = normalized_fm.get('tags', [])
|
||||
# Always return the list (even if empty) so it's preserved
|
||||
return tags
|
||||
# Return None if empty so AI inference is tried
|
||||
return tags if tags else None
|
||||
|
||||
return None
|
||||
|
||||
|
||||
+1
-43
@@ -14,56 +14,14 @@ class RulesEngine:
|
||||
|
||||
def _load_rules(self) -> List[dict]:
|
||||
"""Load all rule files"""
|
||||
if not self.rules_dir.exists():
|
||||
self.rules_dir.mkdir(parents=True)
|
||||
self._create_default_rules()
|
||||
|
||||
rules = []
|
||||
if self.rules_dir.exists():
|
||||
for rule_file in self.rules_dir.glob('*.json'):
|
||||
with open(rule_file, 'r', encoding='utf-8') as f:
|
||||
rules.append(json.load(f))
|
||||
|
||||
return rules
|
||||
|
||||
def _create_default_rules(self):
|
||||
"""Create default rule files"""
|
||||
# Tags normalization rule
|
||||
tags_rule = {
|
||||
"name": "tags_normalization",
|
||||
"description": "Normalize tags to lowercase with underscores",
|
||||
"type": "tags"
|
||||
}
|
||||
|
||||
tag_to_tags_rule = {
|
||||
"name": "tag_to_tags",
|
||||
"description": "Rename 'tag' key to 'tags'",
|
||||
"type": "rename"
|
||||
}
|
||||
|
||||
lowercase_keys_rule = {
|
||||
"name": "lowercase_keys",
|
||||
"description": "All frontmatter keys must be lowercase",
|
||||
"type": "keys"
|
||||
}
|
||||
|
||||
summary_to_description_rule = {
|
||||
"name": "summary_to_description",
|
||||
"description": "Rename 'summary' key to 'description'",
|
||||
"type": "rename"
|
||||
}
|
||||
|
||||
with open(self.rules_dir / 'tags_normalization.json', 'w', encoding='utf-8') as f:
|
||||
json.dump(tags_rule, f, indent=2)
|
||||
|
||||
with open(self.rules_dir / 'tag_to_tags.json', 'w', encoding='utf-8') as f:
|
||||
json.dump(tag_to_tags_rule, f, indent=2)
|
||||
|
||||
with open(self.rules_dir / 'lowercase_keys.json', 'w', encoding='utf-8') as f:
|
||||
json.dump(lowercase_keys_rule, f, indent=2)
|
||||
|
||||
with open(self.rules_dir / 'summary_to_description.json', 'w', encoding='utf-8') as f:
|
||||
json.dump(summary_to_description_rule, f, indent=2)
|
||||
|
||||
def apply_rules(self, frontmatter: dict) -> Tuple[dict, bool]:
|
||||
"""Apply all rules to frontmatter, return (modified_fm, is_conformant)"""
|
||||
if not frontmatter:
|
||||
|
||||
@@ -10,24 +10,6 @@ class TemplateManager:
|
||||
def __init__(self, templates_dir: Path):
|
||||
self.templates_dir = templates_dir
|
||||
|
||||
if not templates_dir.exists():
|
||||
templates_dir.mkdir(parents=True)
|
||||
self._create_default_template()
|
||||
|
||||
def _create_default_template(self):
|
||||
"""Create default template"""
|
||||
default_template = {
|
||||
"title": "heur|orig",
|
||||
"created": "heur|orig",
|
||||
"changed": "heur|orig",
|
||||
"authors": "heur|orig",
|
||||
"version": "heur|orig",
|
||||
"tags": "orig|ai default"
|
||||
}
|
||||
|
||||
with open(self.templates_dir / 'default.json', 'w', encoding='utf-8') as f:
|
||||
json.dump(default_template, f, indent=2)
|
||||
|
||||
def load_template(self, name: str) -> dict:
|
||||
"""Load template by name"""
|
||||
template_path = self.templates_dir / f'{name}.json'
|
||||
@@ -35,8 +17,6 @@ class TemplateManager:
|
||||
if not template_path.exists():
|
||||
# Fallback to default
|
||||
template_path = self.templates_dir / 'default.json'
|
||||
if not template_path.exists():
|
||||
self._create_default_template()
|
||||
|
||||
with open(template_path, 'r', encoding='utf-8') as f:
|
||||
return json.load(f)
|
||||
|
||||
@@ -35,3 +35,17 @@
|
||||
2025-10-23 18:32:49 (commit: 0463d73ef927403e235bfd18ad9d202ae57ec618)
|
||||
notag.md - frontmatter updated
|
||||
|
||||
2025-10-23 19:23:00 (commit: ab4b95c664427f6b08bd7510fd8c925403887ea5)
|
||||
ADD_ONLY_EXAMPLES.md - frontmatter updated
|
||||
GETTING_STARTED.md - frontmatter updated
|
||||
IMPLEMENTATION.md - frontmatter updated
|
||||
PROJECT_SUMMARY.md - frontmatter updated
|
||||
README.md - frontmatter updated
|
||||
STRUCTURE.md - frontmatter updated
|
||||
USAGE.md - frontmatter updated
|
||||
VERSION_FIELD.md - frontmatter updated
|
||||
ai_test.md - frontmatter updated
|
||||
|
||||
2025-10-23 19:23:44 (commit: ab4b95c664427f6b08bd7510fd8c925403887ea5)
|
||||
ai_test.md - frontmatter updated
|
||||
|
||||
|
||||
@@ -1,3 +1,2 @@
|
||||
PyYAML>=6.0
|
||||
python-dotenv>=1.0.0
|
||||
openai>=1.0.0
|
||||
|
||||
Reference in New Issue
Block a user