madomeda/README.md

---
title: Madomeda - Markdown Document Metadata Manager
created: '2025-10-23T18:33:48+02:00'
status: draft
priority: 5
published: false
tags: []
reviewers: []
---
# Madomeda - Markdown Document Metadata Manager

A Python tool for managing and normalizing frontmatter in markdown files within git repositories.

## Quick Start

```bash
# Install dependencies
pip install -r requirements.txt

# Run on current directory (dry run)
python madomeda.py --whatif

# Process files
python madomeda.py

# View database
python inspect_db.py
```

## What It Does

Madomeda automatically:
- ✅ Discovers markdown files in git repositories
- ✅ Parses and normalizes YAML frontmatter
- ✅ Applies rule-based validation (lowercase keys, normalized tags)
- ✅ Extracts git metadata (authors, dates, versions)
- ✅ Generates missing frontmatter from templates
- ✅ Tracks all changes in SQLite database
- ✅ Logs modifications to changelog file
- ✅ (Optional) Uses LLM for intelligent metadata inference

## Features

### Git Integration
- Tracks files via `git ls-files`
- Extracts commit history, authors, and dates
- Respects `.gitignore` patterns
- Ignores hidden files (starting with `.`)

### Rule-Based Normalization
- **Tags**: Converts to `lowercase_with_underscores`
- **Keys**: All frontmatter keys to lowercase
- **tag→tags**: Renames singular to plural
- **summary→description**: Renames summary field to description
- Extensible via JSON rule files

### Template System
Multi-strategy field resolution:
- `heur`: Heuristics (git history, document content)
- `orig`: Original frontmatter value
- `ai <prompt>`: LLM inference with structured output
- `lit:value`: Literal/static value (fallback)

### Database Tracking
SQLite database stores:
- File paths and discovery dates
- Frontmatter versions and conformance
- Git commit information
- All unique tags across repository

## Installation

```bash
pip install -r requirements.txt
```

The first time you run Madomeda, it will automatically create a configuration directory at `~/.config/madomeda` (all platforms) with default settings.

### Optional: LLM Configuration

Edit `~/.config/madomeda/.env`:

```bash
# Option 1: Use environment variable (recommended)
OPENAI_API_KEY=$OPENAI_API_KEY

# Option 2: No LLM (empty string)
OPENAI_API_KEY=""

# Option 3: Plain text (not recommended - security warning)
OPENAI_API_KEY=sk-your-key-here
```

For detailed configuration options, see [CONFIGURATION.md](CONFIGURATION.md).

## Usage

```bash
# Process current directory
python madomeda.py

# Dry run (preview changes)
python madomeda.py --whatif

# Specific directory
python madomeda.py --dir /path/to/repo

# Force update all files
python madomeda.py --force

# Skip confirmation prompts
python madomeda.py --no-confirm

# Ignore specific paths
python madomeda.py --ignore-paths docs/archive vendor

# Use custom template
python madomeda.py --template my-template

# Add-only mode (preserve existing keys)
python madomeda.py --add-only
```

## Example Transformation

**Before:**
```yaml
---
Title: My Document
Tag: Python, Machine-Learning, Data Science
---
```

**After:**
```yaml
---
title: My Document
created: '2025-10-23T17:27:16+02:00'
changed: '2025-10-23T17:27:16+02:00'
authors:
- John Doe
version: 34d2d05
tags:
- python
- machine_learning
- data_science
---
```

## Documentation

- **[GETTING_STARTED.md](GETTING_STARTED.md)** - Step-by-step guide
- **[USAGE.md](USAGE.md)** - Detailed usage examples
- **[CONFIGURATION.md](CONFIGURATION.md)** - Configuration management
- **[RULES_GUIDE.md](RULES_GUIDE.md)** - Writing custom rules
- **[TEMPLATE_TYPES.md](TEMPLATE_TYPES.md)** - Template field types
- **[LITERAL_VALUES.md](LITERAL_VALUES.md)** - Template literal values
- **[STRUCTURE.md](STRUCTURE.md)** - Project architecture
- **[IMPLEMENTATION.md](IMPLEMENTATION.md)** - Technical details
- **[PROJECT_SUMMARY.md](PROJECT_SUMMARY.md)** - Complete overview
- **[ADD_ONLY_EXAMPLES.md](ADD_ONLY_EXAMPLES.md)** - Add-only mode guide
- **[VERSION_FIELD.md](VERSION_FIELD.md)** - Version field behavior

## Project Structure

```
madomeda/
├── madomeda.py              # Main entry point
├── inspect_db.py            # Database inspection tool
├── requirements.txt         # Dependencies
│
├── core/                    # Core modules
│   ├── config.py            # Configuration management
│   ├── database.py          # SQLite management
│   ├── repository.py        # Git operations
│   ├── frontmatter.py       # YAML parsing
│   ├── rules.py             # Rule engine
│   ├── template.py          # Template system
│   ├── llm.py              # LLM integration
│   ├── changelog.py         # Change logging
│   └── processor.py         # Main orchestration
│
└── madomeda.db             # Database (in repo, gitignored)

~/.config/madomeda/          # User configuration (auto-created)
├── .env                     # API configuration
├── templates/               # Frontmatter templates
├── rules/                   # Validation rules
└── prompts/                 # LLM prompts
```

## Templates

Create custom templates in `~/.config/madomeda/templates/`:

```json
{
  "title": "heur|orig:text",
  "created": "heur|orig:datetime",
  "changed": "heur|orig:datetime",
  "authors": "heur|orig:list",
  "version": "heur|orig:text",
  "tags": "heur|orig|ai default:list",
  "status": "lit:draft:text",
  "published": "orig|lit:false:checkbox"
}
```

**Strategy priority**: Left to right until one succeeds.

**Field types**: `text`, `number`, `checkbox`, `date`, `datetime`, `list` (see [TEMPLATE_TYPES.md](TEMPLATE_TYPES.md))

## Rules

Default rules in `rules/`:
- `tags_normalization.json` - Normalize tag format
- `tag_to_tags.json` - Rename singular to plural
- `lowercase_keys.json` - Lowercase all keys
- `summary_to_description.json` - Rename summary to description

## Heuristic Values

| Field | Source |
|-------|--------|
| title | `# heading` or filename |
| created | First commit date (ISO 8601) |
| changed | Latest commit date (ISO 8601) |
| authors | All contributors from git log |
| version | Latest commit tag (if exists), else short hash (7 chars) |
| tags | Normalized from original |

## Command-Line Options

| Option | Description |
|--------|-------------|
| `--dir PATH` | Repository directory (default: current) |
| `--template NAME` | Template to use (default: 'default') |
| `--ignore-paths PATH...` | Paths to exclude |
| `--whatif` | Dry run mode (no changes) |
| `--no-confirm` | Skip confirmation prompts |
| `--force` | Recreate all frontmatter |
| `--add-only` | Only add missing fields, preserve existing keys |

## Database Inspection

```bash
python inspect_db.py
```

Or query directly:
```python
import sqlite3
conn = sqlite3.connect('madomeda.db')
cursor = conn.cursor()
cursor.execute('SELECT tag FROM tags ORDER BY tag')
print([row[0] for row in cursor.fetchall()])
```

## Changelog Format

```
2025-10-23 17:40:36 (commit: 34d2d05)
  sample.md - frontmatter updated
  test.md - frontmatter updated

```

## Requirements

- Python 3.7+
- Git repository
- Dependencies: PyYAML, python-dotenv, openai (optional)

## License

Created as a custom tool for markdown documentation management.

## Getting Help

1. Read the documentation files
2. Run with `--whatif` to preview changes
3. Use `inspect_db.py` to examine database
4. Check `madomeda_changelog.txt` for history

## Contributing

This is a standalone tool. Customize by:
- Creating new templates in `templates/`
- Adding rules in `rules/`
- Modifying prompts in `prompts/`
- Extending code in `core/` modules