Remove test file
This commit is contained in:
Test User
@@ -0,0 +1,260 @@
|
||||
# Madomeda - Markdown Document Metadata Manager
|
||||
|
||||
A Python tool for managing and normalizing frontmatter in markdown files within git repositories.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Run on current directory (dry run)
|
||||
python madomeda.py --whatif
|
||||
|
||||
# Process files
|
||||
python madomeda.py
|
||||
|
||||
# View database
|
||||
python inspect_db.py
|
||||
```
|
||||
|
||||
## What It Does
|
||||
|
||||
Madomeda automatically:
|
||||
- ✅ Discovers markdown files in git repositories
|
||||
- ✅ Parses and normalizes YAML frontmatter
|
||||
- ✅ Applies rule-based validation (lowercase keys, normalized tags)
|
||||
- ✅ Extracts git metadata (authors, dates, versions)
|
||||
- ✅ Generates missing frontmatter from templates
|
||||
- ✅ Tracks all changes in SQLite database
|
||||
- ✅ Logs modifications to changelog file
|
||||
- ✅ (Optional) Uses LLM for intelligent metadata inference
|
||||
|
||||
## Features
|
||||
|
||||
### Git Integration
|
||||
- Tracks files via `git ls-files`
|
||||
- Extracts commit history, authors, and dates
|
||||
- Respects `.gitignore` patterns
|
||||
- Ignores hidden files (starting with `.`)
|
||||
|
||||
### Rule-Based Normalization
|
||||
- **Tags**: Converts to `lowercase_with_underscores`
|
||||
- **Keys**: All frontmatter keys to lowercase
|
||||
- **tag→tags**: Renames singular to plural
|
||||
- **summary→description**: Renames summary field to description
|
||||
- Extensible via JSON rule files
|
||||
|
||||
### Template System
|
||||
Multi-strategy field resolution:
|
||||
- `heur`: Heuristics (git history, document content)
|
||||
- `orig`: Original frontmatter value
|
||||
- `ai <prompt>`: LLM inference with structured output
|
||||
|
||||
### Database Tracking
|
||||
SQLite database stores:
|
||||
- File paths and discovery dates
|
||||
- Frontmatter versions and conformance
|
||||
- Git commit information
|
||||
- All unique tags across repository
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### Optional: LLM Configuration
|
||||
|
||||
Edit `.env`:
|
||||
```
|
||||
OPENAI_API_URL=https://api.openai.com/v1
|
||||
OPENAI_MODEL=gpt-4
|
||||
OPENAI_API_KEY=your-api-key-here
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Process current directory
|
||||
python madomeda.py
|
||||
|
||||
# Dry run (preview changes)
|
||||
python madomeda.py --whatif
|
||||
|
||||
# Specific directory
|
||||
python madomeda.py --dir /path/to/repo
|
||||
|
||||
# Force update all files
|
||||
python madomeda.py --force
|
||||
|
||||
# Skip confirmation prompts
|
||||
python madomeda.py --no-confirm
|
||||
|
||||
# Ignore specific paths
|
||||
python madomeda.py --ignore-paths docs/archive vendor
|
||||
|
||||
# Use custom template
|
||||
python madomeda.py --template my-template
|
||||
|
||||
# Add-only mode (preserve existing keys)
|
||||
python madomeda.py --add-only
|
||||
```
|
||||
|
||||
## Example Transformation
|
||||
|
||||
**Before:**
|
||||
```yaml
|
||||
---
|
||||
Title: My Document
|
||||
Tag: Python, Machine-Learning, Data Science
|
||||
---
|
||||
```
|
||||
|
||||
**After:**
|
||||
```yaml
|
||||
---
|
||||
title: My Document
|
||||
created: '2025-10-23T17:27:16+02:00'
|
||||
changed: '2025-10-23T17:27:16+02:00'
|
||||
authors:
|
||||
- John Doe
|
||||
version: 34d2d05
|
||||
tags:
|
||||
- python
|
||||
- machine_learning
|
||||
- data_science
|
||||
---
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
- **[GETTING_STARTED.md](GETTING_STARTED.md)** - Step-by-step guide
|
||||
- **[USAGE.md](USAGE.md)** - Detailed usage examples
|
||||
- **[STRUCTURE.md](STRUCTURE.md)** - Project architecture
|
||||
- **[IMPLEMENTATION.md](IMPLEMENTATION.md)** - Technical details
|
||||
- **[PROJECT_SUMMARY.md](PROJECT_SUMMARY.md)** - Complete overview
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
madomeda/
|
||||
├── madomeda.py # Main entry point
|
||||
├── inspect_db.py # Database inspection tool
|
||||
├── requirements.txt # Dependencies
|
||||
├── .env # LLM configuration
|
||||
│
|
||||
├── core/ # Core modules
|
||||
│ ├── database.py # SQLite management
|
||||
│ ├── repository.py # Git operations
|
||||
│ ├── frontmatter.py # YAML parsing
|
||||
│ ├── rules.py # Rule engine
|
||||
│ ├── template.py # Template system
|
||||
│ ├── llm.py # LLM integration
|
||||
│ ├── changelog.py # Change logging
|
||||
│ └── processor.py # Main orchestration
|
||||
│
|
||||
├── templates/ # Frontmatter templates
|
||||
├── rules/ # Validation rules
|
||||
├── prompts/ # LLM prompts
|
||||
│
|
||||
├── madomeda.db # Database (gitignored)
|
||||
└── madomeda_changelog.txt # Change log
|
||||
```
|
||||
|
||||
## Templates
|
||||
|
||||
Create custom templates in `templates/`:
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "heur|orig",
|
||||
"created": "heur|orig",
|
||||
"changed": "heur|orig",
|
||||
"authors": "heur|orig",
|
||||
"version": "heur|orig",
|
||||
"tags": "heur|orig|ai default"
|
||||
}
|
||||
```
|
||||
|
||||
**Strategy priority**: Left to right until one succeeds.
|
||||
|
||||
## Rules
|
||||
|
||||
Default rules in `rules/`:
|
||||
- `tags_normalization.json` - Normalize tag format
|
||||
- `tag_to_tags.json` - Rename singular to plural
|
||||
- `lowercase_keys.json` - Lowercase all keys
|
||||
- `summary_to_description.json` - Rename summary to description
|
||||
|
||||
## Heuristic Values
|
||||
|
||||
| Field | Source |
|
||||
|-------|--------|
|
||||
| title | `# heading` or filename |
|
||||
| created | First commit date (ISO 8601) |
|
||||
| changed | Latest commit date (ISO 8601) |
|
||||
| authors | All contributors from git log |
|
||||
| version | Latest commit tag (if exists), else short hash (7 chars) |
|
||||
| tags | Normalized from original |
|
||||
|
||||
## Command-Line Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--dir PATH` | Repository directory (default: current) |
|
||||
| `--template NAME` | Template to use (default: 'default') |
|
||||
| `--ignore-paths PATH...` | Paths to exclude |
|
||||
| `--whatif` | Dry run mode (no changes) |
|
||||
| `--no-confirm` | Skip confirmation prompts |
|
||||
| `--force` | Recreate all frontmatter |
|
||||
| `--add-only` | Only add missing fields, preserve existing keys |
|
||||
|
||||
## Database Inspection
|
||||
|
||||
```bash
|
||||
python inspect_db.py
|
||||
```
|
||||
|
||||
Or query directly:
|
||||
```python
|
||||
import sqlite3
|
||||
conn = sqlite3.connect('madomeda.db')
|
||||
cursor = conn.cursor()
|
||||
cursor.execute('SELECT tag FROM tags ORDER BY tag')
|
||||
print([row[0] for row in cursor.fetchall()])
|
||||
```
|
||||
|
||||
## Changelog Format
|
||||
|
||||
```
|
||||
2025-10-23 17:40:36 (commit: 34d2d05)
|
||||
sample.md - frontmatter updated
|
||||
test.md - frontmatter updated
|
||||
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.7+
|
||||
- Git repository
|
||||
- Dependencies: PyYAML, python-dotenv, openai (optional)
|
||||
|
||||
## License
|
||||
|
||||
Created as a custom tool for markdown documentation management.
|
||||
|
||||
## Getting Help
|
||||
|
||||
1. Read the documentation files
|
||||
2. Run with `--whatif` to preview changes
|
||||
3. Use `inspect_db.py` to examine database
|
||||
4. Check `madomeda_changelog.txt` for history
|
||||
|
||||
## Contributing
|
||||
|
||||
This is a standalone tool. Customize by:
|
||||
- Creating new templates in `templates/`
|
||||
- Adding rules in `rules/`
|
||||
- Modifying prompts in `prompts/`
|
||||
- Extending code in `core/` modules
|
||||
Reference in New Issue
Block a user