improvements added to rules engine
This commit is contained in:
Test User
+562
@@ -0,0 +1,562 @@
|
||||
# Writing Custom Rules
|
||||
|
||||
## Overview
|
||||
|
||||
Madomeda uses a self-contained, JSON-based rules system. Rules are processed in priority order and can normalize, validate, and transform frontmatter fields without any code changes.
|
||||
|
||||
## Rule File Location
|
||||
|
||||
Rules are stored in: `~/.config/madomeda/rules/`
|
||||
|
||||
Each rule is a separate JSON file. The filename is used for sorting (alphabetically), but the `priority` field determines execution order.
|
||||
|
||||
## Rule Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "rule_name",
|
||||
"description": "Human-readable description",
|
||||
"field": "field_name",
|
||||
"priority": 10,
|
||||
"action": "action_type",
|
||||
"pattern": "regex_pattern",
|
||||
"replacement": "replacement_string",
|
||||
"llm_prompt": "Prompt for LLM if heuristics fail"
|
||||
}
|
||||
```
|
||||
|
||||
### Required Fields
|
||||
|
||||
- **name**: Unique identifier for the rule
|
||||
- **description**: What the rule does
|
||||
- **field**: Which frontmatter field this applies to (use `"*"` for all fields)
|
||||
- **priority**: Execution order (lower number = earlier execution, typically 1-100)
|
||||
- **action**: What the rule does (see Actions below)
|
||||
|
||||
### Optional Fields
|
||||
|
||||
- **pattern**: Regex pattern for matching/replacing
|
||||
- **replacement**: Replacement string (can use capture groups like `\g<0>`, `\1`, etc.)
|
||||
- **transform**: Transformation to apply (`lower`, `upper`)
|
||||
- **multiline**: Whether regex uses multiline mode (default: false)
|
||||
- **split_on**: Character to split on (e.g., `","` for comma-separated values)
|
||||
- **from**: Source field name (for rename actions)
|
||||
- **to**: Target field name (for rename actions)
|
||||
- **llm_prompt**: Text to send to LLM if this rule's value needs inference
|
||||
|
||||
## Actions
|
||||
|
||||
### 1. normalize_keys
|
||||
|
||||
Apply transformation to all frontmatter keys.
|
||||
|
||||
**Use case**: Ensure all keys are lowercase.
|
||||
|
||||
**Example:**
|
||||
```json
|
||||
{
|
||||
"name": "lowercase_keys",
|
||||
"description": "Convert all keys to lowercase",
|
||||
"field": "*",
|
||||
"priority": 1,
|
||||
"action": "normalize_keys",
|
||||
"transform": "lower",
|
||||
"llm_prompt": "Ensure all keys are lowercase"
|
||||
}
|
||||
```
|
||||
|
||||
**Alternative with regex:**
|
||||
```json
|
||||
{
|
||||
"name": "remove_spaces_from_keys",
|
||||
"description": "Remove spaces from keys",
|
||||
"field": "*",
|
||||
"priority": 2,
|
||||
"action": "normalize_keys",
|
||||
"pattern": " ",
|
||||
"replacement": "_"
|
||||
}
|
||||
```
|
||||
|
||||
### 2. rename_field
|
||||
|
||||
Rename one field to another.
|
||||
|
||||
**Use case**: Standardize field names (e.g., `tag` → `tags`, `summary` → `description`).
|
||||
|
||||
**Example:**
|
||||
```json
|
||||
{
|
||||
"name": "tag_to_tags",
|
||||
"description": "Rename 'tag' to 'tags'",
|
||||
"field": "tag",
|
||||
"priority": 2,
|
||||
"action": "rename_field",
|
||||
"from": "tag",
|
||||
"to": "tags",
|
||||
"llm_prompt": "Convert tag field to tags array"
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: Rename rules execute before other field-specific rules, so subsequent rules can process the renamed field.
|
||||
|
||||
### 3. normalize_value
|
||||
|
||||
Transform field values using regex, transforms, or splitting.
|
||||
|
||||
**Use case**: Standardize tag format, clean up values, convert formats.
|
||||
|
||||
**Example 1: Split comma-separated to list**
|
||||
```json
|
||||
{
|
||||
"name": "tags_split",
|
||||
"description": "Convert comma-separated tags to list",
|
||||
"field": "tags",
|
||||
"priority": 10,
|
||||
"action": "normalize_value",
|
||||
"split_on": ",",
|
||||
"pattern": ".*",
|
||||
"replacement": "\\g<0>"
|
||||
}
|
||||
```
|
||||
|
||||
**Example 2: Replace spaces with underscores**
|
||||
```json
|
||||
{
|
||||
"name": "tags_no_spaces",
|
||||
"description": "Replace spaces with underscores in tags",
|
||||
"field": "tags",
|
||||
"priority": 20,
|
||||
"action": "normalize_value",
|
||||
"pattern": " ",
|
||||
"replacement": "_"
|
||||
}
|
||||
```
|
||||
|
||||
**Example 3: Lowercase transformation**
|
||||
```json
|
||||
{
|
||||
"name": "tags_lowercase",
|
||||
"description": "Convert tags to lowercase",
|
||||
"field": "tags",
|
||||
"priority": 30,
|
||||
"action": "normalize_value",
|
||||
"transform": "lower"
|
||||
}
|
||||
```
|
||||
|
||||
**Example 4: Remove invalid characters**
|
||||
```json
|
||||
{
|
||||
"name": "tags_alphanumeric_only",
|
||||
"description": "Keep only alphanumeric and underscores",
|
||||
"field": "tags",
|
||||
"priority": 40,
|
||||
"action": "normalize_value",
|
||||
"pattern": "[^a-z0-9_]",
|
||||
"replacement": ""
|
||||
}
|
||||
```
|
||||
|
||||
### 4. validate
|
||||
|
||||
Check if field values match a pattern.
|
||||
|
||||
**Use case**: Ensure values conform to expected format.
|
||||
|
||||
**Example:**
|
||||
```json
|
||||
{
|
||||
"name": "tags_validate",
|
||||
"description": "Validate tag format",
|
||||
"field": "tags",
|
||||
"priority": 50,
|
||||
"action": "validate",
|
||||
"pattern": "^[a-z0-9_]+$",
|
||||
"multiline": false
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: Validation rules mark frontmatter as non-conformant if they fail, but don't modify values.
|
||||
|
||||
## Priority System
|
||||
|
||||
Rules execute in priority order (lowest number first):
|
||||
|
||||
- **1-9**: Global transformations (keys, field renames)
|
||||
- **10-19**: Format conversions (split lists, type changes)
|
||||
- **20-29**: Character replacements (spaces, hyphens)
|
||||
- **30-39**: Case transformations
|
||||
- **40-49**: Character removal (invalid chars)
|
||||
- **50-99**: Validation
|
||||
|
||||
**Recommended naming convention:**
|
||||
```
|
||||
01_lowercase_keys.json # Priority 1
|
||||
10_tags_format_list.json # Priority 10
|
||||
20_tags_replace_spaces.json # Priority 20
|
||||
50_tags_validate.json # Priority 50
|
||||
```
|
||||
|
||||
## Processing Flow
|
||||
|
||||
1. **Global rules** (`field: "*"`) execute first
|
||||
2. **Rename rules** execute second (creates new fields)
|
||||
3. **Other rules** execute in priority order per field
|
||||
|
||||
## Complete Example: Tag Normalization Chain
|
||||
|
||||
Here's how multiple rules work together to normalize tags:
|
||||
|
||||
**Input:**
|
||||
```yaml
|
||||
Tag: Machine-Learning, Deep Learning, AI/ML
|
||||
```
|
||||
|
||||
**Rules (in priority order):**
|
||||
|
||||
```json
|
||||
// 02_tag_to_tags.json
|
||||
{
|
||||
"name": "tag_to_tags",
|
||||
"field": "tag",
|
||||
"priority": 2,
|
||||
"action": "rename_field",
|
||||
"from": "tag",
|
||||
"to": "tags"
|
||||
}
|
||||
```
|
||||
|
||||
After this rule: `tags: "Machine-Learning, Deep Learning, AI/ML"`
|
||||
|
||||
```json
|
||||
// 10_tags_format_list.json
|
||||
{
|
||||
"name": "tags_split",
|
||||
"field": "tags",
|
||||
"priority": 10,
|
||||
"action": "normalize_value",
|
||||
"split_on": ","
|
||||
}
|
||||
```
|
||||
|
||||
After this rule: `tags: ["Machine-Learning", "Deep Learning", "AI/ML"]`
|
||||
|
||||
```json
|
||||
// 20_tags_replace_spaces.json
|
||||
{
|
||||
"name": "tags_spaces",
|
||||
"field": "tags",
|
||||
"priority": 20,
|
||||
"action": "normalize_value",
|
||||
"pattern": " ",
|
||||
"replacement": "_"
|
||||
}
|
||||
```
|
||||
|
||||
After this rule: `tags: ["Machine-Learning", "Deep_Learning", "AI/ML"]`
|
||||
|
||||
```json
|
||||
// 21_tags_replace_hyphens.json
|
||||
{
|
||||
"name": "tags_hyphens",
|
||||
"field": "tags",
|
||||
"priority": 21,
|
||||
"action": "normalize_value",
|
||||
"pattern": "-",
|
||||
"replacement": "_"
|
||||
}
|
||||
```
|
||||
|
||||
After this rule: `tags: ["Machine_Learning", "Deep_Learning", "AI/ML"]`
|
||||
|
||||
```json
|
||||
// 30_tags_lowercase.json
|
||||
{
|
||||
"name": "tags_lowercase",
|
||||
"field": "tags",
|
||||
"priority": 30,
|
||||
"action": "normalize_value",
|
||||
"transform": "lower"
|
||||
}
|
||||
```
|
||||
|
||||
After this rule: `tags: ["machine_learning", "deep_learning", "ai/ml"]`
|
||||
|
||||
```json
|
||||
// 40_tags_remove_invalid.json
|
||||
{
|
||||
"name": "tags_clean",
|
||||
"field": "tags",
|
||||
"priority": 40,
|
||||
"action": "normalize_value",
|
||||
"pattern": "[^a-z0-9_]",
|
||||
"replacement": ""
|
||||
}
|
||||
```
|
||||
|
||||
**Final result:** `tags: ["machine_learning", "deep_learning", "aiml"]`
|
||||
|
||||
## Field Types
|
||||
|
||||
Rules apply to different value types:
|
||||
|
||||
### String Fields
|
||||
|
||||
Rules apply to the whole string:
|
||||
|
||||
```json
|
||||
{
|
||||
"field": "title",
|
||||
"action": "normalize_value",
|
||||
"transform": "lower"
|
||||
}
|
||||
```
|
||||
|
||||
### List Fields
|
||||
|
||||
Rules apply to each item in the list:
|
||||
|
||||
```json
|
||||
{
|
||||
"field": "keywords",
|
||||
"action": "normalize_value",
|
||||
"pattern": " ",
|
||||
"replacement": "_"
|
||||
}
|
||||
```
|
||||
|
||||
Each keyword gets spaces replaced with underscores.
|
||||
|
||||
### Converting String to List
|
||||
|
||||
Use `split_on`:
|
||||
|
||||
```json
|
||||
{
|
||||
"field": "authors",
|
||||
"action": "normalize_value",
|
||||
"split_on": ";",
|
||||
"pattern": ".*",
|
||||
"replacement": "\\g<0>"
|
||||
}
|
||||
```
|
||||
|
||||
Input: `"John Doe; Jane Smith"`
|
||||
Output: `["John Doe", "Jane Smith"]`
|
||||
|
||||
## Regular Expression Tips
|
||||
|
||||
### Capture Groups
|
||||
|
||||
```json
|
||||
{
|
||||
"pattern": "(\\d+)",
|
||||
"replacement": "v\\1"
|
||||
}
|
||||
```
|
||||
|
||||
Input: `"123"`
|
||||
Output: `"v123"`
|
||||
|
||||
### Case-Insensitive Matching
|
||||
|
||||
Use inline flag:
|
||||
|
||||
```json
|
||||
{
|
||||
"pattern": "(?i)todo",
|
||||
"replacement": "TODO"
|
||||
}
|
||||
```
|
||||
|
||||
### Match Whole String
|
||||
|
||||
```json
|
||||
{
|
||||
"pattern": "^[a-z]+$"
|
||||
}
|
||||
```
|
||||
|
||||
### Remove Leading/Trailing Whitespace
|
||||
|
||||
```json
|
||||
{
|
||||
"pattern": "^\\s+|\\s+$",
|
||||
"replacement": ""
|
||||
}
|
||||
```
|
||||
|
||||
## LLM Integration
|
||||
|
||||
The `llm_prompt` field provides guidance when the LLM needs to generate or fix values:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "tags_validate",
|
||||
"field": "tags",
|
||||
"action": "validate",
|
||||
"pattern": "^[a-z0-9_]+$",
|
||||
"llm_prompt": "Generate tags using only lowercase letters, numbers, and underscores. Prefer selecting from existing repository tags when appropriate."
|
||||
}
|
||||
```
|
||||
|
||||
When using `ai` strategy in templates, the LLM receives:
|
||||
- The rule's `llm_prompt`
|
||||
- List of existing tags in repository
|
||||
- Document content
|
||||
- Structured output schema
|
||||
|
||||
## Testing Rules
|
||||
|
||||
### Test with --whatif
|
||||
|
||||
```bash
|
||||
madomeda --whatif
|
||||
```
|
||||
|
||||
Shows what would change without modifying files.
|
||||
|
||||
### Test Specific Rule
|
||||
|
||||
Create a Python script:
|
||||
|
||||
```python
|
||||
from core.rules_processor import RulesProcessor
|
||||
from pathlib import Path
|
||||
|
||||
rp = RulesProcessor(Path.home() / '.config' / 'madomeda' / 'rules')
|
||||
|
||||
test_data = {
|
||||
'Title': 'My Document',
|
||||
'Tag': 'Python, Machine-Learning'
|
||||
}
|
||||
|
||||
result, conformant, violations = rp.apply_rules(test_data)
|
||||
print(f"Result: {result}")
|
||||
print(f"Conformant: {conformant}")
|
||||
print(f"Violations: {violations}")
|
||||
```
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Email Validation
|
||||
|
||||
```json
|
||||
{
|
||||
"field": "author_email",
|
||||
"action": "validate",
|
||||
"pattern": "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
|
||||
}
|
||||
```
|
||||
|
||||
### Date Format Validation (YYYY-MM-DD)
|
||||
|
||||
```json
|
||||
{
|
||||
"field": "date",
|
||||
"action": "validate",
|
||||
"pattern": "^\\d{4}-\\d{2}-\\d{2}$"
|
||||
}
|
||||
```
|
||||
|
||||
### URL Scheme Enforcement
|
||||
|
||||
```json
|
||||
{
|
||||
"field": "url",
|
||||
"action": "normalize_value",
|
||||
"pattern": "^(?!https?://)(.*)",
|
||||
"replacement": "https://\\1"
|
||||
}
|
||||
```
|
||||
|
||||
### Remove HTML Tags
|
||||
|
||||
```json
|
||||
{
|
||||
"field": "description",
|
||||
"action": "normalize_value",
|
||||
"pattern": "<[^>]+>",
|
||||
"replacement": ""
|
||||
}
|
||||
```
|
||||
|
||||
## Advanced: Custom Field Rules
|
||||
|
||||
### Ensure Boolean String Format
|
||||
|
||||
```json
|
||||
{
|
||||
"field": "published",
|
||||
"action": "normalize_value",
|
||||
"pattern": "^(true|false|yes|no|1|0)$",
|
||||
"replacement": "\\1",
|
||||
"transform": "lower"
|
||||
}
|
||||
```
|
||||
|
||||
Then normalize:
|
||||
|
||||
```json
|
||||
{
|
||||
"field": "published",
|
||||
"priority": 31,
|
||||
"action": "normalize_value",
|
||||
"pattern": "yes|1|true",
|
||||
"replacement": "true"
|
||||
}
|
||||
```
|
||||
|
||||
### Slug Generation
|
||||
|
||||
```json
|
||||
{
|
||||
"field": "slug",
|
||||
"action": "normalize_value",
|
||||
"pattern": "[^a-z0-9-]",
|
||||
"replacement": "",
|
||||
"transform": "lower"
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Rule Not Executing
|
||||
|
||||
- Check `field` matches exact field name
|
||||
- Verify `priority` is in expected range
|
||||
- Ensure JSON is valid (use `jsonlint`)
|
||||
|
||||
### Wrong Execution Order
|
||||
|
||||
- Lower priority number executes first
|
||||
- Rename rules always execute before other rules for same field
|
||||
- Global rules (`field: "*"`) execute before field-specific
|
||||
|
||||
### Regex Not Matching
|
||||
|
||||
- Test regex at https://regex101.com/
|
||||
- Escape special characters: `. * + ? ^ $ { } ( ) | [ ] \`
|
||||
- Use `multiline: true` for multiline patterns
|
||||
|
||||
### Value Not Changing
|
||||
|
||||
- Check if earlier rule already modified it
|
||||
- Verify pattern actually matches the value
|
||||
- Ensure `replacement` is specified for `normalize_value`
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use priority ranges** - Leave gaps (10, 20, 30) to insert rules later
|
||||
2. **Name files by priority** - `01_`, `10_`, `20_` for easy sorting
|
||||
3. **Test incrementally** - Add one rule at a time
|
||||
4. **Document complex regex** - Use `description` field
|
||||
5. **Provide LLM prompts** - Help AI understand the rule intent
|
||||
6. **Validate after normalize** - Use separate validate rule at higher priority
|
||||
|
||||
## See Also
|
||||
|
||||
- [CONFIGURATION.md](CONFIGURATION.md) - Configuration directory
|
||||
- [STRUCTURE.md](STRUCTURE.md) - How rules fit into the system
|
||||
- [USAGE.md](USAGE.md) - Using rules with madomeda
|
||||
Reference in New Issue
Block a user