Ansible-Test: IEC 62443-3-3 SL2 Compliance Validation

Ansible playbooks reimagined as a test framework for industrial control system (ICS/OT) security compliance. Every task is a test that collects evidence, never aborts on failure, produces structured JSON, and renders into human-readable reports via Go templates or Python.

---

Table of Contents

1. Architecture
2. Ansible Control Node — QEMU-Bootable
3. Quick Start
4. How It Works
5. The Test Pattern (The "Secret Sauce")
6. JSON Output Schema
7. IEC 62443-3-3 SL2 Coverage
8. Adding a New Test
9. Rendering Reports
10. File Reference
11. Design Decisions & Tradeoffs
12. Comparison to Alternatives
13. Roadmap

---

Architecture

playbooks/
├── site.yml                      # Entry point: orchestrates all FR suites
├── suites/
│   ├── fr1_auth.yml              # FR1: Identification & Authentication Control
│   ├── fr2_use_control.yml       # FR2: Use Control (audit, sudo, sessions)
│   └── fr5_data_flow.yml         # FR5: Restricted Data Flow (firewall, services)
├── library/
│   └── report.yml                # Aggregates test_results[] → JSON → disk
reports/
├── render.go                     # Go renderer (text/template on external .gohtml)
├── render_report.py              # Python renderer (terminal + markdown modes)
├── report.gohtml                 # Go template file for terminal-box-drawing report
├── go.mod                        # Go module definition
└── sample-output.json            # Example output for offline rendering tests
inventory.ini                     # Ansible inventory (target host list)
run.sh                            # End-to-end wrapper: ansible → find json → render
README.md                         # This file

Data Flow

  ansible-playbook site.yml
         │
         ├── FR1 suite  ──┐
         ├── FR2 suite  ──┤  Each task appends to test_results[]
         ├── FR5 suite  ──┘  via set_fact, wrapped in ignore_errors
         │
         ▼
  library/report.yml
    - Assembles __report dict with summary, by_category, failures
    - Prints summary to console
    - Writes JSON to reports/<hostname>-<date>.json
         │
         ▼
  run.sh → python3 reports/render_report.py reports/*.json
       → go run reports/render.go reports/*.json reports/report.gohtml

---

Ansible Control Node — QEMU-Bootable (Alpine Linux)

A minimal, portable Ansible control node that boots on QEMU pc-q35-10.0. Designed to run on Windows hosts, providing Ansible connectivity to Windows, Cisco, VMware, MSSQL, and Linux targets. No systemd — OpenRC + BusyBox.

Build & Launch

# Prerequisites: Docker, QEMU (qemu-full), passwordless sudo for mount
./scripts/build-qemu.sh       # Dockerfile → qcow2 disk (2GB)
./scripts/run-qemu.sh         # Boot VM (KVM, 1GB RAM, 2 vCPUs)

# Access the VM (password: ansible)
sshpass -p ansible ssh -p 2222 ansible@localhost

Architecture

Dockerfile                    # Complete rootfs definition
    │
    ▼
scripts/build-qemu.sh         # Docker export → ext4 → qcow2
    │
    ├── output/ansible-node.qcow2   (bootable disk, ext4)
    ├── output/vmlinuz-virt         (kernel, direct -kernel boot)
    └── output/initramfs-virt       (initramfs)
    │
    ▼
scripts/run-qemu.sh           # QEMU pc-q35-10.0, KVM, virtio
    │
    ▼
  ┌─────────────────────────────────────────────┐
  │  Alpine Linux 3.20 | Kernel 6.6 (virt)      │
  │  OpenRC | BusyBox | no systemd              │
  │  Console: ttyS0 xterm-256color              │
  │  Keyboard: PS/2 atkbd                       │
  │  Docker 26.1 daemon (running)               │
  │  SSH:  ansible@<ip> (password: ansible)     │
  └─────────────────────────────────────────────┘

Target Integrations

Target	Ansible Collection	Protocol	Python
Windows	ansible.windows 2.3	WinRM + Kerberos	pywinrm 0.5
Cisco ASA	cisco.asa 4.0	SSH/CLI	paramiko 4.0
Cisco IOS/Catalyst	cisco.ios 5.3	SSH/CLI	netmiko 4.7
Cisco NX-OS	cisco.nxos 5.3	SSH/NX-API	ncclient 0.7
VMware vSphere	community.vmware 4.3	SOAP API	pyvmomi 9.1
MSSQL	microsoft.ad 1.5	TDS	pymssql 2.3
Linux	built-in SSH	SSH	—

Key Design Decisions

Decision	Rationale
Dockerfile as build system	Dependency resolution, layer caching. Same file usable as docker run or QEMU boot
root=/dev/vda (not LABEL)	Alpine's nlplug-findfs can't resolve labels on virtio-block; device path works
modules=virtio_blk,ext4 on cmdline	CONFIG_VIRTIO_BLK=m — module not auto-detected by initramfs
No apk del purge step	Purging util-linux-dev cascades to removing mount, umount, blkid
Direct -kernel boot (no extlinux)	QEMU loads kernel/initrd directly; no bootloader needed in the disk image

---

Quick Start

Prerequisites

• Ansible ≥ 2.9 (core modules only: shell, stat, copy, set_fact, debug)
• Python ≥ 3.6 (for the Python report renderer)
• Go ≥ 1.21 (optional, for the Go template renderer)
• Target host: Linux with systemd (Debian/Ubuntu or RHEL/Rocky)

Run Against Localhost

# Edit inventory.ini to add your target, or test locally:
echo "localhost ansible_connection=local" > inventory.ini

# Run with sudo (many checks need root):
ansible-playbook -i inventory.ini playbooks/site.yml --limit localhost -K

# Or use the wrapper script:
KEEP_SUDO=1 ./run.sh --limit localhost -K

Render a Report

# Terminal format (default):
python3 reports/render_report.py reports/localhost-2026-07-07.json

# Markdown format (for GitHub/GitLab wikis):
python3 reports/render_report.py reports/localhost-2026-07-07.json --format md > REPORT.md

# Go template (if Go is installed):
(cd reports && go run render.go ../sample-output.json report.gohtml)

---

How It Works

The Test Pattern ("The Secret Sauce")

Every test follows a rigid 3-step Gather → Evaluate → Record structure wrapped in an Ansible block with ignore_errors: yes:

# ── SR 1.5: Password minimum length ───────────────────────────

- block:                                    # ← 1) Wrap everything
    - name: "Gather: Check pwquality minlen" # ← 2) Gather: read system state
      ansible.builtin.shell: |
        grep -E '^\s*minlen\s*=' /etc/security/pwquality.conf 2>/dev/null | tail -1 || echo "NOT SET"
      register: _minlen                     #    store raw output
      changed_when: false                   #    never report as "changed"

    - name: "Evaluate: IAC-05"              # ← 3) Evaluate: judge pass/fail
      ansible.builtin.set_fact:
        test_results: "{{ test_results + [{  #  append to shared list
          'test_id': 'IAC-05',
          'category': 'FR1 - Identification and Authentication Control',
          'requirement': 'SR 1.5 — Authenticator Strength',
          'description': 'Password minimum length shall be ≥ 14 characters',
          'passed': (
            _minlen.stdout | regex_search('minlen\\s*=\\s*([0-9]+)')
            | regex_replace('minlen\\s*=\\s*', '') | int >= 14
          ),
          'expected': 'minlen >= 14 in /etc/security/pwquality.conf',
          'actual': _minlen.stdout | trim,
          'severity': 'high',
          'remediation': 'Set minlen=14 in /etc/security/pwquality.conf'
        }] }}"
  ignore_errors: yes                        # ← 4) NEVER abort the run

Why this works: ignore_errors: yes prevents Ansible from stopping. The set_fact always executes (it never fails). The block ensures the gather and evaluate steps share a scope, so registered variables are available.

Why block + ignore_errors Instead of failed_when

Using failed_when: false on individual tasks is tempting, but:

Approach	Behavior
failed_when: false on shell task	Shell always "succeeds"; but if the shell returns non-zero, Ansible still marks it red in output
block + ignore_errors: yes	Failures are captured, marked orange, but execution continues to the next block. Registered vars are still set. This is the cleanest pattern.

---

JSON Output Schema

Each test produces one entry in test_results[]. The complete report looks like:

{
  "meta": {
    "standard": "IEC 62443-3-3",        // Standard being validated
    "security_level": "SL2",             // Target security level
    "target": "ics-gateway-01",          // inventory_hostname
    "timestamp": "2026-07-07T16:42:00+02:00",  // ISO 8601
    "executed_by": "auditor"             // ansible_user_id
  },
  "summary": {
    "total": 15,                         // Total test count
    "passed": 10,                        // passed == true
    "failed": 4,                         // passed == false
    "skipped": 1                         // passed == "review" or "skipped"
  },
  "by_category": [                       // Results grouped by FR category
    ["FR1 - ...", [{...}, {...}]],
    ["FR2 — Use Control", [{...}]]
  ],
  "by_severity": {                       // Drill-down by criticality
    "critical": [{...}],
    "high": [{...}],
    "medium": [{...}],
    "low": [{...}]
  },
  "failures": [{...}],                   // Only tests where passed == false
  "results": [                           // All results, ordered by execution
    {
      "test_id": "IAC-05",               // Unique identifier (category-number)
      "category": "FR1 - ...",           // Which IEC 62443 FR section
      "requirement": "SR 1.5 — ...",     // Specific SR (System Requirement)
      "description": "Password...",       // Human-readable check
      "passed": false,                   // bool | "review" | "skipped"
      "expected": "minlen >= 14...",     // What compliance looks like
      "actual": "minlen = 8",            // What we found
      "severity": "high",                // critical | high | medium | low
      "remediation": "Set minlen=14..."  // How to fix
    }
  ]
}

passed Field Semantics

Value	Meaning	Icon
true	Automated check passed	✅
false	Automated check failed	❌
"review"	Requires human assessment	🔍
"skipped"	Not applicable to this target	⏭️

severity Field Semantics

Value	Meaning	Examples
critical	Allows immediate compromise	Empty passwords, world-writable sudoers
high	Defeats a core SL2 control	No firewall, no auditd, weak password policy
medium	Weakens a control	Password aging not set, some unused accounts
low	Best-practice gap	Extra listening ports, stale accounts

---

IEC 62443-3-3 SL2 Coverage

IEC 62443-3-3 defines 7 Foundational Requirements (FRs), each with System Requirements (SRs) at escalating Security Levels (SL1–SL4). SL2 adds to SL1.

Mapping of Implemented Tests

Test ID	FR	SR	Description	Severity
IAC-01	FR1	SR 1.1	No duplicate UIDs in /etc/passwd	high
IAC-02	FR1	SR 1.3	No default/unnecessary system accounts	medium
IAC-03	FR1	SR 1.3	No human accounts that have never logged in	low
IAC-04	FR1	SR 1.4	No empty or trivially-weak password hashes	critical
IAC-05	FR1	SR 1.5	Password min length ≥ 14 (pwquality)	high
IAC-06	FR1	SR 1.5	≥ 3 character classes required (pwquality)	medium
IAC-07	FR1	SR 1.7	PASS_MAX_DAYS ≤ 90	medium
IAC-08	FR1	SR 1.7	PASS_MIN_DAYS ≥ 1	low
IAC-09	FR1	SR 1.11	Account lockout after ≤ 5 failures (pam_faillock)	high
IAC-10	FR1	SR 1.6	Password history ≥ 5 (pam_pwhistory)	medium
UC-01	FR2	SR 2.1	No unrestricted NOPASSWD sudo	high
UC-02	FR2	SR 2.1	/etc/sudoers owned root:root, mode 0440	critical
UC-03	FR2	SR 2.5	Shell idle timeout ≤ 900s (TMOUT)	medium
UC-04	FR2	SR 2.4	Audit rules immutable (-e 2)	high
UC-05	FR2	SR 2.8	auditd service active	high
UC-06	FR2	SR 2.8	≥ 4 critical syscall types audited	medium
RDF-01	FR5	SR 5.1	Host-based firewall with rules	critical
RDF-02	FR5	SR 5.1	Default INPUT policy is DROP	high
RDF-03	FR5	SR 5.3	No insecure legacy services (telnet, rsh, ftp)	critical
RDF-04	FR5	SR 5.3	Listening TCP ports documented	low

Not Yet Implemented (FR3, FR4, FR6, FR7)

FR	Key SL2 Controls	Suggested Checks
FR3 — System Integrity	Malware protection, file integrity, secure boot	AIDE/IMA, ClamAV service, /proc/sys/kernel/secure_boot
FR4 — Data Confidentiality	Encryption at rest/transit	TLS version on listening ports, LUKS/dm-crypt, SSH ciphers
FR6 — Timely Response	Audit log forwarding, alerting	rsyslog remote config, auditd dispatcher
FR7 — Resource Availability	DoS protection, backup	Disk quota, CPU limits, backup cron/schedule

---

Adding a New Test

Step-by-Step

1. Choose an FR category file (or create a new suites/frN_*.yml).

2. Copy the test template:

   # ── SR X.Y: Short description ───────────────────────────────
   
   - block:
       - name: "Gather: What to check"
         ansible.builtin.shell: |
           your-check-command-here
         register: _myvar
         changed_when: false
   
       - name: "Evaluate: XXX-NN"
         ansible.builtin.set_fact:
           test_results: "{{ test_results + [{
             'test_id': 'XXX-NN',
             'category': 'FRN — Category Name',
             'requirement': 'SR X.Y — Requirement Name',
             'description': 'One-line description of the check',
             'passed': ( _myvar.stdout | trim | length > 0 ),  # boolean expression
             'expected': 'What compliance looks like',
             'actual': _myvar.stdout | trim | default('NOT FOUND', true),
             'severity': 'high',       # critical|high|medium|low
             'remediation': 'Command or steps to fix'
           }] }}"
     ignore_errors: yes
   

3. Register the suite in playbooks/site.yml:

   - name: "Suite: FRN — Category Name"
     block:
       - ansible.builtin.include_tasks: suites/frN_category.yml
     ignore_errors: yes
   

4. Add to the coverage table in this README.

Test ID Naming Convention

• Prefix: 3-letter abbreviation of the FR (e.g., IAC, UC, RDF, SYS, CONF, RESP, AVAIL)
• Number: Sequential within that FR, padded to 2 digits (01, 02, ...)
• Examples: IAC-01, UC-12, RDF-05

Tips for Writing Robust Checks

Tip	Why
Always use changed_when: false on gather tasks	Tests must never report as "changed"
Use `	trim` on shell output
Use `	default('NOT FOUND', true)` for missing files
Parse numbers with `	int` before comparison
Use `	regex_search(...)` for structured config
Test an or of pam_faillock and pam_tally2	Different distros use different PAM modules

---

Rendering Reports

Python Renderer (reports/render_report.py)

No dependencies beyond Python 3 stdlib. Two output formats:

# Terminal box-drawing (default):
python3 reports/render_report.py reports/localhost-2026-07-07.json

# Markdown (for wiki, PR comment, etc.):
python3 reports/render_report.py reports/localhost-2026-07-07.json --format md

Terminal output uses Unicode box-drawing characters (╔═╗║╚╝), groups results by FR category, and shows a failure-detail section at the bottom.

Markdown output produces a GitHub-flavored table plus per-failure sections with remediation instructions.

Go Renderer (reports/render.go)

Requires Go ≥ 1.21. Uses text/template with an external template file:

cd reports
go run render.go sample-output.json report.gohtml

The template (report.gohtml) is a standalone file — you can customize it without recompiling. Template functions:

Function	Purpose
passIcon	Maps passed value to ✅/❌/🔍
severityIcon	Maps severity string to 🔴/🟠/🟡/🟢
title	Capitalizes first letter of each word
divf	Float division for compliance rate

Extending: Custom Go Templates

Copy report.gohtml to my-report.gohtml, modify it, and run:

go run reports/render.go reports/*.json my-report.gohtml

The template receives the full JSON document as . (a map[string]any). Access fields with (index . "key") since it's untyped.

---

File Reference

playbooks/site.yml

Entry-point playbook. Runs against hosts: all. Pre-tasks create the local reports/ directory. Each suite is included as a named task wrapped in a block with ignore_errors: yes. The final task includes library/report.yml to aggregate results.

Variables set here:

• report_dir: "./reports" — where JSON output lands

playbooks/suites/fr1_auth.yml

FR1 — Identification and Authentication Control. 10 tests covering:

• Duplicate UIDs (SR 1.1)
• Default/unused accounts (SR 1.3)
• Empty password hashes (SR 1.4)
• Password complexity via libpwquality (SR 1.5)
• Password history via pam_pwhistory (SR 1.6)
• Password aging via login.defs (SR 1.7)
• Account lockout via pam_faillock (SR 1.11)

Uses: /etc/passwd, /etc/shadow, /etc/security/pwquality.conf, /etc/login.defs, /etc/pam.d/common-auth, /etc/pam.d/common-password, lastlog

playbooks/suites/fr2_use_control.yml

FR2 — Use Control. 6 tests covering:

• Unrestricted sudo access (SR 2.1)
• sudoers file permissions (SR 2.1)
• Shell idle timeout via TMOUT (SR 2.5)
• auditd immutable mode (SR 2.4)
• auditd service status (SR 2.8)
• Audited syscall coverage (SR 2.8)

Uses: /etc/sudoers, /etc/sudoers.d/, /etc/profile, /etc/bash.bashrc, /etc/audit/audit.rules, systemctl

playbooks/suites/fr5_data_flow.yml

FR5 — Restricted Data Flow. 4 tests covering:

• Host-based firewall with active rules (SR 5.1)
• Default DROP inbound policy (SR 5.1)
• Insecure legacy services disabled (SR 5.3)
• Listening port inventory for review (SR 5.3)

Uses: iptables/nft, systemctl, ss

playbooks/library/report.yml

Report aggregation. Included last by site.yml. Builds a __report dict from the test_results[] list, prints a summary box to the console, and writes reports/<hostname>-<date>.json to the local machine.

Key computed fields:

• summary.total/passed/failed/skipped — basic counts
• by_category — results grouped by FR category string
• by_severity — results grouped into critical/high/medium/low buckets
• failures — only tests where passed == false

reports/render_report.py

Python-based report renderer. Two modes:

• terminal (default) — Unicode box-drawing, grouped by category, failure details
• md — GitHub-flavored Markdown tables

Functions:

• pass_icon(v) → ✅/❌/🔍/⏭️ based on bool/string
• severity_icon(s) → 🔴/🟠/🟡/🟢
• render_terminal(report) — writes to stdout
• render_markdown(report) — writes to stdout

reports/render.go

Go-based report renderer. Uses text/template and golang.org/x/text/cases. Loads an external .gohtml template file so templates can be customized without rebuilding. Registers template functions for pass/severity icons and arithmetic.

reports/report.gohtml

Go template producing a terminal box-drawing report. Iterates over results, renders a summary header, a per-result listing with icons, and a failure-detail section. Uses template functions registered by render.go.

reports/sample-output.json

A hand-crafted example JSON report for testing the renderers without running Ansible. Contains 15 results (10 passed, 4 failed, 1 review) across FR1 and FR2. Useful for CI validation of the report rendering pipeline.

inventory.ini

Standard Ansible inventory. Structure:

[all]
localhost ansible_connection=local

[ics_assets]
# Add real targets here

run.sh

End-to-end wrapper that:

1. Runs ansible-playbook with the given arguments
2. Finds the latest JSON report in reports/
3. Renders it via render_report.py (or render.go if available)
4. Falls back to a Python one-liner summary if neither is available

---

Design Decisions & Tradeoffs

Why Ansible and Not a Dedicated Compliance Scanner?

Decision	Rationale
Ansible	Already deployed in most OT environments. No new agent, no new approval.
Shell-based checks	shell module is the most flexible. We trade idempotence for expressiveness. changed_when: false keeps it clean.
Inline set_fact vs custom module	Custom Ansible modules require Python on the control node. Inline facts work everywhere. Less code to maintain.
test_results[] list vs file-per-test	A single growing list is simpler than per-file concatenation. At 100+ tests the memory footprint is negligible (~50KB).
ignore_errors at block level vs task level	Block-level isolates failures cleanly. A failed gather still reaches evaluate.
JSON as canonical output	Machine-readable, schema-validatable, ingestible by SIEM/SOAR/Jira.
Go templates for rendering	text/template is standard, fast, and supports external template files for customization without recompilation.

Known Limitations

1. Shell-heavy: Tests depend on shell commands. Different distros may need different check commands (e.g., apt vs rpm package verification). Mitigate with when: ansible_os_family == 'Debian' variants.

2. No built-in Windows support: Currently Linux-only. Windows checks would need win_shell/win_regedit modules and a separate suite.

3. No diff/drift detection: Each run is independent. To detect drift between runs, diff two JSON reports externally (jd, diff, or a time-series DB).

4. No immediate pass/fail exit code: ansible-playbook always exits 0 unless a task fails without ignore_errors. For CI integration, parse the JSON summary and use --format json output.

5. Scalability at 500+ targets: Running against large inventories is fine (Ansible's strength), but the JSON file-per-host can be unwieldy. Consider post-processing into a single aggregated report.

---

Comparison to Alternatives

Tool	Type	Pros	Cons
Inspec	Ruby DSL, Chef ecosystem	Rich compliance profiles, CIS/STIG built-in	Ruby runtime; less common in OT
Goss	YAML config, Go binary	Fast, simple, YAML-based	No native IEC mapping; limited to local checks
OpenSCAP	XML-based, SCAP standard	NIST/STIG aligned, XCCDF/OVAL	Heavy, complex, US-govt focused
Lynis	Shell script, single binary	Broad Linux coverage, professional reports	Non-extensible output format
This project	Ansible + JSON + Go/Python templates	Zero new agents, customizable, IEC 62443 mapped	Requires Ansible; shell-dependent checks

---

Roadmap

• FR3 — System Integrity: File integrity (AIDE/IMA), malware scanner status, secure boot, /tmp noexec
• FR4 — Data Confidentiality: TLS version/cipher audit on listening ports, disk encryption, SSH hardening
• FR6 — Timely Response: rsyslog remote forwarding, auditd dispatcher, journald persistent storage
• FR7 — Resource Availability: Disk quotas, CPU/memory limits, backup schedule verification
• Aggregated multi-host report: Single HTML/PDF report across all inventory hosts
• GitLab CI / GitHub Actions integration: Pipeline stage that runs playbook + posts markdown report as PR comment
• Custom Ansible module: Replace shell-based checks with a native iec62443_test module for cleaner code
• CIS Benchmark mapping: Dual-map tests to both IEC 62443-3-3 and CIS distribution benchmarks