ansible-testing/source_documents/testing_tools/overview.md

Testing tool overview

A set of tools to be selected and orchestrated to supply the full test coverage from initial software project to final delivery of projects to customers.

End to en overview of testing tools

An overview of the testing types, their current adoption and whom in BEUMER are responsible for their operation

OSSRA

• State: not adopted
• Owner: TBD
• Tool-name: MITRE HipCheck
• Description: Open Source Software Risk Assessment is done as a part of evaluating supply chain risks from dependencies to OSS. The overall project risk can stem from reliance on an immature, abandoned or badly maintained project. Tools Such as HipCheck can be used to quickly assess risks from project dependencies and single out which dependencies require further analysis or in some cases outright disqualify source projects by policy.

SAST

• State: Imlemented, but analyzed for possible replacement
• Owner: P&T
• Tool-name: SonarQube
• Description:

SCA / SBOM

• State: Partially Implemented
• Owner: P&T
• Tool-name: bespoke tooling, Trivy, DependencyTrack
• Description:

Component / API

• State: Planned
• Owner: SW tests
• Tool-name: TBD
• Description: Component and API testing verifies that services, interfaces and integrations behave correctly at the contract level, including authentication, authorization, input validation, error handling and response integrity. This is important for exposing breaking changes between dependent components and validating expected behavior before release.

IaCST

• State: Planned
• Owner: SW tests
• Tool-name: TBD
• Description: Infrastructure-as-Code security testing validates IaC templates, deployment definitions and configuration baselines for insecure defaults, unsafe settings, drift and policy violations before systems are built. It reduces the risk of introducing cloud or on-prem misconfigurations during provisioning.

IAST

• State: Planned
• Owner: SW tests
• Tool-name: TBD
• Description: Interactive Application Security Testing combines dynamic execution with source-level insight to detect vulnerabilities in running applications, including unsafe data flows, injection points and authentication weaknesses. It is particularly useful for validating real application behavior in a test environment.

DAST / runtime scan (ASVS 5)

• State: Planned
• Owner: SW tests
• Tool-name: ZAP / ASVS-aligned scanning tooling
• Description: Dynamic application security testing exercises the application at runtime to detect misconfigurations, authentication weaknesses, insecure session handling and web application vulnerabilities. When aligned to ASVS 5, it provides evidence that key security requirements are being met in deployed or near-production environments.

DAST - Destructive Pen Test

• State: Planned
• Owner: SW tests
• Tool-name: External security test partner / approved tooling
• Description: Destructive penetration testing goes beyond routine vulnerability scanning to validate exploitability and system resilience against realistic attack paths. This type of testing is typically conducted in controlled environments with clear scope, approval and rollback procedures.

Performance / load / Fuzz

• State: Planned
• Owner: SW tests
• Tool-name: TBD
• Description: Performance, load and fuzz testing measure stability, throughput, response time and resilience under stress, sustained load and malformed or unexpected input. This helps uncover bottlenecks, resource exhaustion issues and reliability failures before deployment.

Integration / Regression

• State: Planned
• Owner: SW tests
• Tool-name: Ansible + test automation frameworks
• Description: Integration and regression testing confirm that components work together as expected across interfaces and operational flows, while also verifying that new changes do not break previously working behavior. This is a core part of release confidence for system-level changes.

Operational Vulnerability Scan

• State: Planned
• Owner: SW tests
• Tool-name: TBD
• Description: Operational vulnerability scanning focuses on live system components such as hosts, services, containers, appliances and network devices to identify known issues that require remediation or compensating controls. It supports continuous assurance that deployed environments remain within acceptable risk levels.

Hardening Benchmark

• State: Planned
• Owner: SW tests
• Tool-name: CIS benchmarks / platform-specific hardening baselines
• Description: Hardening benchmark testing validates that deployed systems, operating systems and infrastructure components match approved security baselines. This reduces the attack surface and ensures configuration settings align with internal standards and regulatory expectations.

FAT / SAT

• State: Planned
• Owner: SW tests
• Tool-name: Site acceptance and factory acceptance test automation
• Description: Factory Acceptance Testing and Site Acceptance Testing confirm that systems meet the agreed specification and operational requirements before handover and customer acceptance. These tests validate readiness, functionality and integration in realistic deployment conditions.

OBOM / Asset Inventory Management

• State: Planned
• Owner: SW tests
• Tool-name: TBD
• Description: An operational bill of materials and asset inventory tracks software, firmware, hardware, configurations and dependencies across the deployed environment. This is essential for asset visibility, vulnerability prioritization, lifecycle management and change control.

Selected tooling

Orchestration

• GitLab CI/CD
• Ansible

Connectivity

• TBD Proxying services

Standards

• IEC 62443-4-2
  • Mandatory in BEUMER. All components must provide SL-C 2 or be deployed in a context wherein countermeasures are in place to supply the gap.
• IEC 62443-3-3
  • Mandatory in BEUMER. SL-T 2 is the mandatory level for the systems supplied to BEUMERs customers.
• ASVS 5.0
  • Possibly a supporting standard in the sense that it can provide testable requirements for i.e. strong cryptography. Community-supploed tests have been developed that may be useful in providing ASVS assessment automation, and a mapping with some requirements from IEC62443 is possible.
• CIS hardening benchmarks
  • CIS provides a large set of hardening benchmarks as well as build-kits. Hardening benchmarks are available for commong infrastructure elements such as VMWare vSphere, Hyper-V, Windows, Cisco devices, Major Linux Distributions.

Testing

• Ansible - can be used both to orchestrate other testing tools, or it can execute a testing suite directly using python frameworks such as PyTest.
• ZaProxy - can be used to automate a "passive" DAST whereing the the tool connects to a remote api and detects misconfiguration from non-exploitive communication patterns. Projects exist that pair an older version of ASVS to ZaProxy scanning metrics, which could be updated to provide evidence for ASVS 5 compliance.

Report Generation

• gomplate
• pandoc