The evidence a continuous testing program has to produce

Why the evidence is the product

This is not a cynical framing. It follows directly from what the obligations say. Commission Implementing Regulation (EU) 2024/2690 does not require entities to test well; it requires them to “document the type, scope, time and results of the tests, including assessment of criticality and mitigating actions for each finding”. DORA does not ask financial entities to believe their gaps are closed; Article 24(5) requires them to “establish internal validation methodologies to ascertain that all identified weaknesses, deficiencies or gaps are fully addressed”. PCI DSS v4.0.1 requirement 11.4.1 requires “Retention of penetration testing results and remediation activities results for at least 12 months”.

In each case the obligation attaches to a record. Testing that happened but produced no record satisfies none of them, and an organization in that position is paying for security work and getting no compliance value from it, which is the worst of both outcomes.

There is a second reason, less obvious and more useful day to day. A record set with states in it is the only way to answer the question a board actually asks, which is not “what did the test find” but “is the number going down”. A stack of annual PDFs cannot answer that. A finding register with dated transitions can.

The nine artifacts

These are the records a defensible program produces. Not all of them are required by every framework, but each one answers a question somebody will eventually ask, and none of them is expensive if it is produced as a by-product of the work rather than reconstructed afterwards.

Two of these are commonly missing even in well-run programs. The frequency rationale is missing because the cadence was chosen commercially and nobody wrote a risk basis for it afterwards. The retest record is missing because retest was never contracted, which is dealt with below.

What each framework actually asks to see

NIS2 and its implementing regulation

Directive (EU) 2022/2555 itself names no records. Article 21(2) requires measures for “security in network and information systems acquisition, development and maintenance, including vulnerability handling and disclosure” and “policies and procedures to assess the effectiveness of cybersecurity risk-management measures”, and that is the level it operates at.

The detail lives in Commission Implementing Regulation (EU) 2024/2690, which binds a defined set of digital infrastructure and digital service providers. Annex point 6.5 requires a security testing policy and procedures, a documented test methodology, and a documented record of “the type, scope, time and results of the tests, including assessment of criticality and mitigating actions for each finding”, with mitigating actions applied in the case of critical findings. Annex point 7 implements the effectiveness obligation and requires entities to determine “what cybersecurity risk-management measures are to be monitored and measured” and “the methods for monitoring, measurement, analysis and evaluation”. A testing program with a finding register and trend data is a direct answer to point 7; an annual PDF is not.

DORA

Regulation (EU) 2022/2554 asks for the program itself as an artifact. Article 24(1) requires financial entities other than microenterprises to “establish, maintain and review a sound and comprehensive digital operational resilience testing programme as an integral part of the ICT risk-management framework”. Article 24(4) requires independence of testers. Article 24(5) requires procedures to “prioritise, classify and remedy all issues revealed throughout the performance of the tests” and validation methodologies confirming those issues are fully addressed. Article 24(6) requires that “at least yearly” appropriate tests are conducted on all ICT systems and applications supporting critical or important functions.

Article 25(1) lists what the program should contain, and the breadth of the list is the point: “vulnerability assessments and scans, open source analyses, network security assessments, gap analyses, physical security reviews, questionnaires and scanning software solutions, source code reviews where feasible, scenario-based tests, compatibility testing, performance testing, end-to-end testing and penetration testing”. An entity presenting a single penetration test as its testing program is presenting one item from a list of thirteen.

ISO/IEC 27001

The relevant Annex A controls are 8.8, Management of technical vulnerabilities, and 8.29, Security testing in development and acceptance. Neither names an interval, and the control text is copyrighted and not reproduced here. What matters for evidence is the shape of a certification audit rather than the wording: an auditor looks for a defined process, records showing that the process ran as defined, and evidence that outputs were acted on. A program with dated cycles and a finding register with closure states answers all three. A single annual report answers the second one weakly and the third one not at all.

PCI DSS

PCI DSS v4.0.1 is the most prescriptive on records. Requirement 11.4.1 requires a documented methodology including “Review and consideration of threats and vulnerabilities experienced in the last 12 months”, a “Documented approach to assessing and addressing the risk posed by exploitable vulnerabilities and security weaknesses found during penetration testing”, and “Retention of penetration testing results and remediation activities results for at least 12 months”. Requirements 11.4.2 and 11.4.3 require organizational independence of the tester. Requirement 11.4.4 requires that findings are corrected and “penetration testing is repeated to verify the corrections”.

The retest record, and why it is usually missing

Of the nine artifacts, the retest record is the one most likely to be absent, and it is also the only one that evidences risk reduction rather than risk discovery. Everything else in the set proves you looked. Only the retest record proves something got better.

It is missing for a structural reason rather than a careless one. Retest is work performed after the engagement that produced the finding, often months later, frequently on a system that has changed again in the meantime. In a single annual engagement there is no natural place to put it, so it either becomes a separate small purchase that nobody makes, or it silently becomes an internal claim: the ticket is closed, therefore the finding is closed.

That claim is exactly what PCI DSS 11.4.4 and DORA Article 24(5) refuse to accept. A closed ticket evidences that a change was made. It does not evidence that the attack path no longer works, and the two come apart more often than teams expect: incomplete fixes, fixes applied to one environment, fixes that move the flaw rather than remove it, and fixes reverted by a later merge.

A usable retest record contains six things: the original finding reference, the date of the retest, the tester, the method used to re-attempt the original path, the result, and the resulting state. Six fields, produced as part of the cycle. The commercial question to settle before signing is not whether retest exists but how many are included, within what turnaround of you marking a finding fixed, and at what point it becomes chargeable work.

What the report itself should contain

NCSC publishes an explicit list, and it is a better contractual specification than most vendor templates. A test report should include: “any security issues uncovered; an assessment by the test team as to the level of risk that each vulnerability exposes the organisation or system to; a method of resolving each issue found; an opinion on the accuracy of your organisation’s vulnerability assessment; advice on how to improve your internal vulnerability assessment process.”

The fourth and fifth items are the ones almost always missing, and they are the most valuable. NCSC frames penetration testing as “a method for gaining assurance in your organisation’s vulnerability assessment and management processes, not as a primary method for identifying vulnerabilities”, and compares it to a financial audit: “Your finance team tracks expenditure and income day to day. An audit by an external group ensures that your internal team’s processes are sufficient.” A report that lists findings but says nothing about why your own processes did not catch them has done half the job.

For the scope side, NCSC is equally specific: the scoping exercise should produce a document stating “the technical boundaries of the test; the types of test expected; the timeframe and the amount of effort necessary to deliver the testing - usually given in terms of resource days”. Resource days in writing is the single most useful line in any testing contract, because it is the number that cannot be quietly reduced.

Retention

PCI DSS sets a floor of 12 months for penetration testing results and remediation activity results. Nothing in NIS2, its implementing regulation or DORA names a retention period for test records, which in practice means the period is set by your own records policy and by the assessment cycle you are subject to.

Three years is a defensible default for most organizations, for reasons that are practical rather than legal. Certification cycles run to three years. DORA sets threat-led penetration testing at least every three years for identified entities. And a trend line needs more than two data points to mean anything, which is the internal argument for keeping the register rather than just the reports.

Store the record set somewhere that survives a change of provider. Findings held only in a supplier’s platform are findings you lose at contract end, along with the evidence that you closed them.

Preparing for the conversation

When an assessor asks about testing, the sequence of questions is fairly predictable. Having the answer to each one as a document rather than as a recollection is most of the preparation.

• What is your testing policy, and when was it last reviewed? Point 6.5.3 of the NIS2 implementing regulation expects the policy to be reviewed and updated “at planned intervals”, so a review date matters as much as the policy.
• How did you arrive at this frequency? The answer is the risk assessment, referenced by name and date. “Our vendor recommended quarterly” is not an answer.
• Show me the last three cycles. Type, scope, time and results, per cycle. This is where a program visibly outperforms an annual engagement.
• Show me a finding that was closed. Original entry, severity rating, owner, fix, and the retest that confirmed it. If you can produce this for one finding, you can produce it for all of them, and assessors know that.
• What happened after the significant change in March? This is where the trigger list from the change guide earns its keep, including the records of changes assessed and deliberately not tested.
• Who tested, and were they independent? Named individuals, their competency basis, and their relationship to the team that built the system.

If you are still working out what shape of program produces these records at a cost you can justify, the cadence designer maps each layer to the evidence it generates, and the definition guide covers what to put in the contract so the evidence arrives without being chased.