Testing after significant change: building a trigger list you can defend

Where the obligation comes from

Four separate sources say a version of the same thing, and it is worth seeing them together because the wording differs in ways that matter.

PCI DSS v4.0.1: the most explicit

Requirements 11.4.2 and 11.4.3 set out internal and external penetration testing respectively, and both list the conditions as separate bullets: “At least once every 12 months” and, on its own line, “After any significant infrastructure or application upgrade or change”. They are two independent triggers, not one compound condition, and reading them as a single sentence is a common misquote imported from the older v3.x wording.

The same structure repeats elsewhere in the standard. Requirement 11.3.1 sets internal vulnerability scans “At least once every three months”, and 11.3.1.3 adds that “Internal vulnerability scans are performed after any significant change”. Requirement 11.4.5 requires segmentation controls to be tested “At least once every 12 months and after any changes to segmentation controls/methods”. Requirement 11.4.4 then closes the loop: exploitable findings are corrected and “penetration testing is repeated to verify the corrections”.

The NIS2 implementing regulation: frequency is yours to set

Commission Implementing Regulation (EU) 2024/2690 does not name an interval at all. Annex point 6.5.2 requires the entity to “establish, based on the risk assessment carried out pursuant to point 2.1, the need, scope, frequency and type of security tests”. Recital 46 then describes when testing typically happens: “The relevant entities may conduct security tests on their network and information systems at set-up, after infrastructure or application upgrades or modifications that they deem significant, or after maintenance.” Note the phrase “that they deem significant”. The regulation delegates the judgement and expects you to have made it deliberately.

The regulation binds a defined set of digital infrastructure and digital service providers rather than every NIS2 entity, but it remains the clearest published statement of what an EU regulator now expects security testing to look like, and national authorities reference it.

DORA: change-triggered testing for specific entities, risk-based for everyone else

Regulation (EU) 2022/2554 requires financial entities other than microenterprises to run a testing program following “a risk-based approach … duly considering the evolving landscape of ICT risk”, and to ensure “at least yearly” that appropriate tests are conducted on all ICT systems supporting critical or important functions. For two categories it goes further and ties testing directly to deployment: Article 25(2) requires that “Central securities depositories and central counterparties shall perform vulnerability assessments before any deployment or redeployment of new or existing applications and infrastructure components, and ICT services supporting critical or important functions”.

OWASP: verification as part of change management

The OWASP testing framework states the engineering version of the same rule, and states it as an unqualified expectation: “After every change has been approved and tested in the QA environment and deployed into the production environment, it is vital that the change is checked to ensure that the level of security has not been affected by the change. This should be integrated into the change management process.” It also sets a floor for the periodic half: “Monthly or quarterly health checks should be performed on both the application and infrastructure to ensure no new security risks have been introduced and that the level of security is still intact.”

What “significant” means, and who decides

None of the frameworks defines it for you, and PCI SSC says so explicitly. Its penetration testing guidance notes that “What is deemed ‘significant’ is highly dependent an entity’s risk-assessment process and on the configuration of a given environment. Because of this variability, a significant change is not prescribed by PCI DSS. If the change could impact the security of the network or allow access to cardholder data, it may be considered significant by the entity.” The same document gives the cleanest one-line statement of why the trigger exists at all: “Penetration testing of significant changes is performed to ensure that controls assumed to be in place are still working effectively after the upgrade or modification.”

PCI SSC does publish a list of categories to consider. Its FAQ on the subject names: new hardware, software or networking equipment added to the cardholder data environment; any replacement or major upgrade of hardware and software in that environment; any change in the flow or storage of account data; any change to the boundary of the environment or to assessment scope; any change to the underlying supporting infrastructure, including directory services, time servers, logging and monitoring; and any change to third-party vendors or service providers that support the environment. That list generalizes well beyond payments, and it is a reasonable starting point for a trigger list in any sector.

Building the trigger list

A usable trigger list has three columns: what happened, what that means you test, and what evidence the test leaves behind. Keep it short enough that a change manager will actually read it, and keep it in the change process rather than in the security wiki.

The last row is not optional decoration. Commission Implementing Regulation (EU) 2024/2690 requires that “In the event that the regular change management procedures could not be followed due to an emergency, the relevant entities shall document the result of the change, and the explanation for why the procedures could not be followed.” Emergency changes are exactly the ones most likely to break something quietly, and exactly the ones least likely to be tested.

Why the trigger list matters more the faster you ship

The interval half of the obligation is a fixed cost. The trigger half scales with your change rate, and that is the variable most organizations have not looked at recently.

The 2024 Accelerate State of DevOps Report, based on a survey of nearly 3,000 practitioners, groups teams into four performance levels derived by cluster analysis of the responses rather than set in advance. Elite performers, 19% of respondents, deploy “on demand (multiple deploys per day)”. High performers, 22%, deploy “between once per day and once per week”. Medium performers, 35%, deploy “between once per week and once per month”. Low performers, 25%, deploy “between once per month and once every six months”. Each share carries an 89% uncertainty interval.

Read that alongside the trigger list and the arithmetic is uncomfortable. If 41% of teams deploy at least weekly, then for those teams a rule that says “test after significant change” is not an occasional exception to the annual test. It is the primary mechanism, and the annual test is the exception. A program designed the other way round will be permanently behind.

The practical resolution is not to test every deploy, which nobody can afford and no framework requires. It is to make significance a property of the change rather than of the calendar, so that the ninety routine deployments a quarter pass through untested and the three that touch authentication do not.

What to test after each kind of change

Matching the test type to the change is what keeps the trigger list affordable. Three rough tiers cover most cases.

• Automated only. Dependency upgrades with no change to security controls, content changes, infrastructure scaling. A scan plus a dependency check is proportionate, and the evidence is the scan record.
• Targeted manual. Anything touching authentication, authorization, session handling, tenancy, payment flow, file handling or a new external interface. Scope the manual work to the changed surface, and include regression against previously fixed findings in the same area.
• Full scope, brought forward. A re-platforming, a new environment, a merger bringing an unfamiliar estate inside the boundary, or a change to the assessment scope itself. This is the case for pulling the annual engagement forward rather than adding a cycle.

One thing to insist on in the second tier: regression against previously fixed findings. A change to an authorization layer is the single most reliable way to reintroduce a bug that was closed six months ago, and a program that never revisits closed findings will not notice.

Recording a decision not to test

This is the part most programs are missing, and it is cheap to add. Not every change warrants testing, and every framework accepts that. What none of them accepts is a gap with no explanation.

Commission Implementing Regulation (EU) 2024/2690 sets the pattern for vulnerability handling: where a vulnerability’s impact does not justify remediation, “the relevant entities shall document and substantiate the reason why the vulnerability does not require remediation”. The same discipline applied to testing decisions turns an awkward audit question into a one-line answer. A short record per change assessed and not tested, naming who decided and on what basis, costs a minute and closes a hole that otherwise looks like negligence.

A workable record has four fields: the change reference, the date, the assessment (why this was not significant for testing purposes), and who made it. Store it with the change, not separately.

Five ways trigger lists fail

• The list exists but the testers never hear. There is no route from the change process to the provider, so triggers fire and nothing happens. Fix this with a standing item in change approval, not with goodwill.
• Significance is defined by cost. A change is deemed insignificant because the testing budget is spent. Write the threshold before the budget so the two arguments stay separate.
• Infrastructure changes are excluded. Teams remember to test application releases and forget directory services, logging, monitoring and time servers, all of which PCI SSC names explicitly as significant-change categories.
• Third-party changes are invisible. A supplier re-platforms a service you depend on and nobody treats it as a change to your environment. PCI SSC lists changes to third-party service providers among the categories to consider, and NIS2 Article 21(2)(d) puts supply chain security among the minimum measures.
• No regression testing. Nothing checks whether a change reopened a previously closed finding, which is exactly what a retest record exists to prevent. That record set is the subject of the evidence guide.

If you are still deciding whether the trigger half of the obligation justifies a full program, the cadence designer takes change frequency as its first input and will tell you when scheduled testing tied to releases is the better answer than a subscription.