Continuous testing, vulnerability scanning and bug bounty: where the lines are

Why the boundary is a commercial problem, not a semantic one

Nobody misuses these terms out of confusion. They get blurred because the words carry different prices and different assurance value, and because a buyer under compliance pressure often needs the words more than the work.

The failure mode is specific and repeatable. An organization needs to satisfy a requirement that says “penetration testing”. It buys a subscription whose deliverable is a scanner running weekly with a portal in front of it. The assessor asks for a penetration test report, and the entity produces a scan output. That conversation ends badly, and it ends badly late, because the gap only becomes visible at assessment.

The three sections below each state what the control answers, who performs it, what you receive and where it stops. The bounty section is deliberately conservative: several claims commonly made about bug bounties are not supported by any authoritative source, and this guide does not make them.

Vulnerability scanning

What it answers

Which known, published weaknesses are present on the assets you pointed the scanner at, as of the last run. It is a coverage-and-currency control: broad, shallow, cheap per asset, and repeatable without human cost.

Where it stops

NIST SP 800-115 is unusually direct about the limits, and the language is worth quoting because vendors rarely do. Network-based vulnerability scanning “uncovers vulnerabilities only for active systems”, “generally covers surface vulnerabilities, and is unable to address the overall risk level of a scanned network”. Scanners “can have a high false positive error rate (i.e., reporting vulnerabilities when none exist)”, they “rely on a repository of signatures”, and they “suffer from the high false negative rates that characterize other signature-based tools”. NIST also notes that “an individual with expertise in networking and OS security should interpret the results”, which is the part most commonly left out of the price.

OWASP reaches the same conclusion from the application side: automated tools “are necessary and valuable, but they are insufficient on their own”, because they “excel at finding known, signature-based vulnerabilities” and “do not understand business logic or context”.

What it is genuinely good for

Continuous scanning is the right control for the questions penetration testing answers badly: has anything new appeared on the perimeter, is a published CVE present on an asset we own, has a configuration drifted from baseline since Tuesday. CISA’s own free service is described in exactly those terms: it “continuously monitors and assesses internet-accessible network assets”, with “weekly reports of all findings” and “ad-hoc alerts about urgent findings”. Note that CISA calls it vulnerability scanning throughout and never calls it a penetration test.

Regulators expect it too, and separately from penetration testing. Commission Implementing Regulation (EU) 2024/2690 requires relevant entities to “perform, where appropriate, vulnerability scans, and record evidence of the results of the scans, at planned intervals”. PCI DSS v4.0.1 requires internal vulnerability scans at least once every three months and again after any significant change. Those obligations are not satisfied by penetration testing, and penetration testing obligations are not satisfied by scanning.

Penetration testing, single or continuous

What it answers

Whether a skilled attacker, working under agreed constraints, can actually break the scope you defined. The distinguishing characteristic in NIST’s definition is combination: “Most penetration tests involve looking for combinations of vulnerabilities on one or more systems that can be used to gain more access than could be achieved through a single vulnerability.” A scanner reports two medium findings. A tester reports that the first one leaks a token that unlocks the second one, and that together they read every tenant’s data.

Where it stops

On the date. NCSC: a penetration test “can only validate that your organisation’s IT systems are not vulnerable to known issues on the day of the test”, and “it’s not uncommon for a year or more to elapse between penetration tests. So, vulnerabilities could exist for long periods of time without you knowing about them if this is your only means of validating security.” That sentence is the entire argument for a cadence, made by a national authority rather than a vendor.

It also stops well short of completeness. OWASP, quoting Gary McGraw: “In practice, a penetration test can only identify a small representative sample of all possible security risks in a system.” Running it more often shortens the interval between samples. It does not turn a sample into a census.

What continuous adds

Frequency, a finding lifecycle that ends in retest rather than in a report, and evidence that stays current. Nothing else. It is the same activity, bought differently, and it inherits every limitation above. That is covered in detail in what continuous penetration testing is.

Bug bounty programs and vulnerability disclosure

What they are

NIST’s glossary defines a bug bounty as “a method of compensating individuals for reporting software errors, flaws, or faults (‘bugs’) that might allow for security exploitation or vulnerabilities”. CISA’s Binding Operational Directive 20-01 gives the fullest authoritative description: “In bug bounty programs, organizations pay for valid and impactful findings of certain types of vulnerabilities in their systems or products. A financial reward can incentivize action and may attract people who might not otherwise look for vulnerabilities. This may also result in a higher number of reports or an increase in low-quality submissions. Organizations engaged in bug bounties will frequently use third-party platforms and service vendors to assist in managing and triaging bug reports. Bug bounties may be offered to the general public or may only be offered to select researchers or those who meet certain criteria.”

A vulnerability disclosure policy is a related but distinct thing: a published route for unsolicited reports, usually without payment. NCSC’s toolkit describes the value in terms of expectation-setting: “By providing a clear policy, you define what to expect from someone reporting a vulnerability (as well as what you will do in response). This means you can work with them in an agreed framework.” It also notes something worth internalizing before you treat an inbound report as an incident: “An unsolicited message from someone telling you about a vulnerability may sound scary at first, but don’t panic.”

Where the real difference sits

OWASP states it in one sentence, and it is a better formulation than most of the comparisons in circulation: the testing techniques apply equally to internal testing, third-party penetration tests and coordinated-disclosure or bug-bounty engagements, and “the differences lie in authorization, rules of engagement, and how results are reported and remediated”.

Unpack that into three practical differences. Authorization: in a test, you authorize named people for a defined window; in a bounty, you authorize a class of people indefinitely. Rules of engagement: in a test, scope, timing and technique are negotiated in advance; in a bounty, they are published as policy and the finder decides what to do inside them. Reporting and remediation: a test produces a report you commissioned with a structure you agreed; a bounty produces reports when someone happens to find something, in whatever form the policy or platform imposes.

Triage obligations run the other way

One thing a bounty or disclosure program imposes on you that a test does not: an inbound queue with response expectations. CISA requires agencies to set target timelines for acknowledgement to the reporter, initial assessment of validity and impact, and resolution including notification of the outcome, and to “specify a target time for resolution, in days”. NCSC’s own reporting service commits to assessment “by our triage team within five working days”. If you cannot staff that, a bounty is premature regardless of the security case for it.

Side by side

How they combine in one program

The three are complements, and the shape most mature organizations converge on is layered rather than exclusive.

• Scanning underneath, always on. Surface discovery, published-CVE exposure and configuration drift, at planned intervals, with the results recorded rather than merely alerted on.
• Commissioned testing on a rhythm. Manual work concentrated on what changed since the last cycle, plus a longer full-scope engagement once a year for the parts that never change.
• A disclosure route, whether or not you pay. Somebody will eventually find something and want to tell you. Having a published policy converts that from an emergency into a process, and it costs nothing but the staffing behind it.
• One register, not three. Findings from all three sources should land in the same lifecycle with the same severity criteria, or you will end up with three different definitions of “high” and no way to report on any of them.

Three procurement questions that separate them

If you have one conversation with a prospective supplier, these three questions will tell you which of the three you are actually being sold.

• Show me a sample finding that a tool could not have produced. An authorization bypass across tenants, a business logic abuse, a chain. If every finding in the sample report maps to a CVE or a signature, you are buying scanning.
• How many manual tester-days are in a cycle, and who are they? CREST requires that every individual involved has “appropriate levels of skills, experience and competency” and that the work runs “towards a defined and agreed test specification”. A supplier who cannot answer in days and names is selling platform access.
• What happens after I mark a finding fixed? The answer should be a retest, with a turnaround, inside the cycle. PCI DSS v4.0.1 requires that “penetration testing is repeated to verify the corrections”, and DORA requires validation that gaps are “fully addressed”. A supplier with no answer here is selling discovery only.

Once you know which control you are buying, the next question is when each one has to run. That is set less by the calendar than by what you changed, which is the subject of the guide on testing after significant change, and by what you have to be able to show afterwards, covered in the evidence a program produces.