Scott vonFischer
A common source of confusion when designing an Information Security Program is the difference between scanning, vulnerability testing, penetration testing, and risk assessments. Each of these is an important component of a vulnerability management program, and have a distinct goal, scope, skill, cost, and level of effort.
Discovery Scanning
One of the first steps in penetration testing is reconnaissance or the process of gathering information to better understand an intended scope. A rudimentary component of reconnaissance is a discovery scan, which identifies the network hosts, operating systems, network addresses, and then the associated ports (service channels) on those systems. Often, the network topology will give insight into the types of applications and devices operating within those in-scope targets.
There are three basic types of discovery scans: host detection, service enumeration, and OS fingerprinting; all can be configured as authenticated or unauthenticated, and use various tools and protocols as a conspicuous or stealth sweep. Host detection will quantify the scope of host systems and help identify zombie hosts. Service enumeration tests identified hosts for open TCP and UDP listening services. OS fingerprinting will probe known OS used ports with specifically crafted packed intended to reveal operating systems types, versions, and other useful information.
Vulnerability Testing
Vulnerability testing is generally an automated scan for known potential vulnerabilities in network devices, servers, and applications without exploitation. Often, the greatest misunderstanding is between vulnerability testing and penetration testing. Vulnerability assessments are performed by using an off-the-shelf software package. The software typically produces a report of detected known vulnerabilities, an indication of the severity of the vulnerability, and basic remediation steps.
Vulnerability tests are good at detecting issues that are already known to the security community, hackers, and the software vendors. There are vulnerabilities that are unknown to the public at large, and these scanners will not find them. Most automated scanners are sophisticated enough that assessments can be managed by junior resources, although moderately skilled resources will provide greater value. A vulnerability assessment does not exploit vulnerabilities, try to obtain sensitive data, or elevate privileges – that is the function of penetration testing.
| Type | Scope | Cost | Risk | Skill | Automation | Frequency |
| Vulnerability Scan | A fully automated scan of a broad target scope. | Low | Low | Low | High | High |
| Vulnerability Test | A fully automated scan of a limited target scope. Findings are reviewed for false positives and red flags. | Low | Low | Medium | High | High |
Use cases for vulnerability testing are:
- identifying obvious technical issues (for example, missing patches, poor configuration items, extraneous services, and unchecked default passwords)
- testing a high number of assets for known vulnerabilities
Penetration Testing
While vulnerability scanning (or assessments) check for known vulnerabilities, penetration testing exploits vulnerabilities in the system architecture. Penetration tests are meant to reveal how organizations are susceptible to exploitation of known and unknown vulnerabilities through different attack vectors, beyond what is determined with vulnerability assessment tools (VAT). The scope of a penetration test targets a specific application, system, or network of systems, and there is always a human factor involved. There is no such thing as automated penetration testing; it requires the use of tools, sometimes a lot, and an experienced person to conduct the testing.
Additionally, penetration tests often exploit new vulnerabilities or discover security flaws that are not known to normal business processes or to the automated VAT. Because of its cost and its higher chance of causing outages, penetration testing should be scheduled and coordinated with resource owners and custodians.
| Type | Scope | Cost | Risk | Skill | Automation | Frequency |
| White Hat
(grey box) |
Tools and processes used to exploit a specific target. Credentials and knowledge of the components are known. | Med | Med | High | Med | Med |
| Black Hat
(black box) |
Tools and processes used to exploit a specific target. Credentials and knowledge of the components are unknown. | Med | High | Advanced | Low | Low |
Use cases for penetration testing are:
- Validating design and architecture of new technology
- Exploit or reject tool-based findings on a limited scope of assets
- Attack simulation
Risk Assessment
A risk analysis is often confused with the previous phases; however, a risk analysis does not require any scanning tools or applications. It is a discipline that analyzes a specific vulnerability found and validated from a penetration test, and attempts to ascertain the risk – including financial, reputational, business continuity, regulatory and others – to the company if the vulnerability were exploited.
Many factors are considered when performing a risk analysis: asset, vulnerability, threat, and impact to the company. A risk analysis, when completed, will have a final risk rating with mitigating controls that can further reduce the risk. Business managers can then take the risk statement and mitigating controls and decide whether to implement them.
The different concepts explained here are not exclusive of each other, but rather complement each other. In many information security programs, discovery scans and vulnerability assessments are the first steps – they are used to perform wide sweeps of a network to find missing patches or misconfigured software. From there, penetration tests discover additional vulnerabilities, see how exploitable discovered vulnerabilities, and then ascertain the risk compared to the cost/benefit of fixing the vulnerability. Of course, you don’t need either to perform a risk analysis. Risk can be determined anywhere a threat and an asset are present.
Use cases for risk assessments are:
- Establish the impact of newly announced vulnerabilities or to interpret the results of a vulnerability assessment
- Assessments of security risks associated with new application or infrastructure
- Assessments of the risks associated with technology and services (especially outsourcing)
- Prioritization of security program activities and projects
- As a key part of a continuous improvement program, in which business processes are identified, assessed, prioritized, and improved on an annual or biannual basis
As shown below, the different types of vulnerability tests roughly equate to the different phases of the “cyber kill chain” (simplified). Discovery is to reconnaissance as vulnerability testing is to exploitation, and as penetration testing is to system compromise.
