Several types of cyber security testing services are fairly related: application security, information security audits, cybersecurity assessments, penetration tests, and red-team assessments. The boundaries between some of these terms can be vague, and they can be misunderstood even by cybersecurity experts.
As in any consulting service, using the same language with the customers is paramount in cyber security. We have decided to elaborate on the frame of reference and dissolve several misconceptions in this post. We hope it helps our colleagues find common ground when discussing the types of cyber security services with their clients.
Application Security Services
First, we need to distinguish appsec from the rest of the terms. When talking about application security, software developers often mean application pentests or security code reviews. In fact, Application Security is a broad discipline, and narrowing it down to application pentests or security code reviews is counterproductive.
One of the OWASP flagship projects, the Software Assurance Maturity Model (SAMM), provides a well-structured map of modern application security practices. It is much more diverse than application security testing or code security analysis.
Penetration Testing makes up about 1% of SAMM. There are five Business Functions in the model. Each of them contains three high-level application security Practices. Each practice, in turn, drops down to Streams A and B. And each of the streams has three Maturity Levels.
Manual Penetration Testing belongs to maturity level 2 of Stream B of the Security Testing practice in the Verification business function, occupying about 1/5/3/2/3=0.0(11) of the SAMM 2.0 document surface.
Even if we extend our definition of appsec to code security reviews, practice maturity assessments, and the audits of appsec process effectiveness, all these activities would be split between Governance and Verification, leaving three more business functions untouched. Thus, all these practices are crucial components of the Application Security domain, but they are far from comprising even half of it.
Unfortunately, few people know OWASP SAMM even exists. Promoting the vastness of the appsec body of knowledge is one of the main OWASP’s goals. It makes tremendous progress in that, but there is still a lot of work.
Information or Cyber Security Audit Services
The word audit is broadly applied to represent any security review. However, its original meaning is limited to testing the design and effectiveness of internal controls. While reviewing control design could be a one-time exercise (we discuss such assessments further), testing control effectiveness implies covering a period of history.
Thus, only security controls that are continuously or regularly performed and leave proper control records can be audited. In short, you can only audit something if it runs for a while and leaves an audit trail.
audit trail
noun: audit trail; plural noun: audit trails
(Computing) A record of the changes that have been made to a database or file.
“You must be able to provide a complete audit trail of user activity across all these systems”
Definitions from Oxford Languages
The picture below visualizes the audit objectives. Compared to other types of testing, audits cover a minor scope because internal security controls are defined by a cybersecurity certification body or a best practice guidance from a respected institution. In other words, formal security controls may not cover a lot of cybersecurity risk, and an audit cannot test risk mitigation in those parts.
On the other hand, an audit requires a lot of information on the subject, and it is often in the best auditee’s interest to provide it to the auditor. However, the audit scope is always precisely defined, usually by a standard or policy. Thus, an auditee is well-positioned to choose whether to share any extra information. In other words, they shall speak the truth, only truth, and nothing but the truth, but certainly not all of it.
While the auditor has a degree of authority, they can only obtain this much information because their intervention into the audit scope is not generally welcome. Any influence auditors assume over what they review could raise conflict of interest concerns. To maintain independence and preserve the audit results, the audit team must remain distant from the assets and data in the audit scope. In other words, they can see, but they cannot touch.
The same holds about the recommendations that the auditors provide in the report. The audit outcome compares the organization’s controls and practices during a specific period versus an external standard or internal policy requirements. And as a result, an auditor can issue a compliance confirmation or a report with non-conformities.
Recommending remedial actions in the report is possible and welcome; however, these recommendations must remain at a high level of abstraction. Simply put, they shall set the goals but not define how the auditee could meet these objectives. Otherwise, detailed guidance on removing the non-conformities and improving internal security controls could compromise the auditor’s opinion.
Auditors cannot audit their own work, and their detailed recommendations could make them complacent or, more importantly, make it look so. Anyway, it could deny them the possibility to audit the same scope in the future. As you may have guessed, no auditor would like that.
Cyber Security Assessment Services
Like an audit, a security assessment uses a framework, a best practice guidance, or a cyber security standard as a benchmark for measurement. Although PCI DSS designates its inspectors as Qualified Security Assessors, cyber security assessments are usually less focused on compliance and are more cyber security risk management-oriented. As a result, security assessments can apply to a snapshot in time of the security controls and technical countermeasures. Most importantly – without the need for retrospective analysis.
A typical security assessment report not only contains security issues but is rich in remediation guidance. It is significantly different from an audit report, making the assessment a consulting engagement rather than an actual auditors’ work. Clients often expect clear and actionable recommendations from their security assessors. And unlike auditors, the assessors are in a position to provide these recommendations, advise on their efficient design, plan a roadmap of related projects, and even help implement it.
Back to the ambiguous terms: a particular type of assessment – application security assessment – reads differently to different people. Most think of it as a penetration test that focuses on a specific software application, but we discourage that. Simply because technically, application security assessment is a proper name for a cybersecurity assessment over a set of internal application security controls.
Thus, a Secure Development Lifecycle review according to the Microsoft SDL methodology or another framework could be called an appsec assessment. Using the OWASP SAMM assessment toolkit for maturity measurement of executed application security practices could be another example of appsec assessment. But for measuring the security of a particular piece of software, we prefer the term application pentest.
Cyber Security Penetration Testing Services
A cyber security Penetration Testing is a controlled simulation of different types of cyber-attacks. This exercise measures the target’s resilience to real-life cyber security threats. But unlike the organized cybercrime groups and nation-state-sponsored hackers, penetration testers have a limited amount of time and resources to do their job. Thus, readers must interpret a pentest report with these limitations in mind.
There are several ways to shape, classify, and distinguish the pentest types based on the client’s objectives, such as compliance, due diligence, insurance, real security, and so on. Independently from these objectives, pentest goals always remain the same: identify sensitive systems and applications, perform a vulnerability assessment, circumvent the security controls, and demonstrate relevant business risk.
The types of penetration tests include black-box, white-box, and grey-box depending on the knowledge pentesters have about the internal workings of networks, systems, and applications in scope. Unfortunately, these “box colors” are often misunderstood as the access level that pentesters have to the assets in scope. We address this misconception in a separate post and a dedicated webinar.
Although widely perceived as one of the sexiest cybersecurity specializations, pentesting has strict limitations in what the testers can do. There are plenty of cyber security certifications and methodologies that teach pentesting tactics, tools, and procedures, but in reality, pentesters cannot apply all of them in any given project. Sometimes because they have scope limitations that simply deny them that opportunity, but more often because of time and budget constraints.
As a result, like a real cyber threat actor who will use the shortest path to compromise the target, pentesters apply only the most efficient techniques. Unlike the attackers, pentesters will present you with a comprehensive report. Unlike the auditors, they will be detailed in their recommendations. And good pentesters will offer you a chance to have a free retest – remediation testing of all the fixes you have made after reading the report.
Red-Team Assessment Services
Finally, let us get to the term that used to be the broadest definition of offensive cybersecurity but has gradually declined into a marketing buzzword. The term red-team comes from the US army and the color, as you might have guessed, originates from the soviet flag. During the cold-war era military exercises, offense troops were designated as “red” to represent the potential threat, whereas defense teams were “colored” in blue.
Mikah Zenko colorfully depicts red-teams in military, business, and cybersecurity in his book Red Team: How to Succeed By Thinking Like the Enemy. He highlights some of his points in the DEF CON Social Engineering village talk in 2019.
Long story short: red-teaming is rather a mindset than a skillset. Everything humans know and create can serve as tools or weapons based on the user’s motivation. And red-teamers specialize in weaponizing skills and technology to challenge security decisions, design, and implementation.
Like many other military terms and concepts, red-team and blue-team took root in cybersecurity. Much more team colors exist now as a mixture of the initial ones. The team-painting is vastly overused as all types of experts get all colors of the rainbow, which confuses the borders of team capabilities. Still, the essential colors – purple for adaptive defense, white for compliance, yellow for security engineering, etc. – remain pretty helpful.
Back to red-teaming as a type of cybersecurity service. Sometimes, when marketers and salespeople say “red-teaming,” they mean just a cooler form of pentesting. It is easy to disguise one service as another because their toolkits, techniques, and operational procedures overlap. But in fact, penetration testing and red-team assessments have a lot of differences.
One of these differences is that red-teamers often have more extended periods available for attack modeling and threat simulation. They extend their tools and procedures to cover more phases of the cyber-attack kill chain. Thus, they focus on finding, validating, and using the vulnerabilities in systems and applications and maintaining continued access to compromised infrastructure, attempts to exfiltrate sensitive data, etc.
The other difference is the lack of strictly defined scope. Red-teaming makes sense when there is a blue team on the other side. The effective red-teaming assessment or red-teaming program scope is the whole company and all its external integrations, partner relationships, supply chain, etc. As a result, the outcome of a productive red teaming is not the report on their findings but rather the improvement they stimulate in the organization’s cyber security business function.
Conclusion
Having a good sense of controls efficiency is the most important aspect of cyber security. To ensure your security controls work, you need to test their design and effectiveness. Types of security testing vary, and none of them can fully answer whether an organization or system is completely secure. However, each has its beneficial outcomes and contributes to the specific security objectives.
Cybersecurity audit assures the organization’s compliance with a security standard, framework, or regulation. Cyber security assessment validates if the organization has robust information security controls arranged in a coherent Information Security Management System. Penetration testing demonstrates how the business could withstand potential cyber-attacks and indicates the gaps in the organization’s cybersecurity. Red-teaming exercises continuously teach the blue team various types of cyber-attacks and how to recognize them.
Security of the organization’s infrastructure, software, and customers and compliance with external requirements and regulations are all valid security objectives. Each security testing technique has its area of application and valuable results. To better understand how they can help you improve your security posture, we have outlined their fundamental properties in the table below.
| Timeframe vs. Interaction | Direct access to scope and evidence | Indirect data and evidence are requested |
| Finite ad-hoc or regular, a snapshot in time | Pentest – measures scope security at a given moment – testers get data and evidence directly – the report provides actionable recommendations | Assessment – measures scope security at a given moment – testers request, observe, and process input data and evidence – the report provides actionable recommendations |
| Continuous over a period or as an ongoing program | Red-teaming – measures scope resilience to long-term threats – testers directly interact with the assets in scope and the blue team – reporting may be on-the-go without summary documentation | Audit – measures control design and effectiveness over time – testers request, observe, and process input data and evidence – the report sets objectives but omits actionable recommendations |
We hope this post will help you better navigate the security services your organization needs or could benefit from, and we at BSG are here to help. Stay safe and take care.