Grey is the new Black Webinar

White-box vs Black-box Penetration Testing

What is a black-box penetration test, and why do many people get it wrong? What is a white-box pentest, and how is it different from a black-box? What is gray-box testing: something in between black-box and white-box or a combination of two? I bet you do not have correct answers to these questions, but do not worry: no one does.

Why do we even have a question?

It is baffling to see how many cybersecurity professionals misinterpret the terms “black-box”, “white-box”, and “grey-box” when it comes to network penetration testing. Some unintentionally, caught in the chaos created by the cybersecurity firms copywriters. Some deliberately, in attempts to delude potential customers and shape the market the way they want it to be.

The disarray in the cyber security penetration testing services market caused by inaccurate assertions about the white-box, black-box, and grey-box pentesting is the reason for recording this webinar and putting it out on YouTube.

The wrong way

Well, what is wrong with the perception of black-box, white-box, and grey-box pentesting? It is difficult to pinpoint any particular misconception, as it is usually inaccurate on a very basic level of interpretation. Many people, the laymen, and cybersecurity pentesting experts alike, genuinely believe that:

  • in black-box testing, pentesters have limited permissions and access only to the functions publicly available “to the world”
  • in white-box testing, pentesters have unlimited access to the functions of and the information about the systems they evaluate
  • grey-box testing is something in between: pentesters have user credentials and can access some functionality (although admin interfaces often remain out of reach), but do not have access to configuration or source code

In fact, even on the Wikipedia-level of understanding, this kind of thinking is profoundly incorrect, severely impaired, and causes a lot of harm in the industry. In particular, the black-box pentest misconception diminishes the external pentesting exercises to mere vulnerability scanning with optional exploitation of potential vulnerabilities. But before we discuss its alarming consequences, let me share what the types of cybersecurity penetration testing actually are.

The right way

Now, let me clarify the correct definitions of black-box, white-box, and grey-box security testing methods. But first, we need to fix the major errors people make when interpreting these terms. Many think of them as the flavors of security testing services that limit pentesters in their initial access to the system, network, or application, and this is fundamentally wrong.

The correct interpretation of the “box color” is not the access rights pentesters get but the knowledge they have. See, black-box means pentesters literally know nothing about how the system works, but at the same time, they have access to all its functionality and interfaces.

White-box, on the contrary, means that pentesters have all available knowledge about the system, including documentation and application source code. However, white-box does not mean the testers have direct access to the system in a production, staging, or testing environment.

And what is the grey box? Unlike many people believe, grey-box pentesting is not a point somewhere in between zero access and full access to the system. Many mistakingly think that in this case, the security testers have some but not full access to application functionality, but in fact, modern penetration testing methodologies do not state or imply anything like that.

The box color has nothing to do with access or permissions; it is about knowledge. So maybe the grey box means that pentesters have some limited knowledge about the system? Again, wrong. Contrary to popular belief, and I bet you may be surprised as well, grey-box pentesting is not a balance between black-box and white-box; it is a mixture of the two.

In grey-box pentesting, we do not have limited knowledge, limited access, or any limitation at all. The grey-box regime of pentesting services means that pentesters simultaneously have certain access to system interfaces, application functionality, and certain information about its internal workings, including the documentation and source code.

Why so?

So why there is a misunderstanding after all? It seems to be pretty straightforward if you read it right. Why do we have a problem?

My theory is that this misinterpretation of basic terminology originates from the false intuitions associated with these greyscale semantics. And of course, the cyber security testing companies that prefer easy penetration testing projects, heavily rely on automated penetration testing, and benefit from the inferior level of customer awareness in the market.

Another explanation can be confusion between the penetration testing methods and vectors. This misinterpretation of box colors as access levels seems to be very similar to the permissions penetration testers normally have during external vs. internal penetration testing projects.

External pentesting implies that cybersecurity experts have no access to the systems and applications in scope and, in the ancient pre-zero-trust terms, are positioned “behind the perimeter”. While internal pentest assumes that penetration testers have certain initial access, such as a new employee’s user profile and VPN connection, at the beginning of the engagement. Is it possible that we are disoriented by black-box and white-box pentesting vs. external and internal pentesting? Probably so.

The consequences

OK, people put their meaning into obscure cybersecurity terminology that only a few hundred thousand people use anyway; why is it a big deal? There are at least three types of circumstances where this poor understanding of essential definitions could cause some damage.

The first threat is that cybersecurity standards, regulations, and “best practices” have precise meanings behind these terms. If a standard or guideline does not redefine the terminology, its requirements and recommendations convey very specific objectives and activities. And in their wording, standards are much more comparable with the formal definitions than with what we fantasize about.

In other words, if a standard, such as SOC2, ISO 27001, or PCI DSS, demands to have an annual third-party black-box external penetration testing of infrastructure and business applications, it means that an independent penetration testing company must be given access to all appropriate systems and functions for the penetration test conclusions to qualify for compliance and the pentest report to be accepted by the auditor.

In reality, the misinterpretation of the term results in pentesters starting with as little as a general attack direction. They often still can penetrate the system or network, which says much about the level of its cybersecurity, but they rarely are as productive during the pentest as the creators of the standard meant them to be. Fortunately, the auditors who evaluate the penetration testing report have no idea about these terms either, which usually saves the day.

The second threat is that the misinterpretation of language shifts the organizations’ security objectives, and instead of making their infrastructure elements harder to compromise, companies focus on introducing more secrecy and obscurity into the equation.

The problem with obscurity is that it cannot protect you long enough. And when the system’s internal workings become known to the attacker (and they inevitably do), the system will be easily hacked. Simply because its creators relied too much on the secretiveness of its design and implementation and did not leverage white-box security testing in the SDLC.

And the third threat that impacts BSG directly is the wrong perception of a fair security pentesting price in the market. The misconception of a black-box pentest creates a false sense of the brevity and straightforwardness of this exercise.

Uncomfortable for many, black-box pentesting is not just brute-forcing the login form and scanning the network for known vulnerabilities. It is a comprehensive analysis of all possible ways to provide input to the system, and then use those entry points to find, diagnose, validate, and contain security flaws.

Apparently, this kind of work needs much more skills, time, and energy, than running commercial penetration testing software. Moreover, numerous regulations mandate manual penetration testing and require it to cover both internal and external vectors, and incorporate elements of social engineering.

On the other hand, white-box and grey-box pentests have the aura of immense, lengthy, and cumbersome exercises. They presumably require the pentesters to go through all the available functionality and read the application source code in its entirety to complete the testing. But in fact, it is far from the truth, because all available functionality must be tested in a black-box pentest as well, and access to the source code only accelerates the process and shrinks the time needed to complete the job.

You see, given access to the source code and other information about the system’s internal workings, pentesters can be much more efficient as they do not need to spend a lot of time to diagnose each vulnerability. Instead, they can refer to the source code and directly see how software developers implement the functionality they are testing at the moment. It saves us tons of effort and a lot of time, allowing us to complete a day’s work within an hour.

Thus, the webinar

So this is why we have decided to record a video where we go through the consequential pentesting method misconceptions and debunk them one by one. We hope you enjoy this video and learn from it. And most importantly, we hope you will share it and pass the word on.

Thank you to all of you who have attended our webinar! It was an excellent experience for the hosts and a fruitful discussion for everybody. We have covered a lot and most importantly: we have put it on record, so now we can direct the interested audience to the YouTube video below. The slides are also available at this link.

Webinar video

What next?

The penetration testing efficiency debate is not over yet, but we believe we have made a good contribution, dissolved a few misconceptions, and clarified a few uncertainties. We will continue to learn from our professional experience, and if our views change, we will update you in a future webinar. So stay tuned, subscribe to this blog our social media accounts, and to our YouTube channel.

Leave a Comment