Sebastian Rode of djangsters¹ recently wrote about his perspective on pentests as a developer. This article takes the reverse perspective and tries to answer how a pentester experiences a pentest.

First of all: As a penetration tester, I do this job to improve the security of applications, organizations and, most importantly, users. Making life difficult for developers or even putting them on the spot for their work is never my goal. However, I know that a pentest, and especially a possible rework of the application after the test, can mean a lot of extra work. That’s why I want to share my views on the process and hopefully give some insights along the way that might help make the next test go as smoothly as possible.

The plan

During a pentest there are several parties involved, each with their own perspective on the process: for the customer the pentest is often a mandatory quality gate before release and part of a larger process, for the pentester it is one of many projects that happens to be next on his schedule. This is a common source of problems.

The challenge – and this is by no means unique to pentesting – is to coordinate the different modes of operation so that the pentest can be completed and any resulting issues resolved before it gets in the way of a release or another deadline.

We need to arrange a meeting with internal IT security (often represented by an Information Security Officer (ISO)) to discuss the general plan and requirements. Then we need to find a time slot when the application is ready for release, but that still gives the developers enough time to implement fixes for the pentest issues before release. And of course we need to be available ourselves during that window. At Lutra Security we like to plan projects with overlap and a bit of margin for error to be able to react to unforeseen changes in the schedule, but sometimes this is not possible and a project has to be rescheduled and this can mean that the test can only start a few weeks later.

When all this is done, our plan is quite simple:

The pentest starts with a kick-off, where we resolve any outstanding issues and check that nothing is missing (documentation, credentials). And then the test begins.

The test

When we do a pentest, we take the documentation we get from our customer and sit down for a couple of days or weeks and try to break as much stuff as we can. I have to admit that it can be fun to see an application crumble, but that is not the goal of what we are doing. We want to find vulnerabilities in the application that a real-life attacker would also be able to find, and while our expertise and tools help with that, there is little point in spending our time looking for things that the developers or admins already know.

So if you’re a developer and you know your application has some problems you haven’t found the time to fix, give us a call. We’ll appreciate the help and won’t use it against you. With the knowledge of what you already know, we can try to give you a fresh perspective on the problem, and we can use the time you’ve saved us to focus on the problems you aren’t already aware of.

The report

The report is the end result and arguably the most important part² of the pentest. It can be an automatically generated scan report or a mostly manually written document, but it should always contain the following elements:

1. An executive summary
2. A list of findings with
   • a risk rating
   • an explanation of the problem
   • suggestions on how to fix it

(If any of this is missing or lacking, feel free to complain to your penetration tester; the report is what you paid for).

The report serves multiple purposes:

1. It documents what we did (our customer paid a lot of money and has no other way of judging whether we did anything at all).
2. It gives the product owner a sense of how secure the application is (would it be irresponsible to release it now?).
3. It helps the developers to fix the problems we have identified.

A common source of irritation are the risk ratings. I have often been in a situation where the developers of an application have disagreed with my assessment. And that is OK!

When I document a vulnerability, I have to consider how difficult it is to exploit and what the impact would be. If I have developed a proof of concept exploit, I have a pretty good understanding of what it takes to exploit the vulnerability, especially if I developed a proof of concept. The impact is usually harder to assess because my understanding of the underlying business processes may be incomplete. Therefore, I try not to include it in my rating: As long as I can’t rule out any potential escalation, remote code execution has a critical impact regardless of whether the server contains the data of other applications or is an empty VM isolated from the rest of the network.

I don’t take it personally if the client comes to a different result than I do. To some extent I expect them to. I will try to rate each finding as accurately as possible, but I will usually choose the upper end of the range of uncertainty, as the customer will be happy to lower the rating, but will almost never suggest a higher rating than I initially gave.

However, please do not try to bargain with me over the rating. If I haven’t overlooked an important aspect (which can happen), I will stand by my rating. You are free to rate any of the findings differently, but for the sake of transparency (and my own credibility) I will not remove my original rating from the document.

Penetration testing is less about eliminating risks than it is about understanding them. And to do that we need to talk about them and even disagree.

General tips

As penetration testers, we have seen a lot of different applications with all kinds of authorization concepts and are familiar with different formats of API documentation. However, we have probably never seen your application before and are not familiar with the way you do things. We will have questions, and I will ask them if I get stuck. So making sure the documentation is up to date, the test suite is working and the test environment is up and running will definitely save both of us time.

Expect results. I’ve done hundreds of pentests, and while there are applications that don’t have a lot of problems, I’ve never found none. While many problems can be fixed quickly, it is not realistic to release software directly after a pentest.

Prepare for the test. Logging might help understand issues better, backups will help to restore the environment if something breaks and to get rid of any corrupted data payloads after the test. And there is no shame in doing a quick cleanup before the test. It’s ok to update your server and delete the productive customer data from the test environment³ before the test starts. You already know that, so we don’t need to tell you. Just keep it that way after the test.

Conclusion

Even if it’s sometimes easy to forget, developers and pentesters are on the same side. We both want to create great software, that improves the life of its users, while at the same moment keep them secure. The best way to achieve this is to work together and try to use our strengths to help each other. There is no shame in making mistakes, only in refusing to learn from them. And just because pentesters can point out a problem doesn’t mean we can do it better. Only together can we build secure software.