1. Using Previous Bugs for Evaluation

A solid starting point is to leverage past security bugs that have been found through assessments or penetration tests in your own organization. Ask your interviewees to find these bugs in source code and explain how they would fix them. This approach provides a realistic, hands-on test that directly relates to the work they’ll be doing.

However, there are a few things to keep in mind when using this method:

Rewriting Issues for Interviews: Your development team may need to rewrite the bug reports into short, self-contained snippets to make them more "interview-friendly". The key is to avoid complex examples that require extensive context to understand. The candidates should be able to quickly grasp the problem from the code itself without needing a full backstory.

Multiple Languages: If your development team works across multiple programming languages, it’s helpful to prepare bug examples in each relevant language. By mapping the language to the candidate’s resume, you ensure that the assessment aligns with their claimed expertise. For example, if they’ve listed Java or Python in their resume, present bugs in those languages to evaluate their specific knowledge.

Diverse Interview Panel: Ideally, the interview process should involve both security and developers. The security team will be able to evaluate how well the candidate understands vulnerabilities and how to mitigate them, while the developers can judge the technical depth of their proposed solutions.

2. Avoiding Classic Development Interviews

One mistake often made during interviews for security roles is using the same development-focused interview questions, which may not adequately assess a candidate’s security skills. For example, during one interview, I was asked to code a doubly linked list manager on a whiteboard; a task far removed from the skills needed for an AppSec role. While I still got the job, the exercise didn’t provide a meaningful assessment of my security expertise.

For AppSec candidates, focus on practical security problems, not generic coding exercises. It’s critical that the interview questions are tailored to the specific demands of an application security engineer role.

3. Hands-on Labs and Real-World Applications

To further assess a candidate’s practical skills, hands-on labs are an excellent tool. There are various open-source projects and platforms available that can simulate real-world vulnerabilities. For example:

• WebGoat and DVWA (Damn Vulnerable Web Application) are popular for testing web security knowledge.
• Damn Vulnerable GraphQL Application is a great resource if your organization uses GraphQL extensively.

Additionally, PentesterLab offers specialized "interview labs" for enterprise customers, which are designed to simulate the kinds of challenges a candidate might face on the job. If you use labs as part of the interview, make sure to observe how the candidate works. Request that they share their screen or perform the tasks in front of you. This allows you to evaluate their problem-solving skills, habits, and overall velocity, which can reveal a great deal about their competency.

4. Stack-Specific Questions

Another approach is to ask the candidate about securing your current technology stack. For example, you can ask about common security pitfalls and security recommendations related to technologies you use, such as OAuth2, SAML, JWT, Java, or other relevant frameworks. These questions give you insight into how well the candidate understands your specific environment and whether they can identify potential weaknesses and provide proper recommendations on how to fix them as well as how to harden your technical stacks.

5. Bringing in an External Expert

If your team lacks deep security expertise, or if you want an additional layer of evaluation, consider hiring an external AppSec expert or penetration tester for a day. They can help interview several candidates and provide a professional opinion on their abilities. This person could be a contractor, an external consultant already familiar with your organization, or simply someone you trust in the industry.

Though this option requires a small financial investment, the benefit of gaining expert feedback on three to four candidates makes it well worth the cost. Hiring the right AppSec engineer is critical, and having an expert assess them can make the difference between a good hire and a great one.

Conclusion

Assessing the technical abilities of an AppSec engineer requires a tailored, hands-on approach. By utilizing previous bugs, interactive labs, stack-specific questions, and external expertise, you can gain a comprehensive understanding of a candidate’s skills. This approach ensures you’re hiring someone who can effectively identify and fix security vulnerabilities, rather than someone who excels only in theoretical scenarios. In the long run, these strategies will help you build a more secure and resilient development environment.