When we released SUSE Manager 1.2 back in 2011, one of the first internal components we open-sourced was our internal testsuite and I have written a bit about it before. Our process was from the beginning completely automated to the extreme. If you do a git commit, you can expect the installable .iso file with the bootable SUSE Manager appliance to appear in a folder after some time. Every couple of hours, this .iso is auto-installed in a reference server and reference client servers, which we use to quickly checkout “things”, and additionally, it is installed on a server, where a list of hundred of tests is performed using a human description of the features, and executed in a real Web Browser.
We could not do this without a bunch of great technologies and products: git and github, Jenkins, where we pull the git repositories and build tarballs + rpm spec files. The Open Build Service, where we submit all the tarballs + rpm .spec files and they get built together as a project and later thanks to the KIWI technology inside the Build Service, turned into an appliance. KVM and libvirt, which we use to run the latest version of our product, and then of course Cucumber, which allows us to describe features in a high-level language, and then implement it to be run on Firefox using WebDriver. But recently we have added another set of features to the testsuite: Security regressions. It was Victor, one of our security engineers, who approached us to talk about using OWASP ZAP to find vulnerabilities and harden the product (his presentation). The project describes itself as:
The Zed Attack Proxy (ZAP) is an easy-to-use, integrated penetration-testing tool. It locates vulnerabilities in web applications, and helps you build secure apps. Designed for use by people with a wide range of security experience, it’s also suited for developers and functional testers who are new to penetration testing. With its automated scanner and powerful REST API, ZAP fits seamlessly into your continuous integration environment, allowing you to automate the finding of common issues while you’re still in development.
As soon as he explained how it worked: the passive tests used a proxy, the pieces of the puzzle clicked: we could run the whole testsuite through the proxy, and add the results analysis as part of one of the last steps of the tests. Those tools produce lot of false positives, so we could keep a “baseline” or “ignore list” and still get the testsuite to fail when new vulnerabilities that we haven’t analysed pop up. Integrating it was straightforward. We packaged the ZAP suite as an rpm that gets installed in the machine that is deployed to run the testsuite against the product. The testsuite code is mostly ruby, so we used the owasp_zap gem for which Victor himself is the author. The testsuite starts the proxy, configures Firefox to go through it and run all the tests. There is an optional step before retrieving the results of performing an active attack on the host (eg. finding SQL injections). The last test retrieves the results, compares against the baseline and then embeds the report in the feature steps, which is considered failed or passed whether new vulnerabilities are found. Martin found a way to organize the results better: we normalize some POST parameters that change all the time so that they don’t result in separate incidents and we also group (using the URL path) the incidents per application component.
We have just started with this: The initials runs immediately showed the need to harden our apache and tomcat configurations but we still have lot of information to analyse and parameters to tune. For example, our developers have found problems in unreleased code commits which were pointed out and fixed immediately, but it would be nice if we could tune the scanner so that those real-life scenarios are detected by the tool, in case a similar mistake is repeated.
