If you’ve talked with me during the last year, you noticed how in love I’m with continuous integration, test driven development and all related with increasing the development speed of a team without compromising quality.
Ten months ago I decided it was time to run w3af’s unit-test suite in a CI system, but due to other more important projects the task was delayed. Finally a couple of weeks ago I started to work full-time on it, and today a huge milestone was achieved:
-
"1259 out of 1641 tests run"and PASS in circleci.com - The build is stable (running the same build N times will always yield the same result)
- The build runs in less than 5 minutes
But how does this help the w3af project? The short answer is: no regressions. Which means that what’s working (and tested) today, will work tomorrow. This allows us to make changes at a much faster velocity knowing that we did not break an existing feature.
The CI system we’re using is CircleCI, which has many awesome features plus the best customer support I’ve ever seen. This is how our build page looks today:
It’s not a secret that there is still pending work related with our test suite, as you can see from the first bullet there are 1259 tests out of 1641 which PASS. This doesn’t mean that there are 382 failing tests! That would be catastrophic! It simply means that not all tests were written to meet the CI system requirements.
Remember the moth VM? Well, I rewrote it as a django application in order to run it as a process in the CI system and have integration tests run against it! This allows us to easily run our test suite in both the CI system and developer workstations.
The future is clear, I’ll write more vulnerable scripts in the new django-moth application and change w3af’s tests to point to it. That should hopefully decrease the number of tests which are not run to zero, which leads to higher code coverage and less bugs.
As usual, all the source code is open source
- Various wrappers and utils for automating test setup and run at CircleCI
- circle.yml, the main configuration for the CI system
- One of our builds at CircleCI – need to login using your GitHub credentials to see the action
If you want to help, feel free to contact me at @w3af!
ShareDEC
About the Author:
Web Application Security Innovator, Researcher and Entrepreneur. Python Hacker.