Distributed Test Projects in Jenkins

If it’s a feature, your pull request had better have tests.

That’s our mantra here at Behance. We’ve found that this philosophy is probably one of the most reliable ways to enforce a strong and stable growth in our applications. But what happens when this is taken seriously in a team of ten developers? Twenty? More? Every week our team juggles multiple features, improvements and hot fixes, usually between multiple applications, so new tests are constantly making their way into builds. Our team is also constantly growing so the number of tests that make their way into master on a daily basis keeps on increasing. This became a monster of a problem fast, because our test build times started to increase rapidly and our continuous integration (CI) server became overburdened with queued tests that needed to be run.

Read more →

Custom PHPUnit Annotations

Here’s a trick that I just implemented on our own testing framework to handle custom annotations. Annotations are comments written in a standard format that are used to declare properties for tests, such as dependencies and grouping. You can read more about the standard PHPUnit annotations here: http://www.phpunit.de/manual/current/en/appendixes.annotations.html

There are only a handful of annotations available natively to the PHPUnit framework. I’ve written some code that allows you to create additional rules you might need, and trigger them with your own custom annotations. You’ll need to instantiate a ReflectionMethod and then parse the doc block for the proper data.

In the sample code provided below, I look for the annotation “@browser” to determine what type of browser to launch (Firefox, Chrome, Internet Explorer, etc.) If the browser is not found, then setUp() does not launch a browser. I create a user through the web app using Selenium, and then test that user with curl requests against our API. The curl requests did not require a Selenium browser, which saved about 10 seconds of setup per test.

Read more →

Testers Wanted, Experience Required

Testing is in a strange state right now. Both large corporations and small start-ups absolutely require testers. The problem is what kind of tester.

If you fall into the trap of thinking that anybody can be a tester, you’ll see the impact in your choices. Similar outcomes would occur if you had the same thoughts about developers. You wouldn’t hire a developer without any experience, why would you do the same for a tester? (The exception to the rule are fresh upstarts who are looking to get their feet wet in the programming world.) Testing is not trivial and should not be treated as such. In this day and age, testers need to know how to program.

Read more →

JenkinsConf NYC 2012

I spent time yesterday with Kohsuke Kawaguchi (@kohsukekawa) and many other Jenkins CI enthusiasts in NYC, discussing the state of Jenkins and observing how different teams are utilizing the tool. For those not in the know, Jenkins is a continuous integration software that allows you to handle build deployments and run tests against them. Seeing the other dev shops and the way they use Jenkins to the extent they do, it really still blows my mind. It almost feels like magic. I wanted to share a few gems of knowledge that I gleaned throughout the day.

Read more →

Testing Simplified: Page Objects

Testing is hard. Maintaining tests is even harder.

When we first started using Selenium WebDriver, writing a test meant creating a one-off function that performed the given task, but duplicated a bunch of code, and used hard-coded elements if any HTML/CSS got involved. Trying to get coverage on something as large as the Behance Network (be.net) became bloated and unmaintainable.

Without an easy way to create and improve tests as features changed, QA + testing became the last priority in development.

Read more →