Warning, this post is a bit technical.  🙂

[Editors Notes] From time to time our engineers take a moment to write a post for our blog. From Adam our product manager writing about Scrum and Kaizen to Piotr writing about our Engineering Best Practices there is lots to read.  Vince has written on the blog before, his last post was about Adventure Engineering in Da Nang, Vietnam. Today, Vince thought that what he has learned about reducing our test runtimes might be helpful to startup community. Enjoy.

One of my biggest challenges for the past few weeks was to get our tests to complete within 10 minutes.

That doesn’t sound so bad, does it? There’s just one tiny detail which would make this journey all the more exciting: Our code takes 25 minutes on average to complete on 25 parallel build servers and It takes about 4 hours if you run it in one.

I made a quick survey with our engineers and we lose roughly around 3 hours per day waiting for CI to finish.

The plan was to selectively run tests and have cucumbers run only on specific branches — epic, develop and master. This means that every time something gets merged to those branches, cucumbers will run after rspecs. And of course, all this will run in parallel on 25 build servers.

How our code tree looks like:

|- master
|- develop
   |- epic/adding-a-thing-with-stuff
      |- feature/with-things  
      |- feature/with-stuff
   |- epic/the-thing-you-do
      |- feature/bla-bla-bla

We didn’t have a way to execute this plan with our current CI solution at that time so we had to find alternatives. We tried quite a few but most of them were either had limited features or were just simply too hard to get started.

Enter Codeship. It took me no more than 3 minutes to get my first build up and running…and of course, failing. Getting started with their ParallelCI wasn’t that hard either. How you configure your builds opened up a ton of possibilities including selective test runs.

To make our tests parallelize (is that even a word?) I had to use this neat little gem called parallel_tests.

Getting started with the gem is simple. You can execute a group of rspecs like so:

bundle exec parallel_test spec/ —verbose -n $TEST_GROUPS —only-group $TEST_GROUP —group-by $TEST_GROUP_BY —type rspec

$TEST_GROUP is an ENV variable that I set on each pipeline. Basically just the number of the pipeline. $TEST_GROUPS is the total number of parallel pipelines. This is 25 for us. $TEST_GROUP_BY is how we group our tests and that would beruntime for us.

That is one other thing you can do with parallel_tests is group by runtime information. Using this grouping took 2 minutes off of our test runtimes. Getting logs from parallel pipelines was a little bit tricky so I had to do a little scripting magic to gather logs from multiple pipelines and concatenating them back together.

It’s simple really, in the setup section I have two scripts: one to upload each log slice to S3 and the other to download the logs and put them together.

Links for reference, apologies for the terrible naming: – Concatenate – Upload

Now that the runtime logs are covered, it’s time to make the tests work. I have this bash script in our code:

#!/bin/bash

bundle exec parallel_test spec/ —verbose -n $TEST_GROUPS —only-group $TEST_GROUP —group-by $TEST_GROUP_BY —type rspec

if [[ “$CI_BRANCH” =~ .*”$BRANCH_FOR_CUKES”.* ]] || [[ “$CI_BRANCH” =~ .*”master”.* ]] || [[ “$CI_BRANCH” =~ .*”develop”.* ]]
then
  echo “Running Cucumber…”
  bundle exec parallel_cucumber features/ —verbose -n $TEST_GROUPS —only-group $TEST_GROUP —group-by $TEST_GROUP_BY —type cucumber || true
else
  echo “Not running Cucumber”
fi

Conveniently, Codeship has a bunch of ENV variables set inside test instances and one of those is $CI_BRANCH. I just simply check if the current branch running in CI matches either master, develop or what I configured in the settings, it will run Cucumbers after RSpecs.

Finally, this is what our setup and pipeline configuration looks like:

This entire approach got our RSpec builds down to an average of 6 minutes and our entire suite to 13 minutes from 25 minutes.

Huge props to the Codeship team for helping out with the transition and thanks for the swag! 🙂

Editors note: This post was originally published here.