Faster CI – Our Journey To Halving Our Test Runtimes

Warning, this post is a bit technical.  🙂

[Editors Notes] From time to time our engineers take a moment to write a post for our blog. From Adam our product manager writing about Scrum and Kaizen to Piotr writing about our Engineering Best Practices there is lots to read.  Vince has written on the blog before, his last post was about Adventure Engineering in Da Nang, Vietnam. Today, Vince thought that what he has learned about reducing our test runtimes might be helpful to startup community. Enjoy.

One of my biggest challenges for the past few weeks was to get our tests to complete within 10 minutes.

That doesn’t sound so bad, does it? There’s just one tiny detail which would make this journey all the more exciting: Our code takes 25 minutes on average to complete on 25 parallel build servers and It takes about 4 hours if you run it in one.

I made a quick survey with our engineers and we lose roughly around 3 hours per day waiting for CI to finish.

The plan was to selectively run tests and have cucumbers run only on specific branches — epic, develop and master. This means that every time something gets merged to those branches, cucumbers will run after rspecs. And of course, all this will run in parallel on 25 build servers.

How our code tree looks like:

|- master
|- develop
   |- epic/adding-a-thing-with-stuff
      |- feature/with-things  
      |- feature/with-stuff
   |- epic/the-thing-you-do
      |- feature/bla-bla-bla

We didn’t have a way to execute this plan with our current CI solution at that time so we had to find alternatives. We tried quite a few but most of them were either had limited features or were just simply too hard to get started.

Enter Codeship. It took me no more than 3 minutes to get my first build up and running
and of course, failing. Getting started with their ParallelCI wasn’t that hard either. How you configure your builds opened up a ton of possibilities including selective test runs.

To make our tests parallelize (is that even a word?) I had to use this neat little gem called parallel_tests.

Getting started with the gem is simple. You can execute a group of rspecs like so:

bundle exec parallel_test spec/ —verbose -n $TEST_GROUPS —only-group $TEST_GROUP —group-by $TEST_GROUP_BY —type rspec

$TEST_GROUP is an ENV variable that I set on each pipeline. Basically just the number of the pipeline. $TEST_GROUPS is the total number of parallel pipelines. This is 25 for us. $TEST_GROUP_BY is how we group our tests and that would beruntime for us.

That is one other thing you can do with parallel_tests is group by runtime information. Using this grouping took 2 minutes off of our test runtimes. Getting logs from parallel pipelines was a little bit tricky so I had to do a little scripting magic to gather logs from multiple pipelines and concatenating them back together.

It’s simple really, in the setup section I have two scripts: one to upload each log slice to S3 and the other to download the logs and put them together.

Links for reference, apologies for the terrible naming: – ConcatenateUpload

Now that the runtime logs are covered, it’s time to make the tests work. I have this bash script in our code:

#!/bin/bash

bundle exec parallel_test spec/ —verbose -n $TEST_GROUPS —only-group $TEST_GROUP —group-by $TEST_GROUP_BY —type rspec

if [[ “$CI_BRANCH” =~ .*”$BRANCH_FOR_CUKES”.* ]] || [[ “$CI_BRANCH” =~ .*”master”.* ]] || [[ “$CI_BRANCH” =~ .*”develop”.* ]]
then
  echo “Running Cucumber
”
  bundle exec parallel_cucumber features/ —verbose -n $TEST_GROUPS —only-group $TEST_GROUP —group-by $TEST_GROUP_BY —type cucumber || true
else
  echo “Not running Cucumber”
fi

Conveniently, Codeship has a bunch of ENV variables set inside test instances and one of those is $CI_BRANCH. I just simply check if the current branch running in CI matches either master, develop or what I configured in the settings, it will run Cucumbers after RSpecs.

Finally, this is what our setup and pipeline configuration looks like:

codeship vince paca

This entire approach got our RSpec builds down to an average of 6 minutes and our entire suite to 13 minutes from 25 minutes.

Huge props to the Codeship team for helping out with the transition and thanks for the swag! 🙂

Editors note: This post was originally published here.

The End of a Legendary Internship

My ten-week internship with PayrollHero reached its end last week. As a fitting closure to my first post on hiring interns in Singapore, I thought it would be a good idea to write about my experience here. I’ve had an amazing summer at PayrollHero and learned more than I thought I would. Ten weeks flew by and it took a few days to organize my thoughts. Here is what I learned from my internship at a startup in Singapore.

PayrollHero Pragya Last DayLesson 1: Learning is built in to PayrollHero’s DNA

The company doesn’t just declare its “Ridiculously Client Focused” psyche on the website, it lives and breathes the idea. To achieve that end, everyone is always learning new things through whichever means that they can. But it doesn’t end at just learning something by yourself, it’s all about sharing it with everyone else. When I joined the communication channel, Slack, the most interesting channel to me was the reading-list. A mandatory channel for all members, the reading-list includes articles that are relevant to the team from all kinds of sources. Anyone can share something that they have read, and anyone can comment on it.

One of the first things I did at PayrollHero was to broaden my reading base. As an economics student, the obvious reading material I blindly follow are opinion blogs by economists, or economics journals, and of course, the newspaper. But within two weeks, I had apps on my phone that gave me news about startups, tech blogs and more. It was awesome to see that everyone was invested in improving themselves and the team. It was also interesting to see what everyone was reading. Rarely does a college kid get the chance to find out what seasoned entrepreneurs are reading. And here I was, discussing the very articles with the founders of PayrollHero.

Lesson 2: You’re an intern, and you’re still taken seriously

Trust me, this one was huge. While I was interning at PayrollHero, I had friends interning at big accounting firms, consultancies and banks. I would often hear stories about their internships where all they did was photocopy documents for their bosses. This was not the case across the board, but I heard it often enough to realize that my internship was unique.

During the ten weeks, I never once had the impression that I was just an intern slaving away at something inconsequential. If it ever seemed like I was not doing enough, I could speak up about it and get more work to do. The great thing about working in a small place is that there is always something to do and the little that you contribute has a visible impact on the company. Furthermore, Steve ensured that the communication lines were open and that an intern in the Business Development team was equally important (and accountable) as any other member on the BD team. The feeling of being a relevant member of a team is definitely something that I will take with me from this internship.

Lesson 3: Diversity

When I read about PayrollHero before sending in my resume, I loved the idea of Adventure Engineering and knew that PayrollHero was based in the Philippines, Singapore and Canada. To me, that seemed to reflect diversity well enough until I met the team. If you hang out with the Singapore team, you will find yourself in a room with around 10 people representing at least 7 different nations. That is incredible and speaks volumes to the success of PayrollHero’s drive to attract talent, no matter where it comes from.

There are many studies that show how diversity improves results in a company. Diversity was a crucial aspect in my experience. Despite being an international student studying in a foreign country, I had never seen so many nationalities from across the world working together before I showed up at PayrollHero. Imagine this, a Canadian, Ukrainian, Polish, Indian and Filipino sitting in a hawker center, eating a plate of prata and talking about the latest episode of Game of Thrones. As ridiculous as that sounds, it’s just another regular day at PayrollHero. You can’t help but appreciate how easily PayrollHero has embodied the concept of diversity in a company.

Lesson 4: Hanging out with the team over beers

If I look back over the ten weeks I was at PayrollHero, some of the most memorable interactions with the team was not during work, but outside it, when everyone hung out with some beers after a long day of work. I learned the most about the people who make up PayrollHero during this time. Whether it was about their professional lives or just a casual conversation about what everyone’s “ah-ha!” moment was during the DDD workshop, these interactions helped me understand people in a world I had little or no idea about.

I think this is especially invaluable to interns because a large part of interning is to try and figure out what to do with our lives professionally. While colleges make an effort to conduct networking sessions with professionals from every field, it does not come anywhere close to actually spending time with the same people day in and day out.

Finally, all these elements together made for an internship that immensely broadened my perspective. When I first walked into PayrollHero, I wondered if an unstructured internship would teach me anything new or if the work would be rewarding. But the truth is, the very fact that it was an unstructured program made me want to push myself to do new things and be open to ideas that I wasn’t exposed to before. At the end of it, I came away having met inspiring people from all over the world, learning about the startup culture and learning more about myself through it all. I hope the little work I did at PayrollHero was useful to the company. I also greatly appreciate the time invested by Steve and Mike and everyone else to make my internship worth it. At the risk of sounding super cheesy right now: good luck, PayrollHero, and may the force be with you!

Editors Note: Thanks Pragya!! While Pragya is off on her next adventure you will still see her posting on the PayrollHero blog a few times a month.  

Executive Interview: Russell Yu, IKI Concepts

I had the opportunity to meet up with Russell Yu, the Director of IKI Concepts in Singapore. Russell is in the process of bringing his successful restaurant concepts from Singapore to the Philippines.

IKI ConceptsIn Singapore, Russell operates Kaiseki Yoshiyuki, Horse’s Mouth Bar and Uma Uma! Original Hakata Ramen. They are very popular spots on Orchard Road. Russell is in the process of bringing Uma Uma! Original Hakata Ramen to the Philippines with the first location opening in November 2015 at the SM Mall of Asia.

Interestingly, one of Russell’s ideas to attract talent in Singapore is to rotate the team through Manila. We know something about this as we do the same thing with our #AdventureEngineering program. We rotate our team through our Whistler, Manila and Singapore offices and use that adventure as a core part of our recruiting effort – and it works. Russell’s take on this is interesting and I think has legs. Singapore’s restaurant and retail businesses have a big challenge when it comes to recruiting. The Singapore unemployment rate is 1.9% [2014 (Q4)] and it is difficult to bring in non-Singaporeans for the roles. So many restaurants and retailers go understaffed. The concept of rotating team members from Singapore through to Manila could be that added benefit to not only attract A players but keeps them around. Interested in learning more about the opportunities at IKI Concepts, reach out to their HR team.

How We Keep Young Star Talents from Leaving

Every employee is valuable. Repeating this old tired phrase really doesn’t mean much anymore. Sure, we know we are of value, but how valuable? Can we measure that? How do we know if our bosses are with us as much as we are with them in this company? Studies have revealed that the millennial generation is the population in the workforce that really values these questions, and use it to determine if they have join the right company to work for.

The PayrollHero Team has millennials as the majority. Just to be honest, our young talents in our team are being headhunted daily because they are doing such a stellar job and other companies are starting to take notice. However, the thought of leaving never occurred. Here’s how we keep our young rock stars with us!

We communicate all the time
There are no walls between management and the employees. We function as a team, always working towards the bigger picture. Our team is constantly on the move, functioning from different offices in Singapore, Canada and the Philippines, which is made easy using productivity apps like Slack. (Read on how we use Slack)

Having great communication means we are clear on the tasks each team has to complete and eliminate any grey areas and speculation. There can never be enough clarity when working with Millennials. Each team gathers for a huddle every morning to discuss completed tasks, give feedback- good or bad, in the moment. This way also we keep accountability crystal clear within the team.

Morning Huddles

Suggested reading: 5 Dysfunctions of a Team by Patrick Lencioni

Being flexible enough to meet halfway
PayrollHero is incredibly blessed with many gifted and talented millennials like Vince, and Maita. And we are more than happy to participate in out of office events as often as possible. We are with each other 10 hours in a day, every weekday and some weekends too and happy to do so. We work hard and play hard together.

Smiles at breakfast

Recognizing each employee’s lifestyle needs is important too. Millennials appreciate when a company acknowledges life outside of work. PayrollHero knows that mixing life and work as harmoniously as possible is one of the secret to making millennials happy to be at work. Fundamentally, we believe in getting things done. So, you can feel free to do activities you love, so long as you complete tasks that is required of you during the week.

Empowering each employee with ownership 
For a millennial, ownership means a lot. At PayrollHero, many members of the company have a stake in the company which ensures that everyone commits. As a millennial myself, I can see that having that kind of ownership is empowering because I can be a part of something bigger. The days of working for a company just for the paycheck are gone. Owning a part of the company is a great incentive for an individual. But more importantly, it pulls the team together to focus towards a common goal: to see the company succeed.

To know exactly how we work, you’ll just need to join us. We are hiring for many positions across all offices and even for home based roles.

Ukrainian Adventure Engineer in PayrollHero Singapore

Announcing Our Latest Import
Capture MYKOLA KYRYK

Our #AdventureEngineering message has reached far and wide! Software developers who have heard about PayrollHero’s Adventure Engineering have written to us, aspiring to be part of the team.

“I felt like it was a perfect match
from the moment I saw this video”
Mykola Kyryk

Here’s what Mykola did to catch our attention from his point of view:

Mykola: We had a couple sessions of remote pairing/discussion sessions. All people I talked to were very interesting interlocutors and I felt like I could bring a lot of value into the team.

Mykola spills the beans about his home country and moving to Singapore.

Tell me about your hometown. How many people are there?
Mykola: I’m from Lviv, Ukraine. It is a nice ancient city (founded in 1256). This city was part of Poland, Hungary, Austria and Russia for many years. So the country side has mixed architecture and culture with a population of around 850,000.

What kind of things you like to do back at home?
Mykola: Besides IT? Well a lot actually. I play basketball and travel a lot. In the last 2 years I’ve been to 8 different countries. I like different extreme activities. I’ve already tried scuba diving, skydiving and caving. I also like to ride my bike and go skiing.

What did the people closest to you say about your plan to move to Singapore to be a PayrollHero Adventure Engineer?
Mykola: That I was mad 😉  Some were surprised that I’m going that far away from my homeland. Most knew nothing about Singapore at all.

What were your first thoughts on Singapore after you’ve arrived?
Mykola: It is hot here! I went from +12 to +32 in 20 hours. Now I don’t find it hot anymore. I was also amazed with how structured and well organized everything is. Really convenient place to be.

What you think about the food and the hawker centers?
Mykola: Like nothing like I’m used to. Ton of options, any cuisine, for any taste. I’m still far from saying I’m used to all of this, but I love the adventure of it all. Hawker centers are great and also cheap!

Do you like the beaches in Singapore?
Mykola: That was the first thing I did when I arrived. I didn’t know how to get to the beaches so I spent almost 2 hours walking there. Night swimming is prohibited here, but I got what I came for. Beaches are really nice, especially at Sentosa island.

Was it easy for your to get your Singapore Employment Pass (EP)?
Mykola: I got mine while I was still in Ukraine. It was really a really quick and straightforward process. Once I arrived I had to go to the Ministry of Manpower to finalize the paperwork and got my EP card in 5 days. Amazingly quick and efficient.

What do you miss about home?
Mykola: I don’t really have time to miss about anything. Singapore is an amazing place. And there is still so many places I want to go and things I want to try. Of course, miss my family, but Skype helps to stay in touch. Hope to see them soon.


Did you know,… We’re Hiring! 

Wanna be an Adventure Engineer with PayrollHero? Check out our job openings and get in touch!

Asia HR Leaders Forum in Singapore with Roffey Park

Earlier today, PayrollHero co-founder Stephen Jagger spoke at the Asia HR Leader Forum organised by Roffey Park. If you don’t know Roffey Park – they are an internationally renowned leadership institute based in the UK and Singapore, focused on developing people who develop organisations. Find out more on what they do here – www.roffeypark.com

Roffey Park had a unique approach for the forum. The most interesting segment would be the facilitated table discussions – where the 30+ HR professionals in attendance were split into 5 smaller groups, and each group was given a specific topic to discuss concerning HR with a “subject-matter-expert” leading that discussion. Each discussion was deeply insightful as participants actively shared their thoughts and experiences on major HR issues: changing employee mindsets, emotional intelligence, developing strategic thinking capabilities, staff morale and understanding the human dimensions of productivity.

DSC_0465_smaller

Facilitated HR Table Discussions

DSC_0464_smaller

Participants in discussion

DSC_0480_smaller

PayrollHero Xray Insights

Stephen closed the forum with a presentation on the importance of integrating a business intelligence component into HR systems. Steve shared the PayrollHero Xray Insights app, which provides businesses with useful analytics based on employee data, so managers can make better, intelligent decisions on human resources. He also ran through how our Adventure Engineer program works to attract and retain talent.

Read more about Xray Insights

“I found the session yesterday really inspiring with the passion you have to make a difference.”

Simon Lau
Leadership Effectiveness & Organisation Development
Group Talent, Learning and Culture Standard Chartered Bank

You can see Steve’s presentation deck below:

Here is some of the material covered by Roffey Park:

 Want to learn more about PayrollHero? Register for our next event on May 14th 2015.

Adventure Engineering in Da Nang, Vietnam

Editors Note: Introducing Vincent Paca, Adventure Engineer at PayrollHero. He will be contributing to the PayrollHero blog from time to time.

Not so many months ago, a talkative fellow wandered in my small Cebu office by chance. Dressed in shorts, flip flops and a black shirt with a man in a red cape. Turned out that was Mike Stephenson, CEO of PayrollHero. He talked about adventure, learning, teamwork and improving one’s self and the next thing I knew I was on a plane to Manila two days later to pair with a few of their engineers. A little under a year and here I am, in Singapore, busting out a blog post about the adventures and misadventures of Adventure Engineering.

Adventure engineers often travel depending on the client’s need. We go wherever we can be more effective at our job. Being ridiculously client focused, as we always say. (yes, we are hiring)

Da Nang, VietnamOne of the things that we wanted to try out this year was to partner up with 3rd party engineering teams to help us increase our velocity. In comes AgilityIO, a 150-strong engineering team based in Da Nang, Vietnam. We believe that a lot of problems can easily be solved when you’re in the same room together with your team. Making the communication loop so much shorter meant making things move faster.

At the beginning of April, Adam, Mike, Piotr and I decided to meet up in Da Nang, Vietnam to personally meet and work with team Agility. Adam, and Mike flew in from Singapore, Piotr arrived from Whistler and I catapulted all the way from Manila.

Getting to Da Nang is no simple task. Flying from the Philippines meant that I had to take a flight to either Hanoi or Saigon, then take another domestic flight to central Vietnam. I landed in Saigon around 12am bags in hand waiting for my next flight at 5am. DO NOT fly at these hours. Saigon domestic airport closes at night and the earliest they open is at four in the morning. I had to stay outside of the building and wait for it to open. Fun times indeed. I arrived in Da Nang around 6:30am. Being in Vietnam also meant that we had to wake up an hour earlier. Our daily huddle always happens at 6:47am GMT+8 and no, it doesn’t matter where you are.

I downed a can of RedBull and a shot of CĂ  phĂȘ sữa đá and headed to AgilityIO’s office with Piotr. Agility has a HUGE office in Da Nang, 5 floors of mobile and backend developers, it seemed like they had it all. We knew from the start that this wasn’t going to be a breeze. Communication and getting your thoughts across was definitely a challenge, but we didn’t let that wall stop us from where we’re going. We’re now two weeks into the project that we’re working with Agility and we’re almost near completion.

piotr, vince and agility team in da nang

A selfie with some of the Agility team in the background

Da Nang wasn’t all work and no play. We went for a food tour around Da Nang at lunch time. Summer from Funtastic Food Tour took us around Da Nang to its secret foodie gems scattered all around the city.

The next day we hopped in a cab and ventured to Hoi An. Our first stop was the Marble Mountains. The Marble Mountains is a cluster of mountains made of, you guessed it, marble. Going up one of the mountains wasn’t that difficult. Partly because the local government attached a modern elevator to the side of the mountain that takes you up halfway of the mountain.

(Adam, Piotr, Mike and myself)

(Adam, Piotr, Mike and myself)

Scaling the peak wasn’t all that bad either. All that marble makes for perfect steps when crawling up to the peak.

da hang, vietnam marble mountain stairsWe just made it in time for lunch at our hotel and just enough time to prepare for our motorbike tour around the countryside of Hoi An. This was the very first time I learned how to ride a motorbike and surprisingly enough it wasn’t that bad. Vuong took us around the scenic countryside of Hoi An, we rode on floating bridges, elevated bamboo bridges and along the rice fields. Exhilarating and enjoyable, the whole 5 hour ride was quite the experience.

bike tour around da nang

Finishing at 7pm left me little time to prepare for my 9pm domestic flight to Hanoi to get to Manila. Even though I left Vietnam a bit exhausted with mud on my shoes and a little bit of dirt here and there, I had a really great time. 10/10 would definitely do again.


Want to read more about our Adventure Engineers? Check out Nico’s story from when he left Manila to work from the Whistler office while getting his Canadian residency.

Interested in joining our team? We are always hiring!

What is an Adventure Engineer?

Private Polymer Workshop in Singapore

polymer google singaporeWe were fortunate to have Unbug join us today in Singapore for a private Polymer Workshop. Unbug is a front-end engineer at NetEase, the Creator of #MIHTool (iOS Web Debugger Pro) and a contributor to Polymer. His work gets recognized all over the world and it was great learning from him.

Our #AdventureEngineers have rotated and some of our Whistler team arrived in Singapore as well as some of the Manila team so it was a perfect opportunity to take some time on a Saturday to learn.

We are big on learning and are always looking for opportunities to expand our knowledge. A big thanks to Unbug for taking the time out to come chat with our team.

If your interested in joining our team, reach out.

polymer_workshop_1024 polymer workshop singapore

Recap: BMA HR Leadership Conference Manila

I just returned from the Business Maker Academy’s annual HR Leadership Conference in Manila, Philippines. It was a full house with over 250 HR practitioners from all industries there to learn more about taking their HR skills to the next level.
BMA HR Leadership Conference

I was asked to speak about how PayrollHero attracts talent, how we retain talent, some of our more innovative programs and of course show them how PayrollHero’s Philippine payroll software platform works.  Here is a quick overview of what I discussed:

  • PayrollHero’s hiring page is unique and breaks down not only the job descriptions, but how we work, our rhythms, required reading, core values and much more.
  • Our Herman Miller chairs, dual monitors, Apple computers, etc. and why giving your team the best you can afford is worth it.
  • We talked about travel, how our team moves around and why having a passport is a key items all employees at PayrollHero need.
  • Learning – a big topic in HR, at PayrollHero we invest heavily in our teams education. From mentors, to conferences to round tables.
  • Our Adventure Engineering program and some examples of how it attracts stellar teammates.

And of course, the last bit of the session was about PayrollHero and how our time, attendance, scheduling, HRIS, business intelligence and Philippine payroll software works.

Here are a couple pictures from the event;

Stephen Jagger Philippine Payroll Software PayrollHero Stephen Jagger Philippine Payroll Software PayrollHero

Singapore Ruby Meeting @PayrollHero

global_123774742The next Singapore Ruby Meetup will be held at PayrollHero’s Singapore office on Feb. 24th at 7pm. If you are a ruby enthusiast, want to learn more, meet other engineers or just say hi come by and check it out.

Here is the agenda:

1. Testing Shell Scripts – Bjorn Andersson / @gaqzi
2. Development to Devops – Justin Burris / @justinbburris
3. 5 Random Ruby Tips – Akira Hirakawa

* Hot Picks
– This is a 5-10 minute session during which anyone can
stand up and say something interesting!

Be sure to register here as space is limited.