We managed to execute 17 millions tests

Last Thursday (Apr 18), we had a All Hands Call meeting briefing us the status of our product as we are closing up our endgame. We were doing an amazing job in term of our software quality. Compared to our previous release, the number of defeats were significantly lower, the number of technical debts were significantly lower, the number of test executions through automation were 10x higher, and were 100x higher than 2 release ago. In total, we managed to execute 21 millions tests.

During the meeting, we heard:

"This team is very very focus team on execution on tests. And it is not a large team. They have 17 people on N team. Of course they've got some help from outside, but that it is a small team. That team, counting all the T execution, has executed 21 millions tests."

The number was not bad at all. Our CI testing posting all the results 24/7.  Then I started to think how much our CI process actually contribute to this number. There was no breakdown in the number. After spending some time digging, I found the report showing the 21 millions test execution and its breakdown a few days later. The report shows that 17 millions tests were posted from our CI process.

It was not surprised for me to see 17 out of 21 millions are executed from our CI process. We are running more than 150,000 tests everyday. What really surprised me was that I found out all 17 members from N team were treated a nice breakfast with our director the next day.

Wait a minute, we are the hardworking team here. We spent so much time building our test system and test our code hourly and daily. We spent so much time monitor and analyze our results hourly and daily. Our code is the core engine of our software and we make sure it is always stay top notch. Not only we were not invited for the breakfast, our work was never mentioned. Sigh...  They simply took our results. I hope they know it is wrong stealing other people’s credit.

At the end of the day, the number is just the number. Shipping quality code is all you care about.

Shame, shame.