We managed to execute 17 millions tests

Last Thursday (Apr 18), we had a All Hands Call meeting briefing us the status of our product as we are closing up our endgame. We were doing an amazing job in term of our software quality. Compared to our previous release, the number of defeats were significantly lower, the number of technical debts were significantly lower, the number of test executions through automation were 10x higher, and were 100x higher than 2 release ago. In total, we managed to execute 21 millions tests.

During the meeting, we heard:

"This team is very very focus team on execution on tests. And it is not a large team. They have 17 people on N team. Of course they've got some help from outside, but that it is a small team. That team, counting all the T execution, has executed 21 millions tests."

The number was not bad at all. Our CI testing posting all the results 24/7.  Then I started to think how much our CI process actually contribute to this number. There was no breakdown in the number. After spending some time digging, I found the report showing the 21 millions test execution and its breakdown a few days later. The report shows that 17 millions tests were posted from our CI process.

It was not surprised for me to see 17 out of 21 millions are executed from our CI process. We are running more than 150,000 tests everyday. What really surprised me was that I found out all 17 members from N team were treated a nice breakfast with our director the next day.

Wait a minute, we are the hardworking team here. We spent so much time building our test system and test our code hourly and daily. We spent so much time monitor and analyze our results hourly and daily. Our code is the core engine of our software and we make sure it is always stay top notch. Not only we were not invited for the breakfast, our work was never mentioned. Sigh...  They simply took our results. I hope they know it is wrong stealing other people’s credit.

At the end of the day, the number is just the number. Shipping quality code is all you care about.

Shame, shame.

Effective Vs. Efficient in QC

 

Here is the definition from dictionary:
Effective (adj.): Adequate to accomplish a purpose; producing the intended or expected result.
Efficient (adj.) Performing or functioning in the best possible manner with the least waste of time and effort.

There is a saying "Being effective is about doing the right things, while being efficient is about doing the things in the right manner."  Efficiency and effectiveness are improvement to a process. However, to some people, they may mean "almost" the same thing. I sometime mix the meaning and usage when using them too.

We can be an efficient QC if we can do the same with less.
We can be an effective QC if we can do more with the same.

We can be an efficient QC if we emphasize on the intended time to produce the same outcome.
We can be an effective QC if we emphasize on the intended outcome with the same time.

We can be an efficient QC if we let the machines set up our environment and run our test.
We can be an effective QC if we let the human do the analysis and investigation.

We can be an efficient QC if we constantly response to change.
We can be an effective QC if we religiously follow a plan.

We can be an efficient QC if we trust more on Process and Tools.
We can be an effective QC if we trust more on Individual and Interaction.

We can be an efficient QC if we automate as many of our test cases as possible.
We can be an effective QC if we could detect more bugs early during the development cycle.

We can be an efficient QC if we are able to test the smaller incremental code changes.
We can be an effective QC if we are able to detect and fix the bug from the incremental code changes.

Brainstorming details should be remain among our group

There was a brainstorming and sharing thoughts among us. We were searching for thoughts, analyzing the good and bad of our past quality control history. We were making comments and opinions on tools and process. We were stating our personal preferences on the directions we wish to go. In this stage, all discussion details should remain among our group. I hesitate to use the word "confidential" for brainstorming process, but it is indeed confidential to some and we need to make sure it is confidential.

We didn't mean to discuss this outside our group before a decision is made. Unfortunately, one piece of email got leaked out. The person getting the email didn't see the full picture but somehow was upset at this piece of info. A few of us reviewed the email, there was no offensive words or comments, rather than analysis of short comings from one process and advantages from another process. 

Sometimes, unexpected things happened. It may be unpleasant. All I need is be strong, shake off the dirt, lay low from this incident and keep on moving. This is just "one of those days"...


Note: There is a good reason for me to lay low.