Measuring Effective Unit Tests
What is an effective unit test?
A good unit test (via Jeremy Miller) is
- Atomic
- Independent & Isolated
- Intention Revealing
- Easy to Setup
- Runs Fast
But how do we know that the test is effective?
The biggest value of unit tests come when they fail. When a test fails, and we fix it, we have saved considerable time in discovering the bug, pinpointing the failing scenario and fixing it. The more bugs we find early on and the faster we fix them, saves us headaches and time. We also know that the longer we leave a bug in the system, the harder it is to fix.
I have debate this before (Measuring Code Integrity) and I think that measuring the Bug Fix Time is a direct measurement of the effectiveness of our unit tests. I have written a small application that sits on the developers desktop and we are starting to use it in Typemock.
This is what the small application looks like. Each developer can see how long it takes them to fix a test. We still have to tweak the calculation.
We are thinking of counting the failure time only after the first test is run (for TDD teams) although this will not count known bugs, as it is standard to write a test before solving them.
We do consider a group of test that fail together in one run and are solved together in another run as one test.
Things they don’t tell you about management – part II
In my previous post, I talked about our web site team problem and the solution that we reached. I though that we solved the web issue, I only managed to make things worse…
The following day, I was talking to my managers about the difference between Victim and Responsibility and about discussing this in our weekly company meeting.
After the discussion, one of my managers came up to me, and told me that a developer, one of those who was always complaining about the missing resources, and the one who just yesterday the management gave all the resources that he requested, decided to quit.
Wow!
This employee had been crying out for this resource for a few months, we just didn’t have them, and the second that he got all the resources he required – he just quit. This is a classic example of the pit of death. The employee was spending all of his energy in complaining and wasting all his time in creating excuses instead of just jumping in the water and working.
What a surprise. I learnt that sometimes when I just give in and give the employee all he asks for – we get the the root of the problem.
The way I see it is that the company has managed to solve this issue. Having a frustrated employee is no good for anyone. And if he is doesn’t want the job after he got all the resources that he wanted, then the company hasn’t found the correct place for him to be, the best thing that we can do is to enable him to shine and grow in another environment. It is a pity as the employee is very talented.
Aftermath
A few days later, the employee wanted to come back. I asked: What has changed?
After several iterations, we decided that we can work together, and now that person is growing at an amazing speed and comes to work, positively with a smile…
(picture by Brian Landis)
Things they don’t tell you about management – part I
Here is a story of a situation that we had to solve back in February. The background of this is our web site and our web site developer. Our web site developer was frustrated that he didn’t have a complete upfront design but the management take is that we have to be agile, get some parts done, time box the design and rethink as we go along, facilitate change and be lean with a YAGNI approach.
This is a typical Agile vs Upfront Design argument.
But the developer was frustrated, and kept on complaining – complaining is really bad, its an energy killer, completely opposite to our Integrity management and our Agile/Lean company culture.
Raise the issue
The first step is to raise the issue as one we need to solve, this is an important step as sometimes these issues can be ignored by the team and the management.
With the help of my coaching, I ask myself the following question.
Although I know that Agile is the better way, perhaps the signals we getting are true? Perhaps we do need more upfront design meetings? This was a breakthrough on my side, as I always thought that I was right, and now I opened the possibility that the developers are correct.
So, in our weekly Integrity Meeting I raise the issue, we had a small discussion and then we asked the developers to join us in the meeting and we asked them:
“We know that when we put our forces together – we can do the impossible, so how can we solve this? What do you need to complete the new web site? We are here to help, what can we give you to help you?”
When the developers started to complain, I explained, that the company doesn’t pay anyone to complain, it doesn’t help, we are paid to find solution. Please tell us exactly what you need and we will give it to you.
They told us the following list:
- Meetings – we need kickoff and brain storming meetings
- Page designs – we need page designs
- Who chooses – who has the final choice
- Text content writer – we need a content writer
- Graphics - we need graphics
- Back-office - we need someone to handle the Back Office
- Test - we need a test web site
We gave them all they needed.
- Meetings – we setup two meetings
- Page designs – we gave them the page designs (we had them already)
- Who chooses – we told them who has the last choice
- Text content writer – we told them who will write and when
- Graphics - we gave them the number of our graphic designer
- Back-office - we told them who our back office developer consultant is
- Test - we set up a task to build the test site.
Now I asked if they need anything else? If they have everything to create the new web site?
The answer was yes, they had all they needed to complete the site.
At the end of meeting everyone felt that we are doing the right thing – and that we solved it correctly.
(Picture from mrs_brownhorse)
Employee Motivation Myths Debunked
David Javitch talks in Entrepreneur.com about 5 motivation myths.
These don’t work
- Money motivates.
- Just keep them happy.
- Ignore Conflict.
- Some people just aren’t motivated.
- Smart employees don’t need to be motivated
I whole heartedly agree with David, it is just his 10 quick ways to motivate them that sound heartless
I think that there is one real good way to motivate your employees:
Make sure that your #1 priority is the growth of your employees. Make sure that they are challenged every day
Bring Me Your Solved Problems
John Hunter has a post that is an answer to Frank Roche : No Problem Without a Solution. John quotes Taiichi Ohno “Having no problems is the biggest problem of all.”
Both sides have a valid point here. On one hand, it is the managers job to make sure that his team can learn to solve problems themselves and not just “pitch them over the wall” (as Frank answers John), its part of their growth.
On the other hand, employees hiding problems from the manager just because they don’t have a solution is a sure recipe for failure.
So what is the golden path?
I say “Having no solved problems is the biggest problem of all.”
Here at Typemock we have a process that I think solves both problems. At our clearings we are all asked “What doesn’t work” and who ever asks the question is required to try and solve it. Here the manager can guide the team member and show him how he can solve the problem, this is usually done by asking the right questions. as Frank says
It reminds me of when I used to ask when I was a kid, ‘Dad, how do you spell….” He’d say, did you look it up? He could have spelled it for me…or he could help me think through it. I always valued that he would have me look it up.
We also ask “What problems did you solve? What is working?” Here the team member can talk about problems they solved and get feedback from it. (So problems are not hidden). If no problems where solved that is the biggest problem
Dealing with Pitching
When I feel that a problem is pitched over the wall, I say “I need solutions not problems, how can we solve this? what are the possibilities”. Once there are a few possibilities that need choosing, it is my responsibility to actually make the choice, but there must be possibilities.
Finding Problems to solve
In either case, sometimes the team themselves, don’t know that there is a problem and every manager should make rounds and find places that need attention.
(photo by samuelcartwright)
Pit of death
The opposite of the Best Case Scenario is the Pit of death.
Falling into the Pit of death can happen quite quickly when we focus on results instead of focusing on actions. Lets take an example, suppose we have a bug to find. Our software works on my machine, but doesn’t work on Gil’s. As we are result oriented we want to solve the bug.
While analyzing and trying to fix the bug, I find that I go through several phases.
First I recognize the reality, the reality is that the application doesn’t work. So I start acting to fix it, I reinstall, reconfigure, uninstall, recompile and redeploy. After doing this for some time and not getting any results, I start changing my point of view (POV). Now my point of view is that the application is buggy.
After some more time trying to fix it and not succeeding, I change my POV, now the laptop is faulty.
Then the cpu is faulty, our IT person who didn’t upgrade the hardware - is faulty, the QA for not checking it - is faulty, the product manager for thinking of such a stupid application - is faulty, my manager for giving me this job - is faulty…
Then comes the real death POV… I am a bad developer, I can’t seem to solve problems, others do it better than I can, this is really discouraging, I guess I don’t have it in me, I better switch professions…
I quit!
The reality hasn’t changed, its just our POV that has changed.
We moved from: This is a challenge – some application is faulty – someone else is faulty, to… I am faulty.
This can happen in all areas, Working Out, Losing Weight, Getting Promoted etc.
Leaving the Pit of Death
Leaving the pit of death is easy. Focus on controllable actions, not results, be in integrity. This will keep us positive and will eventually lead to success.
Here is how, “I intend to try 5 different ways to tackle this bug this hour”. That’s it.
Now try the 5 ways, in the best case scenario, we will solve the case. Otherwise, we are still men of our word and did what we committed to, that is a great feeling, now we can commit to trying another 5 ways in the next hour.
(Photo by magadelic_rock)
Hardware and Software spring Cleaning [off topic]
I did a little spring cleaning to my personal PC. It was full of dust and that drove the dual-core CPU temperature to rise to 60°C / 140°F. Once the physical cleaning was done, I did some software cleaning, uninstalling unused applications, killing unneeded services and shortening the system and user login time.
I used CCleaner to clean the registry and Startup items. But had to manually pick the services that don’t need to run, I wish there was a site/tool that would do this for me.
The biggest pest, was the GoogleUpdater (See how to uninstall it) for a company that does no evil, this was annoying, it kept coming back at unknown intervals. Turns out that this is due to the use of the Task Scheduler.
Then I completed my home movies and photos backup scheme (onto a detachable disk) using SyncToy. What a great simple piece of software, just what I needed. Using this PCHell.com article, I managed to automated the backup.
(photo by Sidewalk Story)
Best Case Scenario
Staying positive and energized is a one of the major jobs of a Manager/Entrepreneur. There are many pitfalls on the way to making our dreams come to realization, but once we go negative, it is hard to go back. So I try (not always that successfully) to refrain from saying “no”, and to not accept a “no” from others, Instead I ask "How can we make this happen?" and "Under what conditions can we do this and that?"
I try to focus on the success and positive actions of my team and not the negative ones. I believe that my team can do better then what even they themselves believe. I found out that when I focus on this, the teams actually surprise me and do become outstanding.
That is why before we do something that is out of our comfort zone, instead of saying “What is the worst that can happen, that we fail”, I say “The best that can happen is that we succeed”. Suppose we have a big technical problem? And many team members are negative, saying “This is not possible, because…”, to leave the negativity, I tend to say “Lets try it, the best that can happen is that we succeed, now lets see how can we make this happen”
Think about the best case scenario and not the worst case scenario.
(photo by soriti)
Pushing and pulling
Now that the company is in integrity it is easier to fail fast and to fix our broken processes. As Johanna Rothman says Graceful Degradation is Not What We Want; Quick Failure is Better
Not only is it easy to do it, we don’t even need a manager to do this as the actual team member can learn how to do this themselves.
Here is one case that happened to us.
Normally there is a lot of pressure on the development team to release new versions, sometimes they manage to release versions on time and sometimes we seem to postpone the release, all for good reasons. But a few weeks ago the development team surprised us, they released not only the Racer version 1.0 but also Isolator version 5.3.1 that is packed with features.
This is amazing, the development team has managed not only to keep up to the pace but overtake it.
Our marketing team was not ready for this and was suddenly was under a lot of pressure to release the new version too, so the new version was not released publicly until our marketing guys could upload it and make the correct changes.
As I have been reading Lean and Kanban lately, I felt that this was a sign of a push company and not a pull one.
So in our weekly meeting, we created a little paper plane factory. Roy, Adi and Dror sat in a line and had to create paper planes and throw them at Sivan. Roy had to do the basic fold, Adi had the hard work of folding the plane and Dror had to throw the plane on Sivan.
On the first round we did this without any constraints and Roy folded all his papers before Adi could make 2 planes, he was then bored and started snoozing. Adi was under a lot of pressure and just couldn’t keep up, while Dror just had to wait and throw the plane, so he waited.
We had a small discussion about this, about how Roy left a pile and then was bored and how Adi was pressured and how Dror was just waiting.
Then we did the same thing, but this time no-one was allowed to proceed unless the next person was ready to take the plane away from you.
This time Roy had to do a lot of waiting. We managed to make the same amount of planes, but with the constraints we found the bottleneck MUCH faster! Roy immediately pointed out that he was waiting. He even offered to help Adi (Which for the games sake, I didn’t allow). We also saw that the time that each plane was in the system stayed constant allowing us to manage it better.
It was eye opening. Having the constraint immediately made a more efficient team as one member helped with the ‘longer/harder tasks’. Another thing I noticed is that when you are stuck (because of the constraint) upstream, you are more inclined to help them when you are stuck downstream and are waiting for other people to finish.
During the Integrity management meeting the development team didn’t commit or even say that he would complete the version, so the marketing team had no idea that it would be coming. It is important to say in the integrity meeting about things that you will be passing to another person/team to make sure that they will have the capacity to handle it, keeping us leaner and having less waste.
Will liability change everything?
Some software application bug must have really pissed of Commissioners Viviane Reding and Meglena Kuneva as they want software developers to be held liable for the security and efficacy of their product.
Alen Cox, one of the leading Linux kernel developers argues against the liability. “Closed-source companies could not be held liable for their code because of the effect this would have on third-party vendor relationships”
Bruce Schneier has a really good article about the commercial effect of liability. In short the commercial cost of fixing your bugs is not as economic (they never heard of Typemock) as adding a new feature, but having a liability law will change this cost and force more CEO’s to spend more money on the quality of their code.
"We’re making software as secure as we possibly can. People don’t look at window-lock makers for the responsibility for burglary — the responsibility tends to rest with perpetrators," said Jerry Fishenden Microsoft’s national technology officer
Are we doing the best we can?
So, are we doing the best we possibly can? Should software developers be liable when they are sloppy and non professional. A doctor is liable if he makes a mistake, we should be liable if we are sloppy?
As an industry we still have a long way to reach the standards that we already have in specific markets, MISRA, DO-178B, PCI, Sarbanes-Oxley and FDA’s medical device standard. Should the software industry create or adopt a standard to draw the line of liability?
If we did have this standard, what would you say would be the minimum required actions that we have to do, to be exempt from liability? What would define a professional vs a sloppy developer.
Subscribe!
RSS 2.0