Unacceptable: Unit testing will take 20 years to catch on

Eli Lopian    Agile, Unit Tests    

During the Typemock Partnership Academy Panel in Oslo, I had a very strange feeling coming from Uncle Bob, when asked “what is the state of unit testing” that was the feeling of acceptance, but I felt a tingle of consent perhaps a bite of giving in.

“We are on track, everything is ok, the state of unit testing – is the state of unit testing, so their is no place to feel happy or sad about it” and “it will take 20 years, and that is ok, most of us, the ‘bad’ developers will die out and be replaced by craftsmen”

Gil Zilberfeld although quiet, nicely pointed out: “I feel like I am listening to a Buddha Guru”

I feel very different, I know that 90% of the developers don’t unit test. That is 9 out of every 10 developers, that is outrageous, 10 years of unit testing and  only 10% of the developers – get it. Some developers we have interviewed said “Yup, I tried unit-testing, it is just a way for management to make us write twice as much code in half the time, I will never do it again”

As Roy Osherove pointed out, there is talk of dividing the developers into 2 groups, “Us” – who unit test, and “Them” who don’t. “We won’t even try to teach them, its their loss.” I can see where this comes from, we would prefer to hire only developers who unit test their code. But the truth is, in hindsight, the majority of the developers working at Typemock, didn’t know how to unit test and learnt it on the job. I actually think that there is already a separation, There always has been one, between the purist (Whether it is Strict-Object-Oriented, UML-Modeling, TDD, craftsmanship you name it) to the pragmatic developer: The highest level of joy, is seeing our product being used, so we need to understand our users, create something that is useful and joyful for them, with the lowest cost of ownership. We will use any tool/process/trick to make this happen.

I had a great discussion with Richard Fennell from Black-Marble over breakfast. Richard told me that Management in companies are focusing at investing millions of dollars on the “user-tests” and see those tests as the ultimate QA solution, while hardly any invest in unit-testing. “We have no quality problem, because we have a QA team”. Now the QA market is a huge and very strong market and unit-testing is a threat to that market. I have heard many QA people, thinking about how this will make them lose their jobs. Richard talked about the normal software developer, who doesn’t have the passion that we have, that see their job as “a job” and don’t spend their free time going to conferences and writing an open source project, they just want to get the job done.

Perhaps this easy-going, let’s wait this out, attitude, is what’s hindering the acceptance of unit testing in the world. It is clear to me that 20 years is too long for this to happen, I know that there is another way, that technology and business value have the power to make that transformation become real.

Debugging Communication Errors

Eli Lopian    .NET Tests, Community, Debugging, Management for Geeks, Time Management, Unit Tests    

I couldn’t sleep all night, thinking that on the last company meeting, when one of my employees complained that he can’t finish his tasks because of personal issues that he has with other employees,  I basically told him and the rest of the team to help each other and be a team, I was managing them instead of empowering them, and knew that the issue probably remains unresolved…

For the next meeting, I decided that no matter what I do, I will make sure that my team goes back to work after they resolved their issues themselves, empowered to continue working and achieving  their goals.

I started the meeting by asking each of them to bring up a personal issue that he has with another employee and to resolve it with the other on the spot. It was amazingly interesting to see how they raised the problems, and found the solution at the same time.

One of the employees complained that he requested help via email. He sent the email, but didn’t receive any answers. He needed certain people in the company to fill out part of a form, but none of them did. Nothing but a simple communication problem; they all got the email but didn’t understand the call for action. His way to resolve the matter was to sit with each one of the people involved personally and to fill the form together, that way he will make sure that they understand their share, and he will make sure that it’s done on time.

One of the developers complained about having troubles with managing his time.  I explained that this is not an interpersonal problem, and asked him questions which led him to realize that his lack of time is because he is always helping people. He can’t say no to anyone, and he doesn’t want to look bad or disappoint anyone, so he ends up making everybody happy by helping them, but having no time to do his own tasks. When he understood, it was a major breakthrough, and he realized that he will have to start saying no to people or at least to say I’m busy now I will help you later; I would love to follow and see if he would really do it…

It’s only two examples out of many. The amazing and inspiring part was to see how each one of them was empowered and found his own way to resolve his problem without being told or managed.

Last, out of the entire company there was one developer that claimed that he has no personal issues with anyone in the company. When I asked him if others has problems with him, he was confident that their answer would be no.

On the second round that we did, each one of the developers needed to share their successful interpersonal communications, almost everyone mentioned him.  I guess that it was as obvious to them as it was to him.

One of the more gratifying surprises from the exercise was hearing about the successful communication routes that have developed that I did not know about.  For example, two developers found the time and were sensitive enough to help a third that was in major distress.  They simply encouraged him, but it was exactly what he needed in order to get back on track.   Another developer thanked the sales team for providing him with relevant historical customer support information.  Yet another thanked a team member in marketing for helping him with the wording in an important email to customers.

By the end of the process, it was clear that the team understood the power of effective communication and they now had better tools to resolve interpersonal issues with team members independently.  And at Typemock we know – your result (read: code) is always better when you’re working with the right tools.

When you know immediately when something is wrong, you are able to fix it, and get an immediate indication that your code is correct without the involvement of the QA. In other words, exactly like the Manager that empowered his team to resolve their problems without his involvement and without managing them, the unit testing helping the developers’ resolve problems on the spot and avoid QA involvement.

2009 retrospect

Eli Lopian    Marketing, Unit Tests    

Its my birthday, which is normally a time for self reflection and lot of fun, alcohol and friends.

As a present my friends gave me a Tandem skydiving, I am waiting for good weather to see why my friends want me to jump out of a flying plane 4 km in the air.

My birthday always falls close to the end of the year, so it is also a good time to see what happen in Typemock last year.

Typemock in 2009

We have had 4 new babes in Typemock from Roy Osherove, Doron Peretz, Shalhevet Segal and Dror Helper. Congrats, First timers: Welcome to parenthood

We have welcomed 4 new employees this year.

We have invested over 15,000 hours in our products and we now  have 6 great products to make unit testing easy

• Isolator
• Isolator for SharePoint
• Isolator for ASP.NET
• Isolator for Silverlight
• Racer
• Tracker

The development team has made a breakthrough with Intellitest, to guide developers into writing good tests easily.

We have started This Week in Testing podcast.

The company has a FedEx day, once every 2 weeks, in which developers can work on their pet project.

Our development process has evolved and is a hybrid between scrum and ToC.

We sold over 100% more licenses, showing us that more developers are jumping on the unit-testing wagon.

Our sales team has managed to help developers successfully implement unit testing in their company. Our site has received a facelift and we have a new coffee machine.

In short, we have had a breakthrough year in our development, our products, our sales and there is much more to come.

To reach these result I have to thank the Typemock team, they are a great team and our amazing customers and partners who made this happen.

Best wishes and a happy new year.
Eli

Gaps I found while dog-fooding Typemock Isolator

Eli Lopian    Product, Unit Tests    

I have been dog-fooding Typemock Isolator with the Metric Dashboard. There are quite a few gaps in the product that I have found while using it.

Before I go into the details of the gaps, I must point out that all these features exists in the older API’s (that are still available), but they are missing in the newer lambda-API’s (called AAA) that are much better.

Firing Events

This is not an Isolation feature per-se, but it is needed to simulate how our unit under test reacts to external events. In my case I needed to test the TestDriven.NET integration, the TestDriven add in can raise events when ever a test suite is run, and whenever a test completes (Thank Jamie).

Verifying the arguments of a constructor

One of Typemock Isolators unique features is the ability to fake objects that are created within the production code, we call these future instances. There was one case where I wanted to test that a constructor was called with a specific argument. The instance was a class that watched a specific directory for new files and load that file. I tested that class in another test, now I just wanted to test that the class was watching the correct directory. i.e. that the following was called

var watcher = new FileWatcher(testOutputDirectory);

I could of course change my code to have an empty constructor and to set the output directory after that

var watcher = new FileWatcher();
watcher.WatchFolder(testOutputDirectory);

But this felt unnatural, there is no reason that the FileWatcher be called without the directory – leading to more logic – testing that the user called WatchFolder – and complicating the application

Custom Argument Verification

Although we do have an API to verify if a call with exact argument was made, I needed to test that a specific argument was passed, but I couldn’t use the exact arguments API. The reason is that the argument was a class with 2 properties, but the Equals method was overloaded to test only one property, and I had to test both properties. I needed a custom checker. Ohad, told me that this existed but is undocumented in the NonPublic api (Api’s for private members)

Sequencing

We have put a lot of thought in the sequencing logic of the Api’s to make it easy to read and write the test on one hand, but keep the test Robust on the other (Robust as in, the test passes when the production implementation changes, but not the logic). In this test I needed to call an API to setup the data, but the date needed to be yesterday. Then with the data acting as normal, I needed to call the code and verify that a new record was added.

// pretend we started yesterday
var yesterday = DateTime.Now.AddDays(-1);
Isolate.WhenCalled(()=>DateTime.Now).WillReturn(yesterday);
var underTest = new DataModel();

// back to today
Isolate.WhenCalled(()=>DateTime.Now).WillCallOriginal();
// This should add a new record for today.
underTest.Update(); 

This didn’t work and I had to put both the Isolate lines together:

// pretend we started yesterday
var yesterday = DateTime.Now.AddDays(-1);
Isolate.WhenCalled(()=>DateTime.Now).WillReturn(yesterday);
// back to today
Isolate.WhenCalled(()=>DateTime.Now).WillCallOriginal();

var underTest = new DataModel();
// This should add a new record for today.
underTest.Update(); 

This is bad since changing the implementation to call DateTime.Now twice in the constructor will fail the test.

Assert the times a method was called

While testing the save logic, I need to test for a race condition and make sure that the save is called only once. The save logic is called from the calculation logic that is periodically called (say once a second). Once the auto-save interval is reached a save is performed in another Thread (to make sure that the application is responsive).

var saveInterval = OptionsSettings.Settings.AutoSaveEveryMinutes;
if (stopWatchForSave.Elapsed.TotalMinutes >= saveInterval)
{
  stopWatchForSave.Stop();
  ThreadPool.QueueUserWorkItem(new WaitCallback(t=>
  {
    Save();
    stopWatchForSave = Stopwatch.StartNew();
  }));
}

But if the Save takes too long and the method is called again, the save will be called again.

To test this I need to make sure that the Save method is called only once, but this API is missing.

Bonus points goes to those who know how to solve this.

Unit Testing the Metric Dashboard – Part 4

Eli Lopian    Unit Tests    

Continuation of unit testing the Metric Dashboard.

Debugging via Unit Tests

Sometimes, it is necessary to debug our code from our unit tests. Having the unit tests are great as they setup the scenario for us. Here is what I saw when debugging a test.

Then I remembered – the colored line around the method is a flag that the method is currently faked, that is great, I know exactly what is happening and can fix the bug quickly.

Unit Testing the Metric Dashboard – Part 3

Eli Lopian    Unit Tests    

Continuation of unit testing the Metric Dashboard.

Watching for new files

The Dashboard listens to the test result folder for new files. As this is done in another thread, the test must wait for the file to be read and processed in order to verify that the feature works. In the older versions of Typemock there was a VerifyWithTimeout api. Here is a hack to do the same with the lambda API’s. This test is actually an interesting test

1. I delete the file before copying just to be sure.
2. Here I am faking the current date to be the same as the date of the test – this is so that I get the correct value from CurrentProtection
3. Hack to Wait for Load() to be called. In this hack I set an event once the method has been called and then I call the real Load() method.
4. Copy file to watched location
5. Wait for Load to be called
6. Here a short wait is needed for the method to complete.

Unit Testing the Metric Dashboard – Part 2

Eli Lopian    Unit Tests    

This is a continuation of the unit testing the Metric Dashboard series, you can find part 1 here

Passing a fake object to constructor

At one point I had to pass a fake object into the constructor. Here I used a feature that will be available in our next versions. This feature currently called intelliTest makes it much easier to create fake objects without needing to go back and forth in the code.

Stage one – when in a correct position the intelliTest window appears

Clicking it opens a new window with the options of types to fake

Choosing the Fake, will:

• Create the fake in the test code
• Insert the local variable in the method
• Add the [Isolated] attribute to the test
• Add the correct using statements and
• Add the correct references to the project

It is really great fun using this feature as it take away all the plumbing work and sets everything up for me.

Unit Testing the Metric Dashboard – Part I

Eli Lopian    Unit Tests    

I have talked about unit testing the Metric Dashboard, today I had some time to dive into the task.

I decided to test the Bugs Caught feature of the Visual Studio Addin, I caught two bugs, the first has to do with the order of reading the trx files, the second had to do with saving the unfixed unit tests between sessions so that we don’t re-count failed test that are run multiple times.

Using Files in Unit Tests

I decided to use real trx files in the unit test (I found several different ways to do this, using real files seemed the easiest). This is what the file structure looks like

To read the file the tests I had to either deploy the files or do the following.

underTest.Load(@"..\..\..\Tests\TestAddin\TestResults\oneFailedTest.trx");

Unit Testing the Metric Dashboard

Eli Lopian    Unit Tests    

While developing the Metric Dashboard, I on purposely did not write any unit tests. I wanted to feel and remember what is like to develop without unit test and see how to write the unit tests after the deed

Manually Unit Testing

The first issue that I had with developing is that I found myself running the code- either with the debugger or without – much more then usual – lets call this Manual Unit Testing . There where times when I know that I was wasting my time, and manually rerunning the same scenarios over and over again. I felt the urge to write a unit test as I knew that I was wasting my time. But the feeling that I had was always: I am nearly there, I’m just about to fix the bug, so why waste time writing a unit test. Most times, I wasn’t there and it took many runs and setups just to reach the correct scenario.

If I had the Metric Dashboard running I would have seen this  

That is 32% of my time in the debugger and running the application, and 68% of my time authoring production code.

Seeing this is a great indicator that I can do better. because after all this work, I still have ZERO tests to cover me.

Application Architecture

I am going to start writing unit tests, but beforehand lets give a short architecture overview.

There are 3 components

Typemock Server

This component is responsible to store the metrics of the team and publish them to the dashboard

• The Server is a Windows Service
• It broadcasts itself to the world so that clients can discover servers automatically and free them from manually typing in the servers address
• It merges and stores the data

Managers Dashboard

This component is the reporting component and allows managers and team leader to see how unit testing helped the team (by catching bugs early on) and by spending less time doing manual unit testing. Using this dashboard we can see how much time is being put into writing unit tests and how useful it really is.

• Auto-Finder, discovers the Typemock Server and then automatically connects to it.
• Data Filtering is done in the Dashboard via different criteria’s
• Export the data as Excel
• Charts and Pies

Visual Studio Add-in

This component tracks the unit testing effort per each solution and saves it locally and on the Typemock Server. Using this each develop can see how much time is being invested in unit tests and the immediate benefits.

• Auto-Finder, discovers the Typemock Server and then automatically connects to it.
• Current Action, discovers what the developer is doing. Debugging, Running the application, Writing unit tests, Writing production code, Idle or in another application.
• Bugs caught, reads the test run data and calculates the numbers of bugs found. i.e. the number of failing tests. This component counts each failed test once until it is fixed.
• Charts and Pies.

What do the colors mean?
The colors are:

• Yellow: Communication aspects
• Green: Data aspects
• Charts: UI aspects
• White: others

Where do I start?

So now that I want to get going, my biggest question is where do I start?

Measuring Effective Unit Tests

Eli Lopian    Code Integrity, Unit Tests    

What is an effective unit test?
A good unit test (via Jeremy Miller) is

• Atomic
• Independent & Isolated
• Intention Revealing
• Easy to Setup
• Runs Fast

But how do we know that the test is effective?
The biggest value of unit tests come when they fail. When a test fails, and we fix it, we have saved considerable time in discovering the bug, pinpointing the failing scenario and fixing it. The more bugs we find early on and the faster we fix them, saves us headaches and time. We also know that the longer we leave a bug in the system, the harder it is to fix.

I have debate this before (Measuring Code Integrity) and I think that measuring the Bug Fix Time is a direct measurement of the effectiveness of our unit tests. I have written a small application that sits on the developers desktop and we are starting to use it in Typemock.

This is what the small application looks like. Each developer can see how long it takes them to fix a test. We still have to tweak the calculation.
We are thinking of counting the failure time only after the first test is run (for TDD teams) although this will not count known bugs, as it is standard to write a test before solving them.
We do consider a group of test that fail together in one run and are solved together in another run as one test.