Unit Testing the Metric Dashboard – Part 4

Eli Lopian    Unit Tests    

Continuation of unit testing the Metric Dashboard.

Debugging via Unit Tests

Sometimes, it is necessary to debug our code from our unit tests. Having the unit tests are great as they setup the scenario for us. Here is what I saw when debugging a test.

Then I remembered – the colored line around the method is a flag that the method is currently faked, that is great, I know exactly what is happening and can fix the bug quickly.

Unit Testing the Metric Dashboard – Part 3

Eli Lopian    Unit Tests    

Continuation of unit testing the Metric Dashboard.

Watching for new files

The Dashboard listens to the test result folder for new files. As this is done in another thread, the test must wait for the file to be read and processed in order to verify that the feature works. In the older versions of Typemock there was a VerifyWithTimeout api. Here is a hack to do the same with the lambda API’s. This test is actually an interesting test

1. I delete the file before copying just to be sure.
2. Here I am faking the current date to be the same as the date of the test – this is so that I get the correct value from CurrentProtection
3. Hack to Wait for Load() to be called. In this hack I set an event once the method has been called and then I call the real Load() method.
4. Copy file to watched location
5. Wait for Load to be called
6. Here a short wait is needed for the method to complete.

Unit Testing the Metric Dashboard – Part 2

Eli Lopian    Unit Tests    

This is a continuation of the unit testing the Metric Dashboard series, you can find part 1 here

Passing a fake object to constructor

At one point I had to pass a fake object into the constructor. Here I used a feature that will be available in our next versions. This feature currently called intelliTest makes it much easier to create fake objects without needing to go back and forth in the code.

Stage one – when in a correct position the intelliTest window appears

Clicking it opens a new window with the options of types to fake

Choosing the Fake, will:

• Create the fake in the test code
• Insert the local variable in the method
• Add the [Isolated] attribute to the test
• Add the correct using statements and
• Add the correct references to the project

It is really great fun using this feature as it take away all the plumbing work and sets everything up for me.

Unit Testing the Metric Dashboard – Part I

Eli Lopian    Unit Tests    

I have talked about unit testing the Metric Dashboard, today I had some time to dive into the task.

I decided to test the Bugs Caught feature of the Visual Studio Addin, I caught two bugs, the first has to do with the order of reading the trx files, the second had to do with saving the unfixed unit tests between sessions so that we don’t re-count failed test that are run multiple times.

Using Files in Unit Tests

I decided to use real trx files in the unit test (I found several different ways to do this, using real files seemed the easiest). This is what the file structure looks like

To read the file the tests I had to either deploy the files or do the following.

underTest.Load(@"..\..\..\Tests\TestAddin\TestResults\oneFailedTest.trx");

Unit Testing the Metric Dashboard

Eli Lopian    Unit Tests    

While developing the Metric Dashboard, I on purposely did not write any unit tests. I wanted to feel and remember what is like to develop without unit test and see how to write the unit tests after the deed

Manually Unit Testing

The first issue that I had with developing is that I found myself running the code- either with the debugger or without – much more then usual – lets call this Manual Unit Testing . There where times when I know that I was wasting my time, and manually rerunning the same scenarios over and over again. I felt the urge to write a unit test as I knew that I was wasting my time. But the feeling that I had was always: I am nearly there, I’m just about to fix the bug, so why waste time writing a unit test. Most times, I wasn’t there and it took many runs and setups just to reach the correct scenario.

If I had the Metric Dashboard running I would have seen this  

That is 32% of my time in the debugger and running the application, and 68% of my time authoring production code.

Seeing this is a great indicator that I can do better. because after all this work, I still have ZERO tests to cover me.

Application Architecture

I am going to start writing unit tests, but beforehand lets give a short architecture overview.

There are 3 components

Typemock Server

This component is responsible to store the metrics of the team and publish them to the dashboard

• The Server is a Windows Service
• It broadcasts itself to the world so that clients can discover servers automatically and free them from manually typing in the servers address
• It merges and stores the data

Managers Dashboard

This component is the reporting component and allows managers and team leader to see how unit testing helped the team (by catching bugs early on) and by spending less time doing manual unit testing. Using this dashboard we can see how much time is being put into writing unit tests and how useful it really is.

• Auto-Finder, discovers the Typemock Server and then automatically connects to it.
• Data Filtering is done in the Dashboard via different criteria’s
• Export the data as Excel
• Charts and Pies

Visual Studio Add-in

This component tracks the unit testing effort per each solution and saves it locally and on the Typemock Server. Using this each develop can see how much time is being invested in unit tests and the immediate benefits.

• Auto-Finder, discovers the Typemock Server and then automatically connects to it.
• Current Action, discovers what the developer is doing. Debugging, Running the application, Writing unit tests, Writing production code, Idle or in another application.
• Bugs caught, reads the test run data and calculates the numbers of bugs found. i.e. the number of failing tests. This component counts each failed test once until it is fixed.
• Charts and Pies.

What do the colors mean?
The colors are:

• Yellow: Communication aspects
• Green: Data aspects
• Charts: UI aspects
• White: others

Where do I start?

So now that I want to get going, my biggest question is where do I start?

Measuring Effective Unit Tests

Eli Lopian    Code Integrity, Unit Tests    

What is an effective unit test?
A good unit test (via Jeremy Miller) is

• Atomic
• Independent & Isolated
• Intention Revealing
• Easy to Setup
• Runs Fast

But how do we know that the test is effective?
The biggest value of unit tests come when they fail. When a test fails, and we fix it, we have saved considerable time in discovering the bug, pinpointing the failing scenario and fixing it. The more bugs we find early on and the faster we fix them, saves us headaches and time. We also know that the longer we leave a bug in the system, the harder it is to fix.

I have debate this before (Measuring Code Integrity) and I think that measuring the Bug Fix Time is a direct measurement of the effectiveness of our unit tests. I have written a small application that sits on the developers desktop and we are starting to use it in Typemock.

This is what the small application looks like. Each developer can see how long it takes them to fix a test. We still have to tweak the calculation.
We are thinking of counting the failure time only after the first test is run (for TDD teams) although this will not count known bugs, as it is standard to write a test before solving them.
We do consider a group of test that fail together in one run and are solved together in another run as one test.

Does you code have integrity?

Eli Lopian    Code Integrity, Unit Tests    

I have been talking about integrity management, but the word integrity actually comes from the technical condition of being whole. That is why in star trak we hear that the Enterprise’s hull integrity is failing. The ship is not complete. When a car leaves the manufacturing company, it gets stamped 100% integrity, meaning that it is complete.

So what is your code integrity?

Your code is whole if it does what it supposed to do. It doesn’t mean that the client finds it useful or that it is easy to use, but it does mean that if the developer meant for the code to do something, that it actually does it.

Probably the best way to insure code integrity is to have a set of tests that the developer can run to make sure that his code has integrity.

So how would we measure code integrity?

I think that the number of test, test density and code coverage are not sufficient. They don’t really tell us of the effectiveness of the tests, so we don’t know the level of our code integrity. I am debating this, I am not so sure.

How would you measure your code integrity?

Zune Meltdown as a Management Problem

Eli Lopian    Management for Geeks, Unit Tests    

Ester Schindler has a post in cio about the 3 lessons developers should learn from the Zune meltdown. She finishes it with:

Quality Assurance doesn’t apply only to fixing code. It applies to fixing broken application development processes, whether it’s because Quality Assurance departments aren’t given the respect and resources they deserve or because developers leave it to QA to find every problem rather than bearing personal responsibility for their own code…
As one correspondent noted, "It may have been a development problem or a requirements problem. Such things are always, ultimately, management problems."

This is something that I can really relate to: The problem is a  management problem!

Management needs to focus on quality in the development process. Using the correct tools, it is easy and cheaper then expected.
As bugs are cheaper to find earlier in the process, Unit Testing is the cheapest way to add this kind of quality and responsibility.

Many development teams are understanding this and joining the unit testing trend.