Agile Demos Smells

Eli Lopian    Agile, Code Integrity    

There are many things that I look at when I see a demo, one of thing that fascinates me are demo smells. These are automatic actions that developers do when the demo doesn’t quite go well that underlay conceptual problems.

For example when seeing a demo at the end of an iteration, does the developer have some script that he keeps running from time to time, does the developer delete files when something wrong happens, is the process monitor running. Looking at these actions gives great insight into the potential architectural bugs.

If there is a batch file, there might be some kind of environment problem, if files are being deleted there might be some integrity problem, if the process monitor is up there might be a process synchronization problem.

It is normally cheaper in the long run to solve the root problems and save time when running the application then to keep performing these automatic actions.

Ask about these actions, you will get surprising answers! Then see what can be done to remove these actions.

Bug Fix Time not a good metric

Eli Lopian    Code Integrity, Product    

After a few weeks using the BugFixTime metric, I found that metric too hard to understand and leaves the developer and managers clueless to what that have to do to fix the metric.

We have done some internal thinking and some feedback and have created the next generation of this tool built to help teams develop with integrity

With this tool we can see the percent of time we are spending writing unit tests,  what percent we spend debugging our application and what is left writing production code.

The theory is that we debug our code when there is a bug, but when this is done without a unit test, then we are Manually testing, and the time we spend doing this is longer (we have to setup the environment) then doing this via a unit test, and is not as cost beneficial as writing a test – a unit that can be automatically run.

We found that when developing with unit tests, the percent of time spent debugging drops drastically and that time is spend writing the unit tests. But we get more value for our buck – we get a security net of automatically tested code.

In other words – we end up spend about the same amount of time writing production code, but we get better quality and so we need to spend less time in the integration/system testing phase.

We are able to see the metrics over time and see how much are improving.

The metrics are connected to a Typemock Server that enables us to see the total amounts for our teams, and show how much time we are spending unit testing, how much time we are saving from debugging, and how much the tests are protecting us.

Here is what the Team view looks like

Currently the tool works for msTest and is tested for Visual Studio 2008.

Measuring Effective Unit Tests

Eli Lopian    Code Integrity, Unit Tests    

What is an effective unit test?
A good unit test (via Jeremy Miller) is

• Atomic
• Independent & Isolated
• Intention Revealing
• Easy to Setup
• Runs Fast

But how do we know that the test is effective?
The biggest value of unit tests come when they fail. When a test fails, and we fix it, we have saved considerable time in discovering the bug, pinpointing the failing scenario and fixing it. The more bugs we find early on and the faster we fix them, saves us headaches and time. We also know that the longer we leave a bug in the system, the harder it is to fix.

I have debate this before (Measuring Code Integrity) and I think that measuring the Bug Fix Time is a direct measurement of the effectiveness of our unit tests. I have written a small application that sits on the developers desktop and we are starting to use it in Typemock.

This is what the small application looks like. Each developer can see how long it takes them to fix a test. We still have to tweak the calculation.
We are thinking of counting the failure time only after the first test is run (for TDD teams) although this will not count known bugs, as it is standard to write a test before solving them.
We do consider a group of test that fail together in one run and are solved together in another run as one test.

Measuring Code Integrity

Eli Lopian    Code Integrity    

I have been pondering this for quite some time now. How do we measure the code integrity. Code Coverage doesn’t really do it, and the number of test or test density also doesn’t.

I proposed to go back to the commercial essence of unit testing and bug slaying. In a team that doesn’t unit test, we have seen that 87% of the bugs found in the QA, are the result of sloppiness, a unit test would have found that bug, as the developer was aware of that case scenario, but just didn’t test it. Only 13% of the bugs found, where bugs in the design and scenarios that were not considered.

The thing is that a bug found in the QA must go back to the developers who must track it down and fix it, just to send it back to the QA (and hope that he didn’t add any other bugs). According to IEEE, this process cost $1000 per bug. While a bug found in development cost only $25-$100. That is 10-40 time less!

When checking this out, the direct benefit of coding with integrity is that the bugs don’t even reach the QA. If we could measure the bugs that are found in the development, we could know how effective our unit tests really are.

So if we measure the time it takes to fix a bug (Bug-Fix-Time) we should see that developers writing good unit tests will fix the bugs in less time then developers who don’t. We know that the industry standard is 6 days to fix a QA bug and minutes to hours to fix a failed unit test. That is a huge difference.

Lets measure the amount of unit test that failed on the developers desk/build server and the time it takes to fix them and know our BFT.
We should strive to get this down to a minimum to have a better code integrity.

Big Steps to Code Integrity

Eli Lopian    Code Integrity    

Typemock is making big steps towards its goal of code integrity. Code integrity is what we consider the new paradigm of writing software. Moving the responsibility of code-correctness back to the developers. It is too easy to get sloppy and to leave it to the QA to find our bugs, thus developing without integrity and leading to waste in time and money. When developing with integrity, we are all responsible to make sure the code does what was intended to do.

We can reach to higher levels of code integrity, with automating unit tests, code reviews and management for excellence. We are helping teams do this by allowing developers to unit test their code, whatever design or technology they choose. Our new tool Racer allows validating Parallel Computing problems. This is a great step towards making our profession an engineering one, that we can be proud of.

Does you code have integrity?

Eli Lopian    Code Integrity, Unit Tests    

I have been talking about integrity management, but the word integrity actually comes from the technical condition of being whole. That is why in star trak we hear that the Enterprise’s hull integrity is failing. The ship is not complete. When a car leaves the manufacturing company, it gets stamped 100% integrity, meaning that it is complete.

So what is your code integrity?

Your code is whole if it does what it supposed to do. It doesn’t mean that the client finds it useful or that it is easy to use, but it does mean that if the developer meant for the code to do something, that it actually does it.

Probably the best way to insure code integrity is to have a set of tests that the developer can run to make sure that his code has integrity.

So how would we measure code integrity?

I think that the number of test, test density and code coverage are not sufficient. They don’t really tell us of the effectiveness of the tests, so we don’t know the level of our code integrity. I am debating this, I am not so sure.

How would you measure your code integrity?