Typemock Patents- you’re missing the point

Eli Lopian    Product    

Roy just asked me to blog about a twitter-conversation on our Typemock patents. 
It is true, we do have a patent. Its number is: PCT/IL2007/001152 
It is natural for companies to file patents of their innovations and inventions.

We have no answers to the questions that where asked and we haven’t any policy about this as this is not important enough.

What is important and what we are focusing on is that less than 10% of the developers actually write unit tests! We are focusing on innovating to get the right tooling that will enable the majority of developers to unit test their code, becoming craftsmen and professional and take the whole software development community another step forward, not on delaying innovation.

image by ihtatho

Gaps I found while dog-fooding Typemock Isolator

Eli Lopian    Product, Unit Tests    

I have been dog-fooding Typemock Isolator with the Metric Dashboard. There are quite a few gaps in the product that I have found while using it.

Before I go into the details of the gaps, I must point out that all these features exists in the older API’s (that are still available), but they are missing in the newer lambda-API’s (called AAA) that are much better.

Firing Events

This is not an Isolation feature per-se, but it is needed to simulate how our unit under test reacts to external events. In my case I needed to test the TestDriven.NET integration, the TestDriven add in can raise events when ever a test suite is run, and whenever a test completes (Thank Jamie).

Verifying the arguments of a constructor

One of Typemock Isolators unique features is the ability to fake objects that are created within the production code, we call these future instances. There was one case where I wanted to test that a constructor was called with a specific argument. The instance was a class that watched a specific directory for new files and load that file. I tested that class in another test, now I just wanted to test that the class was watching the correct directory. i.e. that the following was called

var watcher = new FileWatcher(testOutputDirectory);

I could of course change my code to have an empty constructor and to set the output directory after that

var watcher = new FileWatcher();
watcher.WatchFolder(testOutputDirectory);

But this felt unnatural, there is no reason that the FileWatcher be called without the directory – leading to more logic – testing that the user called WatchFolder – and complicating the application

Custom Argument Verification

Although we do have an API to verify if a call with exact argument was made, I needed to test that a specific argument was passed, but I couldn’t use the exact arguments API. The reason is that the argument was a class with 2 properties, but the Equals method was overloaded to test only one property, and I had to test both properties. I needed a custom checker. Ohad, told me that this existed but is undocumented in the NonPublic api (Api’s for private members)

Sequencing

We have put a lot of thought in the sequencing logic of the Api’s to make it easy to read and write the test on one hand, but keep the test Robust on the other (Robust as in, the test passes when the production implementation changes, but not the logic). In this test I needed to call an API to setup the data, but the date needed to be yesterday. Then with the data acting as normal, I needed to call the code and verify that a new record was added.

// pretend we started yesterday
var yesterday = DateTime.Now.AddDays(-1);
Isolate.WhenCalled(()=>DateTime.Now).WillReturn(yesterday);
var underTest = new DataModel();

// back to today
Isolate.WhenCalled(()=>DateTime.Now).WillCallOriginal();
// This should add a new record for today.
underTest.Update(); 

This didn’t work and I had to put both the Isolate lines together:

// pretend we started yesterday
var yesterday = DateTime.Now.AddDays(-1);
Isolate.WhenCalled(()=>DateTime.Now).WillReturn(yesterday);
// back to today
Isolate.WhenCalled(()=>DateTime.Now).WillCallOriginal();

var underTest = new DataModel();
// This should add a new record for today.
underTest.Update(); 

This is bad since changing the implementation to call DateTime.Now twice in the constructor will fail the test.

Assert the times a method was called

While testing the save logic, I need to test for a race condition and make sure that the save is called only once. The save logic is called from the calculation logic that is periodically called (say once a second). Once the auto-save interval is reached a save is performed in another Thread (to make sure that the application is responsive).

var saveInterval = OptionsSettings.Settings.AutoSaveEveryMinutes;
if (stopWatchForSave.Elapsed.TotalMinutes >= saveInterval)
{
  stopWatchForSave.Stop();
  ThreadPool.QueueUserWorkItem(new WaitCallback(t=>
  {
    Save();
    stopWatchForSave = Stopwatch.StartNew();
  }));
}

But if the Save takes too long and the method is called again, the save will be called again.

To test this I need to make sure that the Save method is called only once, but this API is missing.

Bonus points goes to those who know how to solve this.

Bug Fix Time not a good metric

Eli Lopian    Code Integrity, Product    

After a few weeks using the BugFixTime metric, I found that metric too hard to understand and leaves the developer and managers clueless to what that have to do to fix the metric.

We have done some internal thinking and some feedback and have created the next generation of this tool built to help teams develop with integrity

With this tool we can see the percent of time we are spending writing unit tests,  what percent we spend debugging our application and what is left writing production code.

The theory is that we debug our code when there is a bug, but when this is done without a unit test, then we are Manually testing, and the time we spend doing this is longer (we have to setup the environment) then doing this via a unit test, and is not as cost beneficial as writing a test – a unit that can be automatically run.

We found that when developing with unit tests, the percent of time spent debugging drops drastically and that time is spend writing the unit tests. But we get more value for our buck – we get a security net of automatically tested code.

In other words – we end up spend about the same amount of time writing production code, but we get better quality and so we need to spend less time in the integration/system testing phase.

We are able to see the metrics over time and see how much are improving.

The metrics are connected to a Typemock Server that enables us to see the total amounts for our teams, and show how much time we are spending unit testing, how much time we are saving from debugging, and how much the tests are protecting us.

Here is what the Team view looks like

Currently the tool works for msTest and is tested for Visual Studio 2008.

Simple or Simply Terrible

Eli Lopian    Product    

I read the Lead Blog post: Simplicity.

Simplicity means the achievement of maximum effect with minimal means. - Dr. Koichi Kawana

I though that was the definition of Effectiveness.

I think that a better definition is: Clarity, Precision, and Ease

Once you are clear, precise and easy, you will probably be effective. That is what makes simplicity so hard, that is why we spend hours and hours discussing how to make our products simple.

Managing the support

Eli Lopian    Management for Geeks, Product    

We take our support really seriously here at Typemock, our customers see and talk about this great value. But there is much more to support then fixing bugs.

We learn from our support how our customers are using the application and what are the pit falls. From every case we proactively try to find out how to create a feature that no one asked for, but that has great value. We change the application so that the user falls into the pit of success

This is how we came up with the Recursive-Fakes. By seeing how users write their tests we saw that a lot of their time is spend writing a fake, running the test and then seeing it fail because the fake was part of a call chain [cat().tail().wag()] and doing this recursively (hence the name) until all calls are satisfied.

We supported the call chain scenario for quite some time, but we didn’t understand the pain that our user has to go to setup his tests. After Recursive-Fakes was created, the user falls into the pit of success and gets it right the first time. Once an method is fake all the chains from that method are fake too. This feature was so strong that it became the default and both other frameworks have partially implemented this feature too.

Comfort Zone

In order to do this, we have to leave our comfort zone in our support. We can not just close a case after the customer is satisfied, even though we would really love to. We have to take responsibility that it won’t happen again. That users will fall into the pit of success and not need to look it up in the documentation or search the forums. To do this we must think of a feature that will do this and feed it to our backlog of features. This is where we must use loads of creativity, and although it is difficult, we must do this to excel.

Measuring Support

There is a problem measuring support. On one hand once the product gets better, we should be getting less and less support. On the other hand, because more people are using the product, we should expect more support cases. So I would like to see a trend of our product getting better, but we have to make sure that there is enough people in support.

The current important metrics are

• Cases that one support reps can handle in a week
  This is a derivative of the time it takes to fix an issue. This metric is important to see if we need to add another support rep.
  We expect this to raise as we get better in answering issues and as our product evolves and become more maintainable
• Cases open per week
  Are we getting more support cases?

Other measurements for the quality of support are

• Time to fix issue
  Our customers want their issues solves, so we should measure the time it takes us to solve the issues.
  We expect this to metric to become faster as our product becomes better
• Time to first treatment
  We know how agonizing it is to send a support issue and not get any answer. We care about our customers and strive to answer them as quick as possible
• Number of pit success features
  This is how we can tell if our support is effective at creating customer features, and not just repeating the same issues.

Although it seems easy, these metrics are actually hard to extract from most issue tracking systems. But these metrics can be used for integrity management. Each rep can ‘commit’ to the number of cases he will handle, the time for first treatment, and the number of pit success features he can create.

What metrics do you use?

Why do we ignore your arguments?

Eli Lopian    Product, Release    

With the release of Isolator Version 5.3, we have added the ability to simulate external components based on the arguments passed to those components. We called this Conditional Behavior.

Our default is to ignore arguments, to fake a method without taking the arguments into consideration, neither the number of arguments (overloads) or the values of those arguments (conditional behavior).

There is a reason for this.
Lets see the values with which we create our API’s, taken from Roy’s post

• Consistency
• Discoverability
• Explicitness
• Single point of entry
• Readability
• Single way to achieve things
• Backwards compatibility

The value that I want to discuss is Explicitness: we decided early on to be as explicit as possible about the API, so that the least guessing needs to take place by the user

So how do we decide on our defaults?
We fight about them, we find out what will need the least guessing, but keep the tests short (Readable)?

We talk about what most users mean, what most users expect.
We talk about failing fast when writing the code and we talk about Brittle vs Fragile when running the code.
We argue about what default usage will break if the user changes (refactors) the production code, without changing its logic. We want those tests to succeed.

This is ultimately why we ignore the arguments, so that changing a value passed to a method or using an overloaded method instead, will not fail the test.
Those cases where the difference is required, are discoverable while writing the test. The test will fail fast, if an overloaded method needs a different behavior, or if the method acts differently based on its arguments, and that will allow the writer to add the conditional behavior.

Isolator for SharePoint – Free License

Eli Lopian    .NET Tests, Product, Release, SharePoint    

Get a free Typemock Isolator license:

[Update: Contest is finished, Thanks to all you bloggers]

We’re announcing today about a new product: Isolator for SharePoint. It is almost the same as Typemock Isolator, but will only work on APIs that are directly connected with sharepoint’s API. That means that if you only need to test sharepoint stuff, you can get a powerful product, for a much cheaper price than the full Isolator (you can always upgrade later if you need to isolate more APIs).

With this release, we want to gather the power of crowds, by offering you an incentive for blogging about this news: Get A free Typemock Isolator license (the full one) just because you have a blog. Here’s the official word:

Are you a blogger, webmaster, or Internet columnist? Get a Free Typemock Isolator License (Personal Edition) by helping us launch our new product, Isolator For SharePoint, the only tool that can unit test SharePoint applications without a SharePoint server – see the Microsoft Guidance.

Go ahead, post about it now and your Free License will be on its way!

So how do I get the Free License?

Just make a post on your blog or site about the latest Typemock product, that includes the following text:

Typemock are offering their new product for unit testing SharePoint called Isolator for SharePoint, for a special introduction price. it is the only tool that allows you to unit test SharePoint without a SharePoint server. To learn more click here.

The first 50 bloggers who blog this text in their blog and tell us about it, will get a Full Isolator license, Free. for rules and info click here.

Then please send an email to sharepoint@typemock.com including your name and the post URL, and your Free License will be on its way!

Hurry up – this offer is limited to the first 50 bloggers.

A few simple requirements:

(1) You must own a website, blog space or internet column, older than 2 months, or get permission from the owner of such a site.
(2) Your post must include the text above (including the links).

Ruby Style Isolating – Aspect Faking

Eli Lopian    .NET Tests, Product, TDD    

I have talked in the past about Ruby Style Isolating (Dynamically Typed), now it is part of the AAA syntax. 

The big value of this feature is that you don’t have to inherit a type in order to replace it with a fake, the downfall of this is that when you refactor your code, you might fail the test.

There are many uses. One example is if you have a class without virtual methods,  you can still overload it and swap the real type with the fake one.

This gives you testability strengths and the ability to keep your design intact.

Example Suppose we have the following class:

public class NiceClassWithoutVirtual
{
    public int NonVirtualMethod()
    {
        return 30; // an example not virtual
    }
}

Now we want to use a manual Stub to replace our NonVirtualMethod: we can do the following: note the new before the method.

public class FakeNiceClass: NiceClassWithoutVirtual
{
    public new int NonVirtualMethod()
    {
        return 2; // our fake value in new not virtual method
    }
}

We can use the Stub as follows

var real = new NiceClassWithoutVirtual();
var fake = new FakeNiceClass();
Isolate.Swap.CallsOn(real).WithCallsTo(fake);

All real.NonVirtualMethod() calls will be directed to fake.NonVirtualMethod()

Fake an Aspect

Note that any methods that are not defined in the swapped type will be run as normal. This allows ‘Aspects’ of the original class to be faked. It is possible to fake a group of methods that have a specific meaning, for example just the ISerializable aspect or just the IEnumerable aspect – so we can fake the collection behavior aspect of a type. This is exactly how we implemented the Swap-Collections Feature. 

Releasing Isolator 5.0 and Racer alpha

Eli Lopian    Product, Release    

We have been working very hard this summer and we are just ready for 2 major releases.

Isolator 5.0

Based on customer feedback we are going to change our pricing model and add 3 new packages

• Special Bundle, containing:
  • Typemock-Isolator
  • Annual Update Subscription
  • Ivonna for ASP.NET testing
  • TestDriven.NET for seamless Visual Studio testing
• Personnel License, for single developers will cost only $199
• Open Source License, for Open Source projects will be free

We have also added a new API for better structured isolation (Mock, Stub, Fake…).

Highlights are:

• No need to understand the difference between Mock, Stub, Fake, Double they all use the same API.
• Easier to automate isolation by recursively faking all chained (sub) calls so:
  

  ClassToIsolate faked =
    Isolate.Fake.Instance<ClassToIsolate>(Member.ReturnFakesRecursivly);
  // now all sub calls will be faked, so this will not fail
  faked.GetSomething().Parent.GetDetail().DoSomething();
  
  // can setup return values via
  Isolate.WhenCalled( ()=> faked.GetSomething().Parent.GetDetail().Name)
     .WillReturn("Cool");
  
  

• Better structure of interaction validation. No recording stage and validation done at the end of the test (aka, AAA- Arrange,Act,Assert)

  // A. arrange
  ClassToIsolate faked =
    Isolate.Fake.Instance<ClassToIsolate>(Member.ReturnFakesRecursivly); 
  
  // A. act
  ClassUnderTest.MethodUnderTest(faked);
  
  // A. assert
  Isolate.VerifyWasCalledWithAnyargument(
  ()=> faked.GetSomething().Parent.GetDetail().DoSomething());
  
  

Racer Alpha

Our newest product helps solve a huge problem for multi-thread/multi-core software development, by automatically finding and pinpointing deadlocks. To use the racer simply call a multi-threaded code via the API and the Racer will start running and re-running the code in different permutations to find the deadlocks. Once a deadlock is found, you can debug the code straight to the problematic permutation!

The Racer uses both Dynamic code analysis and Static code analysis to perform its magic.

See a preview of Typemock Racer on Roy’s Blog

See an example on how to find the deadlock of the Dining philosophers problem.

Of course both the tool play well together and you can validate multi-threaded code in an isolated component.

Is the visibility of tested methods important?

Eli Lopian    Product, Release    

We are having quite a discussion lately about the importance of the visibility of tested methods.

See our internal Blog for the juicy stuff