0x26 – Where 100% C0 code coverage is just not good enough

I guess you are using rcov too to check that all your code was active in your tests at least once. Well, then, here are some bad news:

  • rcov code coverage only checks whether or not a line was executed, meaning any code from that source line. And this usually trips on
    do_something if some_condition?
    

    because what happens if some_condition? always fails in your tests? (say, “Rails.env.production?”). A somewhat chatty way to detect such cases is to write it down on multiple lines:

    if some_condition?
      do_something 
    end
    
  • Due to the heavily dynamic structure of a Ruby application nearly nothing is guaranteed to yield the same results when calling at a different time or in a different context, even code as simple as
    %w(a b c).sort
    

    may yield unexpected – for the naive reader – results. So even if you have 100% code coverage, the case might be that the entire test framework which runs the tests for you behaves differently depending if called from rcov, and then your code might behave different within the framework and without it.

No, I don’t strive for a 100% test coverage. I employ TDD any now and then: there are cases where that works just perfectly. I employ a big enough test suite to give me confidence in my app. And I employ black box tests – the real world is where the real app will be running. So testing against that is what gives me confidence in my application.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s