Stop Regression Testing

Some of my colleagues and I have been building a large software application, the open-source SMACCMPilot autopilot system, using Haskell-based embedded domain-specific languages (EDSLs) to generate embedded C code.  Our thesis is that EDSLs both increase our productivity and increase the assurance of our application; so far, we believe our thesis is holding up: we’ve created a complex 50k lines-of-code application in under two engineer years that is (mostly) free of undefined and memory-unsafe behavior.

Of course, there are always bugs.  Even with a fully formally-verified system, there are likely bugs in the specification/requirements, bugs in the hardware, or bugs in the users.  But our hope is that most of our bugs now are logical bugs rather than, say, arithmetic bugs (e.g., underflow/overflow, division by zero) or resource bugs (e.g., null-pointer dereference or buffer overflows).

However, since the EDSL we are using, Ivory, is relatively young—we’ve been writing SMACCMPilot in the language while developing the language—there have been cases where we have not checked for unsafe programs as stringently as possible.  For example, while we insert arithmetic underflow/overflow checks in general, we (ahem, I) neglected to do so for initializers for local variables, so an initializer might overflow with an unexpected result.

If we were writing our application in a typical compiled language, even a high-level one, and found such a bug in the compiler, we would perhaps file a bug report with the developers.  If it were an in-house language, things would be simpler, but modifying a large stand-alone compiler can be difficult (and introduce new bugs).  Most likely, the compiler would not change, and we’d either make some ad-hoc work-around or introduce regression tests to make sure that the specific bug found isn’t hit again.

But with an EDSL, the situation is different.  With a small AST, it’s easy to write new passes or inspect passes for errors.  Rebuilding the compiler takes seconds.  Fixing the bug with initializers took me a few minutes.

So introducing regression tests about the behavior of initializers isn’t necessary; we’ve changed the compiler to eliminate that class of bugs in the future.

More generally, we have a different mindset programming in an EDSL: if a kind of bug occurs a few times, we change the language/compiler to eliminate it (or at least to automatically insert verification conditions to check for it).  Instead of a growing test-suite, we have a growing set of checks in the compiler, to help eliminate both our bugs and the bugs in all future Ivory programs.

Edit (16 Dec. 2013): a few folks have noted that the title is a bit strong: it absolutely is, that’s probably why you read it. :)  Yes, you should (regression) test your applications.  You might consider testing your compiler, too.  Indeed, the point is that modifying the compiler can sometimes eliminate testing for a class of bugs and sometimes simply supports testing by inserting checks automatically into your code.


5 Responses to “Stop Regression Testing”

  1. Using an ESDL as a strategy for eliminating bugs | Functional Paradise Says:

    […] Stop Regression Testing is perhaps a little strong, all things considered, but I do find the idea that if you’re expressing your application as a language, you’re in a position to work around certain sorts of bugs once, in one place, kind of compelling. […]

    • Lee Pike Says:

      You’re right; the post was meant to be a bit controversial. :) You should be testing your application for bugs that can’t be caught by the compiler.

  2. cbrewbs (@cbrewbs) Says:

    Regression tests are to notice changes in business logic. If someone pushes a change that all widgets are now priced $5 less, which they shouldn’t have; typechecking isn’t going to catch that. If your molecular dynamics simulator is suddenly giving wildly different results after 1000 iterations you have something to look into.

    Regression tests are data driven tests which you expect to be idempotent on a code change. They are orthogonal to type checking.

    • Lee Pike Says:

      Good point. Let me make two counter-points. First, I would argue that “regression testing” has (should have?) a more liberal definition than just testing business logic. For example, in embedded projects, John Regehr might advocate that regression tests should also test compiler correctness. Regression tests might also test for reliance on undefined behavior, which is prevalent in low-level programs.

      Second, while I didn’t explicitly discuss this in the post, I would advocate that while regression testing is important, we can often do better. For example, Ivory supports contracts in the language. From contracts, we can do things like model-check programs, to prove the business logic is correct (i.e., satisfies the requirements) or automatically generate tests from contracts and types (e.g., QuickCheck). Moreover, my overall motivation is to get software closer to where modern hardware design is today, in which significant portions of testing is replaced with formal verification.

  3. Matthew Squair Says:

    From a project standpoint reducing regression tests has significant advantages. Normally you can’t afford that many test assets so running tests through late in the project can end up as a significant choke point. I agree with your point on business logic (mid to low level requirements?) errors, testing the end product for that is a very expensive way to find a problem.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: