Testing

As repeated over and over again, testing is a fundamental part of the development process and should never be done later, when errors start to appear. Tests are also good to explain what is expected from your program before you code it and will certainly help with the design and implementation.

The rule of thumb is: never deploy without tests. But, writing the tests after your program is ready means that when you find errors (you WILL find) you'll have to refactor the whole code to fix them, or write quick and dirty hacks what's even worse. Good testing suites are important to success. Writing the test suite yourself will make you think that you're safe but not necessarily you thought about every aspect of testing and some problems might still go unnoticed.

This is the first test that should be written and it's as basic as writing simple programs that will use your data model or methods. You must simulate the use cases in software as you would like to see it being used even before you have written a single line of code or even defined the names of the classes yet. It's also called functionality test.

This is a very good opportunity to see how your preliminary model fits to a real world use and how the names of classes, methods and properties fit in the whole to assure the user (another programmer) won't get confused. It also lets you define the required output for certain operations and assert the result of the method call was identical.

In object oriented code it's common to focus on one class and map the rest accordingly and sometimes when you set the wrong class as the central one, all the rest seems a bit dodgy. This test will help you define if the structure of the classes are correct and to find an easier way of mapping the relationships and you may end up realizing that other classes were actually more central than you had previously thought.

It's easier to keep all those tests into one single file with one function for each group of tests. Don't set dependencies on anything, create all your mock objects inline, deal with them and openly state the expected output or value by doing tests. Check this example:

void testMyProgram_Interface1 () {
  interface1::MyClass mock ("aaa", 12, 0xDEADBEEF);
  interface1::MyContainer mock_list;

  // First time, ok
  try {
    mock_list.add(mock);
  } catch MyException& e) {
    print "Error inserting first Interface1 mock: " + e.what();
  }
  // Second time is forbidden, should throw an exception
  try {
    mock_list.add(mock);
  } catch MyException& e) {
    print "OK";
  } catch (...) {
    print "Error inserting second Interface1 mock, expecting MyException";
  }
}

This test method states a few design issues:

1. MyClass needs three parameters to initialize.
2. MyContainer is a list of MyClass. Sometimes it may not be obvious (given the names only).
3. The method to add a new item is add and accepts only one argument.
4. You shouldn't even try to insert the same parameter twice. That's the most important design issue in this test.

The first three might be obvious to see and given that the tester have the object model in hands there shouldn't be even doubt about it but in this case we don't have the object model, we're building it. I know, from my experience with the subject or a talk with the user that I need those three parameters to MyClass and that's the first place I'm stating it.

Later on, when I finally implement MyClass and start running this test on it, the compiler will complain if I haven't written a constructor with three parameters. In other words, this test cases won't let me forget about the overall design decisions as I go deeper in the code. It's quite easy to forget the high-level decisions especially if they're not that crucial.

Of course, if during the coding phase you agree that the third parameter is not required anymore you can remove it from your tests and it's then that the best part of writing tests first show up: The rest of your test will complain the lack of the parameter and yuo'll have to recode the whole test cases just to accommodate the change, which means in turn that your code will also be re-written to reflect the changes in the test case, which means you don't need to write workarounds in the future when the missing parameter would break the code.

Therefore, keeping a bird view test case of your object model brings you some important quality assurance:

1. You'll always know the overall impact of every change
2. You'll always be sure that the general principle was not affected after the change
3. There's no need to workaround, as you know exactly everything that is happening with your code

Unit testing is the practice of testing each method / function separately but you can perform more than one test per unit. There are standard libraries for unit testing, notably jUnit, CppUnit and PerlUnit and you're encouraged to use them. Unit tests are usually written together with the development, while building the objects. Write on each test what behaviour is expected from that method ignoring any context whatsoever, treat the methods as black boxes and let the context to the sanity/functionality tests.

You can build a start-up method that will create the mock objects and than the unit tests will run, one by one, reporting success or failure and specific error messages. Normally the testing doesn't fail upon errors but keep going trying to find as many errors as possible and report the errors only at the end.

The unit test libraries have methods like assert that perform checks on data and report on any error without breaking the program. This way you know how all failures instead of just the first. There are some idiosyncrasies on unit test libraries but the general guidelines are:

// Test Class
class MyClassTest : public TestCase {
  // Objects to test
  MyClass *mock1;
  MyClass *mock2;

  // Initialize
  void SetUp () {
    mock1 = new MyClass();
    mock1->setName("foo");
    mock1->setParam(123);

    mock2 = new MyClass();
    mock2->setName("bar");
    mock2->setParam(321);
  }

  // Test cases
  void testArithmetic () {
    ASSERT ( mock1 + mock2 == 444 );
    ASSERT ( mock1 - mock2 == 000 );
  }

  void testMerge () {
    mock1.merge(mock2);
    ASSERT ( mock1->getName() == "foobar" );
  }

  // Cleanup
  void CleanUp () {
    delete mock1;
    delete mock2;
  }

To run the tests you must add them to a suite and run:

TestSuite suite;
suite.add (MyClassTest.testArithmetic);
suite.add (MyClassTest.testMerge);
suite.run();

It is a bit of hardwork, but you don't create new methods every day and the time it'll save you in the future compensate. New developers won't need to know a huge list of manual tests and when things break they can run the regressive tests (of which the unit test is part) and see where the problem is. Even if the tests don't show exactly the spot, it'll be much easier to find and fix it because you still have the functionality test to tell you all major design issues.

After deploying the program, even having a complete functionality test and lots of unit tests bugs will eventually show up. Also, new rules may apply to your software or even special situations that you haven't thought in the beginning will also reveal themselves to you in a later stage. For all that you need to write proper tests to assure once you fix them they'll never come back. Those are the regression tests.

The regression tests is nothing more than a set of scripts or documented routines to help you go through all tests before deployment. Every change in every part of the code you must run all tests before going to production. During the development or while fixing the bug you can run only the unit tests related to the methods affected, but once it's done and the uni tests can't show any error, a broad test is necessary to show you that none of those changes affected any other part in the code.

You can also add new methods to your sanity test or create a new set with the same layout to simulate bugs in the state in which they were found (before fixing it) and try to simulate them after the fix. If you do create more methods (recommended) include them in the final tests before deployment every time.

To start a test routine can be quite painful, lots of new code need to be written, new routines need to be started and you'll end up forgetting things all the time, but it'll worth the extra work when you start getting bugs before they go to production. Once you got the routine it'll become second nature, as well as committing to revision control, documenting every change and reviewing old code.

If you use build scripts (Makefiles, Ant or Scons), the best you can do is to incorporate the test routine to them. Create rules to test each part, one rule to run all unit tests and one rule to run all tests.

Makefile:

test: sanity_test regression_tests unit_tests

sainty_test:
  run sanity test #1
  run sanity test #2
  ...

regression_test:
  run regression test #1
  run regression test #2
  ...

unit_tests: unit_test1 unit_test2 ...

unit_test1:
  run unit_test1

...

Your test routine should be the following:

• New project? Write functionality tests
• New method? Write unit test
• Changing the design? Run functionality test
• Changing a method? Run the unit test
• Deploying? Run unit test AND functionality test
• Fixing bugs? Write regressive tests
• Deploying again? Run unit test AND functionality test AND regressive test

Always keep your tests updated and your software will be much more stable and easy to maintain and the time you spent writing tests will compensate by the time you'll save not fixing bugs in the middle of the night.

When your code is tested and re-tested and you're sure you can put it in production, don't do it. Unit testing and regression testing is as far from the reality as kissing the pillow. Nevertheless, it's uttermost important to have those and when you deal with problems in the real case (that your tests didn't catch) re-write the tests to catch it.

After all the tests are done you need a stage environment, with exactly the same data as in production, updated weekly if possible, to test with the real data. Whether it's a second webserver or a second database, you need an identical environment, with the same constraints as the production (it can be a bit smaller, though) to make sure it'll really work.

Also, installing the code to this stage environment must be identical to the production environment. It's better if it's a different machine with the same infrastructure but it can be a jail (chroot) or even a test environment defined by a different configuration file or an additional parameter in the command line.

You must also have additional procedures to compare the test environment with the production after you made all changes. Whether you run the old program in production and the new in test or you just run the fixed part in test and compare the results to production data, you must assure that the results of the new run is consistent and won't affect the production dataset.

After you have done that you can install it in production and monitor for problems and side effects. It's a good idea too to describe in your bug-tracking system what differences you're expecting to see, what data will be modified and how to check to see if it really worked. This is important not only if the tester is a different person that puts into production but also for future reference for yourself to remember what happened and what kind of problem did that change fixed.