September 02, 2015

Automated Testing in Startup Mode

One of my friends joined an early-stage startup and we had a brief talk about testing his sophisticated and distributed software product as it grew beyond unit testing.  These are some points that we came up with for the startup in a hurry. 

The framework is not your goal - testing is your goal

You will quickly get beyond unit tests for higher-order testing.  Beware of spending too much time on the “framework”.  People get caught up in building or even extending a fancy framework long before they really know the nature of their testing and then they spend way too much time tending the framework instead of writing tests.

Moral: getting the testing done is first priority.  Write your tests first.   Then provide the minimum framework to get that testing done.  When you have enough test experience, you’ll know what you need from a framework.  

Confidence in doing enough of, and the right testing

In a distributed test environment for a complicated product, you have huge numbers of possible test scenarios.  Get your testers trained on combinatorial testing really early.  It provides good training on how to think about effective testing.  The NIST document listed below provides a readily available tool to identify a reasonable and valuable subset of possible tests.  Millions of potential test combinations can be reasonably reduced to appropriate working combinations.   See

A large number of automated tests can quickly become a burden

Start-ups tend to simply run all tests because they don’t have that many.   However, very quickly that startup will develop a large body of tests where the names or directory names of the tests are the only indicators of what the test code actually tests.

Eventually running this body of tests in a CI environment contributes enough friction that you want to be selective about the testing that is done. You will need a directory (human-readable list) of your tests and what they actually test so that you (being a tester, a developer or a manager) can trade off the run-time cost versus the coverage.  Remember the higher-order tests, such as functional, integration and system tests, will cost more in terms of development and run time and benefit accordingly from proper documentation as to what and how they test.

The best means of keeping an up-to-date directory of tests and their test activity is simply internal code documentation which is exposed via Java doc, PyDoc (better ePyDoc), etc.

Kevin Whitney

Automation Architect