One of my friends joined an early-stage startup and we had a brief talk about testing his sophisticated and distributed software product as it grew beyond unit testing. These are some points that we came up with for the startup in a hurry.
You will quickly get beyond unit tests for higher-order testing. Beware of spending too much time on the “framework”. People get caught up in building or even extending a fancy framework long before they really know the nature of their testing and then they spend way too much time tending the framework instead of writing tests.
Moral: getting the testing done is first priority. Write your tests first. Then provide the minimum framework to get that testing done. When you have enough test experience, you’ll know what you need from a framework.
In a distributed test environment for a complicated product, you have huge numbers of possible test scenarios. Get your testers trained on combinatorial testing really early. It provides good training on how to think about effective testing. The NIST document listed below provides a readily available tool to identify a reasonable and valuable subset of possible tests. Millions of potential test combinations can be reasonably reduced to appropriate working combinations. See http://csrc.nist.gov/groups/SNS/acts/documents/SP800-142-101006.pdf
Start-ups tend to simply run all tests because they don’t have that many. However, very quickly that startup will develop a large body of tests where the names or directory names of the tests are the only indicators of what the test code actually tests.
Eventually running this body of tests in a CI environment contributes enough friction that you want to be selective about the testing that is done. You will need a directory (human-readable list) of your tests and what they actually test so that you (being a tester, a developer or a manager) can trade off the run-time cost versus the coverage. Remember the higher-order tests, such as functional, integration and system tests, will cost more in terms of development and run time and benefit accordingly from proper documentation as to what and how they test.
The best means of keeping an up-to-date directory of tests and their test activity is simply internal code documentation which is exposed via Java doc, PyDoc (better ePyDoc), etc.