To automate or not to automate - that is the question

OK, first of all, my main area of work is software functional test automation, so you might think I would advocate the automation of everything and claim it is the panacea to war and pestilence. Not so. In fact, I think over exaggerated claims by some software vendors and automation practitioners damage the industry.

However, as you might expect, I certainly believe that software test automation done the right way can be very valuable and cost effective.

I have a client I have been working with for 4 years now and they are still getting a great ROI from the Silk Test project.  What are the critical success factors of the project?  Here are some characteristics which I believe have made it so successful:

• The software under test is quite stable - certainly the GUI is pretty stable and the API. This means that the tests are not too hard to maintain. Also, the way the tests have been written, means that changes to the GUI require code changes only (where possible) in the GUI.inc files.
• The test suite replaces many many man hours of manual testing. This is because tests have been chosen where lots of data changes and needs to be verified - very slow and difficult to do manually - a great way to use computing power which will check the same things over and over again without getting bored.
• The tests are written in an OO way such that a number of versions of the SUT can be tested using the same test code.
• The SUT is enterprise/distributed software which can be configured in many ways - we are able to test a huge number of permutations with the same test software.
• The software under test is deployed on many virtual machines using different operating systems (e.g. Windows 2008, Windows 2008r2, Windows 2005 and so on) - you can imagine how long it would take to regression test changes in the SUT on all these combinations - hence the benefit of automation increases - as it saves even more man-hours of testing.
• Doing a full regression using automation gives much quicker feedback to the development team than regression testing manually - quicker feedback equals quicker to market and cheaper to bug fix.
• The regression suite gives the customer a great deal of confidence and frees up manual testers to be more creative.  They are less bored and more motivated - and they find more bugs as a consequence.

As always, let me know what you think!