When CI triggered tests are failing - this might be the issue...

I had an issue which was bugging me for some time and have found a fix.

I had some Test Complete GUI tests which ran just fine when I manually ran them from Jenkins, but when the build was triggered by an upstream project or by an SVN check-in, the tests failed.

I was getting an External Error in a pop up - which was misleading - to me anyway.

It turned out, the solution was to open an RDP session from another server on to the Jenkins slave as the Jenkins user - this provides a desktop for the Jenkins user, which then enables the GUI tests to run successfully.

Hope this saves someone some head scratching.

Automation Engineers are from Mars, Manual Testers are from Venus...

I can't stand the them and us attitudes that seem pretty prevalent about automation engineers versus manual testers.  You hear things like:

• automation engineers are hopeless testers - give me a manual tester any time
• manual testers of course can't possibly code - give me a programmer

Of course, there are some people who perhaps can only do one of those things well, but I believe many can do both well with practice.  I like to think of automation co-existing with manual testing and both augmenting the other.

The belief that automation only finds the same bugs misses the automation process.  Yes, I agree, if you just run the scripts, then the same code paths will be exercised and the same checks will be done, but the process of writing automation scripts isn't as simple as that.

The fact is, as the System Under Test (SUT) changes and new releases are put into the test environment, the automation scripts run, and due to changes in the GUI and the work flow, the tests will often 'break' - this may be due to a bug being found - great - and we have found it fast.  Often however, it won't be due to a bug - it will be a 'bug' in the automation script - or to be more precise, the automation script isn't flexible enough to cope with the change to the SUT.  So what happens?

Well, the automation engineer will start to investigate.  He or she will probably manually look at the SUT and or step through the automation script.  So, he will start to focus on the part of the SUT which has changed (and therefore an area where a real bug may have been introduced).

Another area where automation engineers find bugs is in the process of writing a script for the first time:

Often the automation engineers won't be overly familiar with the SUT - this is a good thing - as a fresh pair of eyes I believe can often find unexpected bugs - the new pair of eyes doesn't exercise the SUT in the expected well trodden path - they can act as a 'monkey' test.  Just last week (using TestComplete), I was automating a new test on some new software (to me) and found a bug - the automation script found the bug, but it was my manual 'testing' - I was just playing with the system really to try and understand how to stop a process through the GUI - that had broken the SUT.  The web app had to be restarted to 'fix' the SUT - quite a serious bug - and one that hadn't been found to date by manual testing.

And finally, the key thing for me is that automation testing takes a massive burden off of the shoulders of the manual testers, who no longer have to plough through boring manual scripts making lots of error prone checks.  The automation can do that and they are freed up to do higher value exploratory testing.