One of the big differences between Selenium IDE and a Selenium RC solution is
the ability to perform ‘soft’ assertions. Selenium IDE users can append commands
with verify or assert to determine whether the test execution should stop
when a failure is observed. A popular use for this is to first assert that you
are on the correct page (assertTitle) and then verify elements on the page.
If you were only able to assert then your tests may fail early on, not
revealing further failures that may exist.
In Selenium RC you are limited by your test framework, and from my experience
using the Java client library there isn’t a satisfactory equivalent to the soft
assertion feature of Selenium IDE. Some solutions propose that you catch the
assertions, log the occurrence of the failure, and check that there are no
failures at the end of the test. The problem here is remembering to check for
these verification failures.
Another solution suggests putting the check for verification failures in a
method that is run by the test framework after every test, however when these
fail (in TestNG) they are marked as configuration failures, and the default HTML
report can still report your test suites as passed.
Cédric Beust (creator of TestNG) has discussed soft assertions on his blog, but most of the proposed
solutions differ from the simple implementation that would encourage more
Selenium IDE users to adopt Selenium RC.
TestNG has support for custom listeners, which can run when tests
pass/fail/skip, as well as before and after invocation. By adding a custom
listener to check for verification failures after invocation, we can get the
details of all verification failures that have occurred, and report them at the
same time as we report our hard failure, or if there are no hard failures we can
change the result to a failure and report the verification failures.
This solution uses part of the TestNG soft failures patch by Dan Fabulich in
order to combine the stack traces of multiple failures.
Details of the patch are available here.
I have created a simple Eclipse project that can be downloaded, extracted and
run. To keep it simple, this project does not use Selenium. You will need to add
these files into your own project.
TestBase.java
This is the test base class, that imports TestNG and overrides the assert
methods so that test classes extending this class are decoupled from TestNG. I
have kept the methods in here to a minimum, in reality you will want to override
more of the assert methods and have more equivalent verify methods.
TestListenerAdapter.java
This adapter implements TestNG’s IInvokedMethodListener and overrides the two
methods afterInvocation and beforeInvocation. This means that when you
extend this class you only need to override the afterInvocation method.
CustomTestListener.java
This is where the verification failures are combined for the report, and
successful tests with verification failures are turned into failing tests.
VerifyTests.java
A few example tests that will all fail with verification or assertion failures.
Run these tests to see the differences in the HTML report.
Click here to download the above
files, including dependencies, Eclipse project file, and Ant build.xml file.