More on our new functional test framework: scripting

Yesterday, I introduced EasyTest for XAF, our new functional testing framework for applications built using our application framework. Today, I'd like to reveal a few more details.

A bit of history

Why produce a test framework in the first place? Let's look at some background. There are, I suppose, two main classes of testing tool for .NET applications:

1. There's the dig-in-deep tool. This hooks into the application’s message loop and records/plays back particular messages. This approach was quite popular in the good old days. As you perform some actions on the UI, the testing tool records the actions as a set of mouse moves and clicks, and key presses; a set of messages, if you will. The resulting test script is completely unreadable, since in essence, it’s just a large list of screen coordinates and message ids and parameters. You can’t really guess what’s going on by looking at the script. You can’t understand what functionality is being tested. When such a script fails because the developer rearranges the controls on the form (or for any of a long list of reasons), it usually has to be rerecorded again.

2. There's the keyword-driven tool. This tool doesn’t record screen coordinates or key presses, but instead runs a test script that represents a set of actions performed on particular interface elements, such as buttons and text fields and the like. These scripts are much more readable and you don't usually need much experience to write them. Plus, the tests don’t break when you rearrange the controls on a form since the tool recognizes the names of the controls, rather than their locations. On the downside, though, a keyword-driven testing tool has to “know” about particular controls (properties, events, and even methods) in order to support them. If the tool doesn’t support a given control, say one from a third party vendor, you need an adapter. If the adapter doesn't exist, you'll only be able to use the control at its most basic, "ancestral" level.

We felt that neither of these approaches were good enough for testing XAF applications. For a start, XAF is designed to produce multi-platform applications, either a WinForms application or an ASP.NET application or both. The set of supported platforms is likely to be extended in the future to WPF and Silverlight applications. Even now, with only these two current platforms, it's likely you’re going to need two different testing tools, which, of course, means writing or recording different scripts.

The other thing to appreciate is that XAF already has a set of high level abstractions, such as Views or List Editors. There's no point in testing the abstractions, which is what another test tool would be forced to do. If a testing framework existed that knew about and that would take these abstractions into account, the test scripting commands would become much more compact and concise.

And from those considerations, EasyTest for XAF was born.

EasyTest Scripts

The language that EasyTest scripts use is declarative. You tell EasyTest what needs to be achieved, and the system interprets these wishes into actions for the particular type of application that's being tested. The test scripts are platform-independent.

As already mentioned, we wanted to use XAF high-level abstractions, but at the same time, we wanted end-users to be able to write functional test scripts. Since an end-user most likely doesn’t know about the various XAF abstractions, the script language shouldn’t use them. So, the scripting commands are named differently, which is why Property Editors ended up being Fields, List Editors got renamed to Tables, and the objects represented by List Editors to Records.

In order to understand the commands used, let's look at an example EasyTest script:

;#DropDB MySolution

#Application MySolutionWin
#Application MySolutionWeb

*Action Navigation(User)

*SelectRecord
 Full Name = ''
 
*FillForm
 Full Name = Test User
 
*Action Cancel

#IfDef MySolutionWin
*HandleDialog
 Respond = Yes
#EndIf

!CheckFieldValues
 Full Name = 'Test User'

The first thing to notice is that there are three types of statements: the directives, which are prefixed with the # sign, the commands, which are prefixed with an asterisk, and the property (parameter) values, which are the rest. You'll note also that the first statement is prefixed with a semicolon, which is the indicator that the rest of the statement is a comment.

The directives don't perform any actions. They just provide information about the application being tested, the database to be used, any timeouts that should be set, and so on.

The commands of course perform the actions that the test is supposed to execute. You declare what should happen on a "macro" level and EasyTest will determine the best way to implement that requirement according to the type of application being tested.

Let's go through the script line by line:

;#DropDB MySolution

Firstly, it's a comment (because it's preceded with a semicolon), but if it weren't it would be a directive to delete the application’s database, so that a newly created clean database can be used for testing purposes. Since this operation could potentially destroy a lot of valuable data, the command has been temporarily commented out. The MySolution parameter specifies a database to be deleted; this is defined in the EasyTest configuration file.

Usually an XAF solution has a single EasyTest configuration file to serve as the central store for configuration information. It essentially allows you to use the same set of scripts to test different applications by supplying different configuration files. Hence, whenever you need to change any settings, you only need to do it in one place.

#Application MySolutionWin
#Application MySolutionWeb

These commands specify the applications to be tested. Again, the MySolutionWin and MySolutionWeb parameters point to the EasyTest configuration file.

*Action Navigation(User) 

This command performs an action. In this instance, it’s the Navigation action, and this is used to navigate to the View specified by the User navigation item.

*SelectRecord
 Full Name = ''

This command works with a List Editor. The command selects the object that has the FullName property set to an empty value, and invokes the Detail View for that selected object.

*FillForm
 Full Name = Test User

This command fills the Detail View’s Property Editors with the specified values.

*Action Cancel

This command performs the Cancel Action.

#IfDef MySolutionWin
*HandleDialog
 Respond = Yes
#EndIf

This block of commands are only executed for the Windows Forms application because of the IfDef directive. The HandleDialog command stipulates that the test framework must handle the confirmation dialog. The reason it's inside the IfDef block is because XAF ASP.NET Web applications don’t have this dialog.

!CheckFieldValues
Full Name = 'Test User'

This command checks that certain field values conform to the conditions specified by secondary parameters. The command is prefixed with the exclamation point instead of the asterisk to specify that the command is expected to fail. So, in this instance, the command ensures that the FullName property doesn’t equal ‘Test User’.

Running the Tests

As stated in the previous post, you can run EasyTest functional tests either from Visual Studio (which is in essence a developer scenario), or from a standalone test runner.

The test runner is a command-line tool that can execute EasyTest scripts. The tool even outputs logs in XML files, and, if a test didn’t pass, in addition to writing to the log file, the test runner takes a screenshot, capturing the state of the application’s UI at the time of failure.

Free DevExpress Products - Get Your Copy Today