More on the workbench, automatic detection and report of application changes, testing with privileged prompts and more.

In short, take a model extraction script as we saw in the previous post, run it against a newer version (presumably the next minor update), hit the 'Compare' link (near the top left corner) and it will tell you what has changed.

Sounds a bit weird but works like a charm, given the model of 2 different versions of the same application it will tell you where it has changed in a (nicely?) formatted report.
What's the use? well, in testing is near a dream tool for me, we work in a continuous integration model, we get several versions of the application per day and we used to have more than 25 persons working over it at the same time, that is without counting UX (user interface) specialist and translators (who also commit or contribute to the changes on a daily basis), keeping track of what changes, where and when was a really difficult thing to do.

Now the paradigm is inverted, when something changes our build management tool is configured to run a model extraction and compares the last 2 versions, if there are differences it generates the report and triggers a notification.
What changes are reported? well if you add a dialog, add / remove a button (or other ui element), change the style like fonts or colors, changes in the texts being displayed, most of the visual changes.

One could say if tests are in place and properly written they should fail and notify, well... in your dreams... most tests did lack a lot of assertions and did little UI asserts. In fact if they did do all the necessary assertions then the maintenance cost would be rather huge.

Anyway the topic can go really deep, enough is to say that there was nothing else needed than the basic extraction script to get alarms displaying what has changed, here's a couple of (not so good) shots of such a report.

Privileged prompts, a pain(?) to automate

Suppose you have your test machine properly configured, at certain point the app may require to do something for which windows security will ask the user to confirm if it wants to proceed:

If it were easy to automate that, security would be a joke (and computer viruses would rejoice at it), the challenge is to automate it in the most realistic way, with real security settings in the test machine.
Writing a kernel driver or other low level solution would be very platform specific, not to mention the care and cost it would require, no problem, murphy does the trick for you.

Here's a simple model that handles the case:

You may notice that from the privileged dialog it only does the Yes / No button, that's because the extraction script instructed to do so, if you want to go crazy you can go as much as you want, for example:

(The script to produce that had about 8 lines of code which I intend to shrink to 3)

Funny thing, after the model was extracted I found 2 things right away:

• In the "Change when to be notified..." dialog, pressing OK without changing anything required me to confirm the changes, however no changes were made (is at least arguably if that's ok)
• The certificate information is expired but there was no warning whatsoever from windows

Here's a closer look at the certificate info:

Checked the date and time and they're correctly set to 2014, also tried the same in this very computer and also says it is valid until 2011. If you know why please let me know, to me it looks like a (serious? minor?) security issue, then again I'm not a security expert.

Wrapping up (the post is getting long) you can have a view slightly simplified to avoid too much arrows crossing:

Ok, time to start my weekend, hope I wasn't too confusing.

-Mat

A tester workbench, an exploratory testing tool

Murphy can do many things but the starting point for it is to know your application, to do so it will try to visit all the dialogs the application has by looking at the screen, click anything it can, enter values and so.
This is called model extraction, it creates an internal representation of the application and the transition between states / dialogs.

An example of a simple application would be:

Most of it is done automagically, you only need to specify the application you want to explore. It cannot guess many things like specific values (think installation keys and so) or specific sequences of actions, however it is fully customizable and those cases can be handled easily by tweaking the extraction script for that application.

The above example was created with 2 lines of code (slightly styled for display purposes):

extractor = base_extractor.BaseExtractor('7zipWinScraper', '7z920.exe')
extractor.add_boundary_node('Browse For Folder')

The very first thing we can do with it is to interactively use that graph, just click what you want to do and it will do it for you.
Let's suppose for example you want to test what happens if the application tries to install in a folder where it does not have access right, to do so, just do this:

Type c:\readonly_folder in the text field (image at right) and press the run button, that's it.
Murphy will do the rest for you, it will create a virtual machine, launch the application type the value and press the install button.

Obviously something bad is going to happen, to see what we just open a vnc or remote desktop connection to that machine and see the fireworks, in this case we got:

Also murphy will tell us that something went wrong:

MurphyConfused: Cannot recognize view Completing The 7_Zip 9_20 Setup Wizard

2014-01-30 21:40:13,410 - root.web_workbench.planner - INFO - Run finished.

This example is rather simple but enough to give us an idea of the power of the tool, we clicked 2 steps but there's no limit to the complexity.

It also comes with a "GPS" included: you don't need to click consecutive steps, in the above example, you could for example do only 1 click in the close button of the installation finished dialog, that will create a virtual machine with a default installation of the 7zip application (yes, there are limits, but they can also be overcome with a bit of code).

But for the same price, we get a nice extra feature: cloud. In the company I work for, we use a cloud for testing (we use https://github.com/F-Secure/dvmps), you're not limited to 1 run at the time, you could ask for 10 machines in 10 different states at the same time and it will all happen at the same time saving a lot of time in many cases, you can read the news while murphy does a good chunk of the tedious work.

Last but not least, you can proofread the dialogs in thumbnails and screens mode (as opposed to a statechart), you can also export the model as a self contained file, quite handy for things like translators so they can have contextual information of the texts (which helps to avoid typical transliteration mistakes)

This is just the tip of the capabilities of murphy, there are many others to mention in future posts :)

-Mat

A simple riddle...

While writting my next post on murphy it occurred me this simple riddle, can you take it?

"You will find it when you can't find it"

It's ok to post "I found it" but please don't post spoilers.
Need a hint? leave a comment... (c'mon, it's easy...)

-Mat

About time...!

It took a while, quite a while but it's finally happening, Murphy goes open source!
What would that be? well, it can do things like this:

No, it is not a flowchart drawing tool, those graphs were extracted from the apps automatically, how much effort does it takes? a few lines of python code.
So what is it? an experiment that I started a few years ago at work, it can do various things that are quite helpful like extract the model of an application, generate those graphs, report changes between different versions of an application, a nice interactive exploratory testing tool and a few other things.
More to come, just need to find the time to write it :)

-Mat

PS: no url yet, but soon, quite soon

Updated 15/3
Sources can be found at https://github.com/F-Secure/murphy, more information can be found in the later posts I published, I'll do a summary page about it soonish

Updated 20/3
A 4:11 minutes video can be watched at http://youtu.be/zUYmzYI_pvY