Thursday, April 3, 2014

For translators, the missing leg.

I briefly wrote about some of the goodies for the translators and the dismantler thingy has been a debt to pay, while I'm not going to formally cancel the debt I do plan to show and explain what is it today :)

This is a rather primitive but illustrative view of what the dismantler is about


We can easily have an html view of any of the dialogs along with the text extracted, what for? copy the text straight from the page.
I find it rather primitive that in many cases what a translator gets is a raw and crude text file filled with strings, one typical problem is that they lack 'context' information and it is easy to make mistakes and transliterate instead of translate.
However getting a zip file that they open in the browser with the flow of the application (as shown in previous posts) they can then click on the desired dialog and get this view, even more interesting could be that they can edit the text directly in the html page to send then the translated text.
Many things can be done along those lines, this is just a teaser of what else is possible and how Murphy goes along the whole flow of application development and testing.

Now for the sad news... I've been quite busy lately writing new code that it is intended to start taking over with more production quality oriented things, more docs, unit tests and all the bells and whistles required for serious use, however my activity on murphy has dropped quite some points in my priority list due to many different reasons so updates wont come that often in the future, hope you understand this is something that is mostly done in my spare time and to fully realize it it needs quite serious invest of time, as there is life after murphy (and other interested experiments to do too) I appreciate your patience and comprehension.

One more! Tomorrow Ohio!
(March 31- April 4, Cleveland - Ohio, https://sites.google.com/site/icst2014/home)

Hope you drop by at the TAIC conference , if you happen to bring a beer for me give it to Pekka and specifically tell him that is for me or he'll drink it right away :)

-Mat

Wednesday, March 26, 2014

The journey of the experiment, what's ahead

What's new? nothing, what's coming? a lot.

To begin with some serious experimental usage will be put in place for Murphy (some really secret stuff so I wont tell) which translates into various needs.

First of all experimental code needs to be translated into production quality code, I've been quite busy with it lately with this long and painful transition, last 3 evenings spent into rewriting the whole graph library. The new one looks quite ok and the algorithms fits nicely, short and concise (your building blocks are really important) and testability is quite good (yes, nosetest based test cases will start appearing in the source tree)
I did take a bit of time to entertain myself with some cool operator overloading (yes, is not a cursed word you know), here's a peek into how it looks:

graph = Graph()
node = graph.new_node()
edge = node.new_edge("edge 1")

dot = graph.as_graphviz() # we can use graphviz for a nice looking graph
as_dict = graph.as_dict()
graph = Graph.from_dict(as_dict)
#easy serialization, generated dicts are not recursive
json.dumps(as_dict)

#Get a path between 2 nodes
a_path = graph['Node 1'] >> graph['Are you sure you want to quit?']

#go from current location to the given node
graph.traveller >> graph['Installation Finished']
(In case you're still thinking >> roughly translates to 'to') 

#create a node that represents the state of the world as it is now
node2 = graph.new_node(from_world=True)

The whole graph library with the crawler it is in itself a general purpose problem solver, with a bit of extra layers it can be used for some really interesting stuff, I'm really looking forward to have some time to play with that (outside of the scope of Murphy,of course).

The user interface is screaming for help, even after I spread requirejs to prevent global namespace pollution it is in a desperate need of a framework, I'm still undecided, current options are between angularjs and emberjs but there are many and I'm not very familiar with them.
Back then, the first UI was done with Tkinter, oh boy... that was painful, then it was a simple notepad based html page but now there are many bits and pieces and needs a good componentization as it is getting out of hands.

And, I am in debt with the 'Dismantler' thingy, I have yet to truly show what it is about, the basic idea of it is to decompose the UI elements in a friendly way for translators, but of course I have yet to find the time to do an experimental demo of it.

Then again, debt must be paid, refactoring and writting test automation code will continue for several days longer.

-Mat


Friday, March 21, 2014

Freshly baked version and some hard stuff

Yez, the github code was updated today with latest and greatest and hopefully not too broken new version, I still have in my todo list the versioning, I promise next time I commit code I'll do it :)

In any case, let's move forward with some really hardcore stuff, it is probable quite safe for you to not to read beyond this point as i'm going to go very technical, but I MUST write this at least for myself.

One of the most important parts of murphy is the ability to crawl the application and make a map or a model out of it, I had split the problem into 4 stages that go from quite simple and most useful with a notable increase in complexity in each stage.

Before going into the stages we need to first need some basic concepts, to begin with the model is a directed graph, each node represents a state while each edge the transition into another state, that's trivial but we're doing a graph of the UI of an application.
In this case a node represents then the state of the UI, in a windowing environment that usually means a window or dialog, edges will then represent what I can do with that dialog (pressing buttons, enter text, tick checkboxes, etc)
Things are not THAT simple, let's see a simple dialog:


If we take literally the image as a state, then each combination of pressed / unpressed check boxes, radios and so will explode the number of different states and images to store, to reduce that the state is parametrized and so the image too, we can then recognize from a screenshot comparing against our parametrized image, a parametrized image looks like this:


It is then a matter of finding an image inside a screenshot of the whole screen, the red areas are ignored during the comparison (as their content can change)
Without going deeper into how and why (or i'll spend the whole night explaining) that is more or less the basics of recognizing the different states.

As we are interested in doing a human readable statechart or graph of the flow this is good enough for most cases. Let's move on to the crawler and the different stages then.


Stage 1: Big splash

The simplest crawler tries to visit as many dialogs / states as possible with as little work as possible, at this stage we're mostly interested to have a bird view of the application, to do so the crawling uses a very simple process, again some simple flow (in text mode this time):


While this strategy gives us fast results we can see that certain things tends to be confusing, in the center of the graph we can see the 'Cancellation Confirmation' and a mess of arrows around, not even all the paths are valid, if you come from the 'Welcome To SuperApp' state you cannot then go to the 'No__Select Installation Directory', it is an invalid flow.
Separating that state requires a more advanced crawler.
This is the crawler that is at the moment in github.


Stage 2: More details but not exhaustive

As we step up to the challenge we want something nicer and we're willing to spend a bit more of machine time, this crawler can decouple this kind of 'reusable dialogs'.
Internally what it does is to associate a state with a context, if there's some invalid case like the above one (from the welcome, press cancel, then no_select installation directory) it opens up as a separate state, now the graph looks like this:


Now we're talking... all paths are valid (or almost, for most cases), as a convention you can see that in the text the __Blah... is a disambiguation of where does it comes from, this is useful when we write manual scripts (one day I may write about that)
Context means (in the context of this discussion) the path we travel to arrive to such node, more specifically only the predecessor node/edge, as we're still under a manageable complexity we only care about the immediate before node.
(And I am happily and proudly showing you the first results of the soon to come version :))


Stage 3 and 4: when hell freezes

This ones are a real pain, we step up the challenge significantly and now the graph is not closed anymore, at this point we care about all the nodes / edges that affects the possible next step. Wha???
Yes, let me translate that, lets suppose that the 2nd node in the graph has 1 checkbox, it can be on or off, and depending on it the 8 node may vary, detecting the flow dependencies beyond the immediately before node. Context is now the full path we travel!

Heavy stuff, and will come, but the plan is to get there gradually since most of the benefits are collected with simpler crawlers there's no hurry to get to stage 3 and 4.
However, the purpose of the stage 3 and 4 is to allow super cool functionality in the workbench, you will then be able with the mouse to instruct testing / exploration in an extremely powerful way, you could then for example do the following things:

  • Get a quick graph (of stage 2 quality)
  • Select areas of the grap
  • for each area you could specify if you want exhaustive exploration (try many different ways to reach the same state)
  • loop areas, for example try a series of values in a field, for example a collection of valid and invalid inputs

And so on, you get the idea right? the potential is really interesting.

But time is up and post is long, so I'll be on my way.

-Mat

PS: I warned you it was going to be heavy stuff...




Thursday, March 20, 2014

A crash but gentle video introduction to murphy.

The promised video has arrived, crash introduction in 4:11 minutes what murphy is about.


(for better resolution watch it at http://youtu.be/zUYmzYI_pvY and put the settings / quality to a high value, better yet if you let me know how to put high res videos in blogger :) )

Some of the things you see in the video are not yet in the source code at github, I still need to check nothing major got broken, some features you see there are in early and experimental stage, you know, the usual disclaimer.
More info to come as soon as I push the changes to github, until then I hope you enjoy the video

-Mat

Thursday, March 13, 2014

Quick update and introducing 'Da skinator'

Yeap, been few days already, but quite hectic ones, I did rushed a few small yet handy features but I wont write about them yet as I'm also working on a small video that shows them and some of the known stuff, hopefully within the next few days will be ready. And yes, spend quite some time paying code debt and sanitizing some code in the web workbench, work is not finished but it was getting out of hands already.

So what's that 'skinator' thingy? well, before going into that you have to pronounce it right, it is like the old Addams Family pinball when it says 'the mamushka', so sounds something like 'Daaa Skiiiinaator'!

Ok, now we know how to say it lets get into the explanation, sometime ago, about a year or so It occurred to me that because the scraper component in Murphy disassembles the user interface I then have a lot of information on how to reconstruct it, so what good is that?
Well, product customization is not a rare thing, co branding and stuff that modifies the original application and produces a custom version of the application has it's uses, too abstract so far? ok, let's look at a screenshot


The popup window shows a reconstructed version of the UI from the information the scraper got from the application (please be gentle, I spend just 2 hours today to do it so it is rather crude at the moment)
You could try out how would it look if you change the font size, colors and so, even cooler, as you watch the whole flow you could see how much of the application would look if you change for example the palette.
The point is that it would allow you to quickly preview mass changes in your application without the need for recompiling or even running it to see how it would look, for that it could then be zipped and sent to the customer for getting early feedback.

It is not a trivial thing to do but it is not that difficult either, anyway I don't have any immediate plans to move forward that functionality but wanted to show other kind of things that are possible to do once we have the model of the application.

Hope next post is about the video being ready.

-Mat


Thursday, March 6, 2014

Thursdays are no wednesdays but still...

I know, I said tuesday or wednesday but it actually happened today, an update of the source code in the git repo.

I got some feedback already about a couple of minor glitches to fix, and the updated I pushed today has some of them already, there's no changelog neither versions yet, it is all 'trunk', sorry for that as I'll work it out in the coming days.

However, I managed to squeeze a nice small addition, you can now have a live view when murphy does it's stuff, here's the shot:


So, in the top-right corner there's a small preview window which constantly updates with what is happening in the virtual machine, useful for debugging / troubleshooting mostly. I'd like in the near future to also integrate it with the other functionality, meaning when you request a machine in a specific state you could see it working it's way to it in the 'Live' window. Probably even taking control of it right from the browser without the need to launch an external client, is not much to do as I have all the building blocks spread around.
The live view also introduced 2 nice things: requirejs (oh boy, how messy javascript can be...) and a small windowing library for html-javascript, it is something I've been poking around some time ago, while html is great for some things it sucks for others so it helps me a lot to have it here. Yez, you  can resize, move the window around, close it and so.
Before I forget, the live preview is disabled by default until it has enough testing, but you can easily turn it on with a parameter in the base_extractor.py

Where are we heading nowadays then? well, first I'd like to get some feedback on the disk / network / registry capturing feature and more particularly how to show the information better, it is quite crude and raw at the moment.
Then? ah, start harnessing the possibilities of that information, meaning capture what happen when the network get's cut and the application tries to use it, sounds simple, but hey, if I cut the network then murphy stops receiving the images and sending the commands! so, it is not so trivial to do and may require a few attempts at it.
Then? arggggg, need to pay back code debt, fix some of the old code which is in bad shape, but that's an ongoing thing in background anyway. I will start converting things as to use requirejs on the javascript side and the inclussion of the pebble library.

Speaking of which, I'll explain a bit better what I *tried* to say in the last post:

>>I wonder if it could be simplified as much as to not to even have to use the get(), that'd be awesome.

What I mean is, that the following code:

@process()
def do_job(foo, bar=0):
    return foo + bar

t = do_job(x, y)

returns a task object, so you must do then t.get()

However, with python is possible to return a proxy object, which in fact can do the get() call for you, to better picture what i'm talking about imagine the following case:

@process()
def calc_a(foo, bar=0):
    return foo + bar
@process()
def calc_b(foo, bar=0):
    ...
@process()
def calc_c(foo, bar=0):
    ...
@process()
def calc_d(foo, bar=0):
    ...

a = calc_a(x, z)
b = calc_b(x, z)
c = calc_c(x, z)
d = calc_d(x, z)

result = (a * b)
if result > pp:
    result += (c * d)
else:
    result -= (d / c)

If calc_X returns a proxy object, the get is performed automatically on first use, and the synchronization is handled automatically for you 'in the background', otherwise the code would look like:

@process()
def calc_a(foo, bar=0):
    return foo + bar
@process()
def calc_b(foo, bar=0):
    ...
@process()
def calc_c(foo, bar=0):
    ...
@process()
def calc_d(foo, bar=0):
    ...

a = calc_a(x, z)
b = calc_b(x, z)
c = calc_c(x, z)
d = calc_d(x, z)

result = (a.get() * b.get())
if result > pp:
    result += (c.get() * d.get())
else:
    result -= (d.get() / c.get())

I find it interesting because nowadays things tend to go into the closure's way, which IMHO are more confusing for this type of cases, the same code with closures 'a la javascript' or dart is indeed longer and a bit more obfuscated.
We did brainstorm further with Matteo and got some other interesting ideas that could be tried, imagine for example the decorator:

@parallel()
def calc_a(foo, bar=0):
    return foo + bar

The 'parallel' implementation could execute it threaded, or as a separate process or even as a remote process on another machine, of course it is not quite trivial to do neither to know beforehand what would be most efficient but there's the challenge! and an interesting one, you could give hints or preferences about where to execute, or collect timming information on the fly and use it later... quite interesting...

But then again, the post is getting quite long and I already started to babble.

-Mat

Monday, March 3, 2014

X ray vision for Murphy, real soon.

Yez, after poking around and some heavy coding, I'm almost ready to roll a new version with experimental support for x-rays.

What's that? as posted earlier, Murphy can see now when an application writes to disk, uses the network or registry and relate that information to the user interface, opening interesting possibilities in the automatic extraction of the user interface, it has some other uses too, for example a user pointed me out that it helps him when he does penetration testing to quickly point part of the surface area for attacks, more to come on that later.

Here's a demo / test app with  latest stuff.



And this is how does it looks if you click on those icons


The feature is still in an early stage so it will change and adapt based on usage and feedback.
Here's a closer look at the icons, which I pulled from the tango project (except for the registry icon that I draw myself)


Took a bit since I fixed other stuff on the way, plan is tomorrow or latest by wednesday to commit it to github. Check then if you're anxious to try it out.

In related news, there's quite some debt to be paid in the code, and just found out a nice piece of code to help me with some parts of it.
The web server does execute some stuff as separated processes as a way to maintain the web server stability and the code is a bit green, never matured enough, so this library (Pebble) is super cool for that, from the page:

@process(timeout=10)
def do_job(foo, bar=0):
    return foo + bar

That's all it takes! then you simply call do_job(1).get() as if it were a normal function!
Just to drool a bit... it even takes care of handling any exception it may raise, including stack trace and bring it back to the caller process, it is just damn cool!
Of course, the fine print is that anything you pass as parameter or return as result must be pickable but then again that is not an issue in most cases.

The decorator does the magic and returns a task object, on the task object you call .get(), that's why instead of
   do_job(1)
I wrote
  do_job(1).get()

I wonder if it could be simplified as much as to not to even have to use the get(), that'd be awesome.

Kudos to Matteo for the nice job there!

-Mat

Wednesday, February 26, 2014

The fine print

I feel i've neglected to mention few things for the uninitiated in Murphy, mostly due to lack of time and the little I had was spent with this new cool feature I'm adding. But fear not as I plan to pay my debt now.

Murphy was born as my wish of getting rid the boring and tedious tasks of testing which could be automatically automated to a high degree (as redundant as that sounds), so it never had a clear direction nor idea of how, but of what to achieve.
As soon as I got the initial sparks a colleague pointed me into the model based approach, which at first I did not wanted to look deeper as to not to pollute my thoughts with preconceived ideas.

On my journey to create my tester automaton I deviated from what model based advocates and literates do, which is to focus in the test case generation, I saw many benefits could be harvested before reaching the state of 'mass generation of test cases' (like model extraction, model comparison, the interactive model and so), even as of today Murphy lacks the kind of traditional test case generation.

So I'll be disappointing model test based advocates and literates, Murphy cannot be considered it (in the traditional sense) however I hope at some point I could claim it does. It is not far from it but my energy at this moment goes into other aspects.

Got that out of my system, so let's go with other warnings :)

There has been a lot of different ideas and approaches tried out in the code, so the code has some debt to be paid, old stuff got forgotten in there, unpolished things and all kind of bad things that can happen when you experiment over the code trunk, heh, i'm totally to blame about it but it is not as bad as it sounds, just so you know if you dig into the code.

Even so it should be said that what is published is still a trimmed down of the original beast, I had to cut down parts that were not ready to be published, experiments in the middle, the self tests, even some functionality like the model comparison had a (bad) scissor cut (it is mostly working, but the real thing works better), I hope to rectify this in future releases, especially the model comparison thingy which is extremely useful.

And before I forget, let me warn you again, it will not work out of the box for every application, in fact very little has been tested with real life applications outside the company I work for, it is expected (to your disappointment) that it is likely not gonna work for you, I hope one day will.

So why bother? well, I did invest a lot into this just because I had fun and believe that there's room for a tool like this, my wish and hope is to inspire others to experiment and do things along the same lines, if I see at least another project inspired by this or get contributions and new cool ideas from the community then all the time I've spent was worth it.

If you reached this point and are reading this, you gotta be quite bored by now, go and give a download at https://github.com/F-Secure and play around with it! or tell me what cool things you're doing!

-Mat

Tuesday, February 25, 2014

And so it happened

Finally, it has been released!

First of all, big thanks to all who made this possible (specially my wife and kids)

Second, my very big disclaimer:

This software is *experimental*, incomplete and is not meant for end users, it is (at least at this moment) for thinkers and doers, it wont go down without a fight and getting your hands dirty.
Last but not least, it comes with no warranty of any kind, needless to say, use it at your own risk.

You can get murphy at https://github.com/F-Secure/murphy

Hope you enjoy poking around with it as much as I did

-Mat

Monday, February 24, 2014

De-teaser and what is coming to town...

Last post I hinted at what it was happening and let the imagination fly, now there's some details:


The Icon pointed by the arrow shows us that the application, after pressing the install button performed something in the hard drive, right click on it and we can see...


So we can see in details what actually happened, file by file, to the degree of details we want to filter.
This is not limited to what an app writes to the hard disk but also changes in the registry and network activity.
What is it used for? well the very first obvious application of it is that the crawler can then reset the whole thing, go to the select directory, leave the hard disk with less space than what the application needs and press the install button, so we can see what would happen if somebody tries to install having little hard disk space.
Same applies with network activity, it would be possible to cut off the network before performing (or during) an action that uses the network and record what happens, etc etc, you get the idea right?

I then experimented with another application, I was a bit surprised as to how eager was to inform thru the network what was happening with it (even before accepting any license terms...).



Cool stuff... like taking an x-ray picture of an app, it is still in quite an early stage but my proof of concepts did work as expected and looks promising. (sorry I did not had enough time for better screenshots)

In related news... nop, the code is not ready yet, buuuuuuuuuuuut keep an eye at https://github.com/F-Secure/murphy because either late today or tomorrow it will happen (if the caffeine is able to keep me up and running for the final checks...)

-Mat





Wednesday, February 19, 2014

An update, a teaser and a request for imagination

Wopsy, more time that I wanted since last update, was busy but good busy, wassup?

I had been itchy about some speed issues with Murphy and the vnc routines i'm using, to begin with there was some strange slowness with the networking code, given that I used out of the box components from python, took some measures and seemed that the urllib and built in web server from python has some weird flushing and networking issues that were slowing things down unnecessary (along with some bugs of my own).
One thing lead to another and to make a long story short I ended up writing down some easy to use yet fast socket routines along with a full vnc client, all in python.
Yes, included in it I wrote from scratch the vnc client library (includes raw, hextile and zlib decompression) and uses the web browser as the UI, this opens the door for doing something I wanted to include in murphy for a long time, a simple record & playback for the more simplest cases.
Buuut, nice tweaks that got along was that I included a protocol register, so you can do neat things like type vnc://192.168.56.1:5900 and it will open a web page with the vnc client pointing to whatever ip address you used in the vnc: protocol, neat! and to my very surprise performance is quite decent (no, you cant play fps games on it...)
The networking and vnc client are yet to be plugged with murphy, and I'm debating myself as to include in the same package or keep them as separate (but needed) library, i'm still undecided.

 As for the teaser, I started working into one ... let's say... interesting feature. It is still taking shape and I'm collecting opinions, suggestions and ideas... so here's a preview...


You have to pay attention to the details, I wont tell you (yet) what i'm doing, however It would be interesting to know what YOU think it will come.

So here's my request for imagination, what do you think it will soon murphy be able to do? what will you want it to do?

Hopefully in a day or two I'll find the time to do some writing about it.

-Mat

Monday, February 10, 2014

Ohio and beyond!

Wow, Mr Murphy was accepted in TAIC PART 2014!

March 31- April 4, Cleveland - Ohio, https://sites.google.com/site/icst2014/home

Thanks to Pekka Aho, Teemu Kanstren and Atif Memon for their work in the paper (mine was just a minor role, kudos should go to them)
If you're around, there is going to be many interesting papers presented, the Murphy one is related to the experience we did in F-Secure while creating it, interesting stuff for QE's

And... no, the source code isn't published yet, hopefully the guys here will finish with the final checks soon.

-Mat

PS: if you're into, there was another published paper in EESSMod 2013 - Experiences and Empirical Studies in Software Modeling at http://ceur-ws.org/Vol-1078/, a direct link to the paper is here again, related to the use of Murphy inside F-Secure

Wednesday, February 5, 2014

Testing in the cloud and writting murphy scripts

Murphy was designed to be able to run tests in a remote machine mainly, while it is still possible to run the tests locally there are many advantages to do it remotely.
Obvious advantages are that you can run as many parallel tests as machines you have access to, and the costs nowadays are dirty cheap (I remember something like 70 usd for an hour of 10.000 cpu's in Amazon or something along those lines)
Even when murphy extracts the model of an application, it does so by running the application to analyze in a remote machine, don't worry, you don't need a cloud to use it if you dont have, a virtual machine using VirtualBox, VmWare or KVM will do just fine.
If you're not familiar with it, there's a lot of magic you can do, like saving the state of the machine and restore it later and other very handy things.

Enough of the cloud, the idea should be somewhat clear already, let's go to the scripts.

While the main focus is to try to test as much as possible without writing test scripts it is still needed in some cases (hopefully less and less as murphy evolves), let's explain it over some sample model:


So how do we write a script over that? pretty simple, what we're looking at is the flow with images, if instead we look at the text view what we see is:


Test scripts are written in python so there's no limit to what you can and cannot do, you simply load a model and call methods. Let's suppose we want to write a test that launches the application and in the main dialog cancels the installation, that script is written as:

from murphy.tester import run_remote_test

def test_install_cancelled(worker):
    worker.In("Node 0").GoTo("Were Sorry You Did Not Installed Us")

if __name__ == '__main__':
    run_remote_test(test_install_cancelled, "abc/my_model.json")

That's it, that will assert everything, the test will fail if anything does not match with what you see in the flow.
Under the hood it has a GPS and will find the route from Node 0 to the installation cancelled dialog for you and execute all the needed intermediate steps, of course you can be verbose and explicitly state step by step, for example

worker.In("Node 0").Do("Launch application").Then("Cancel")

Trying out several invalid installation keys is a simple iterator with the values to test, the concept is still the same.
(sorry about the typo in the graph, I need to fix the example in the source code)

The names you see in the nodes are a best guess from the core extraction routines, but as everything is customization you can rename then to your own liking (specially if they're annoyingly long)

Those are the basics, it can get as complicated as the test cases you want to write.

As for automatic generation of test cases, it is a desired feature but absent at the moment, the idea is to export the model into another open source tool that takes care of that.

Ok then, I gotta think what the next post will be about.

-Mat

Sunday, February 2, 2014

Weekend experiments: android testing and murphy

Yeap, I put together some experiments I had been working on and got the initial support plugged into Murphy, here's a shot:


Still in early stage, but managed to get quite far quite fast, I hope having model extraction and test automation supported in android in a near future.
Android disclaimer: I'm focusing in native applications (java), so if you're developing web view based apps it wont help you there, I already know is a pain to adapt with pre 4.3 android, there's hope that 4.4 may make it easier.

-Mat

Friday, January 31, 2014

More on the workbench, automatic detection and report of application changes, testing with privileged prompts and more.


In short, take a model extraction script as we saw in the previous post, run it against a newer version (presumably the next minor update), hit the 'Compare' link (near the top left corner) and it will tell you what has changed.

Sounds a bit weird but works like a charm, given the model of 2 different versions of the same application it will tell you where it has changed in a (nicely?) formatted report.
What's the use? well, in testing is near a dream tool for me, we work in a continuous integration model, we get several versions of the application per day and we used to have more than 25 persons working over it at the same time, that is without counting UX (user interface) specialist and translators (who also commit or contribute to the changes on a daily basis), keeping track of what changes, where and when was a really difficult thing to do.

Now the paradigm is inverted, when something changes our build management tool is configured to run a model extraction and compares the last 2 versions, if there are differences it generates the report and triggers a notification.
What changes are reported? well if you add a dialog, add / remove a button (or other ui element), change the style like fonts or colors, changes in the texts being displayed, most of the visual changes.

One could say if tests are in place and properly written they should fail and notify, well... in your dreams... most tests did lack a lot of assertions and did little UI asserts. In fact if they did do all the necessary assertions then the maintenance cost would be rather huge.

Anyway the topic can go really deep, enough is to say that there was nothing else needed than the basic extraction script to get alarms displaying what has changed, here's a couple of (not so good) shots of such a report.





Privileged prompts, a pain(?) to automate

Suppose you have your test machine properly configured, at certain point the app may require to do something for which windows security will ask the user to confirm if it wants to proceed:


If it were easy to automate that, security would be a joke (and computer viruses would rejoice at it), the challenge is to automate it in the most realistic way, with real security settings in the test machine.
Writing a kernel driver or other low level solution would be very platform specific, not to mention the care and cost it would require, no problem, murphy does the trick for you.

Here's a simple model that handles the case:


You may notice that from the privileged dialog it only does the Yes / No button, that's because the extraction script instructed to do so, if you want to go crazy you can go as much as you want, for example:

(The script to produce that had about 8 lines of code which I intend to shrink to 3)

Funny thing, after the model was extracted I found 2 things right away:

  • In the "Change when to be notified..." dialog, pressing OK without changing anything required me to confirm the changes, however no changes were made (is at least arguably if that's ok)
  • The certificate information is expired but there was no warning whatsoever from windows
Here's a closer look at the certificate info:


Checked the date and time and they're correctly set to 2014, also tried the same in this very computer and also says it is valid until 2011. If you know why please let me know, to me it looks like a (serious? minor?) security issue, then again I'm not a security expert.

Wrapping up (the post is getting long) you can have a view slightly simplified to avoid too much arrows crossing:


Ok, time to start my weekend, hope I wasn't too confusing.

-Mat

Thursday, January 30, 2014

A tester workbench, an exploratory testing tool

Murphy can do many things but the starting point for it is to know your application, to do so it will try to visit all the dialogs the application has by looking at the screen, click anything it can, enter values and so.
This is called model extraction, it creates an internal representation of the application and the transition between states / dialogs.

An example of a simple application would be:


Most of it is done automagically, you only need to specify the application you want to explore. It cannot guess many things like specific values (think installation keys and so) or specific sequences of actions, however it is fully customizable and those cases can be handled easily by tweaking the extraction script for that application.

The above example was created with 2 lines of code (slightly styled for display purposes):

extractor = base_extractor.BaseExtractor('7zipWinScraper', '7z920.exe')
extractor.add_boundary_node('Browse For Folder')

The very first thing we can do with it is to interactively use that graph, just click what you want to do and it will do it for you.
Let's suppose for example you want to test what happens if the application tries to install in a folder where it does not have access right, to do so, just do this:



Type c:\readonly_folder in the text field (image at right) and press the run button, that's it.
Murphy will do the rest for you, it will create a virtual machine, launch the application type the value and press the install button.

Obviously something bad is going to happen, to see what we just open a vnc or remote desktop connection to that machine and see the fireworks, in this case we got:

Also murphy will tell us that something went wrong:

MurphyConfused: Cannot recognize view Completing The 7_Zip 9_20 Setup Wizard
2014-01-30 21:40:13,410 - root.web_workbench.planner - INFO - Run finished.

This example is rather simple but enough to give us an idea of the power of the tool, we clicked 2 steps but there's no limit to the complexity.

It also comes with a "GPS" included: you don't need to click consecutive steps, in the above example, you could for example do only 1 click in the close button of the installation finished dialog, that will create a virtual machine with a default installation of the 7zip application (yes, there are limits, but they can also be overcome with a bit of code).

But for the same price, we get a nice extra feature: cloud. In the company I work for, we use a cloud for testing (we use https://github.com/F-Secure/dvmps), you're not limited to 1 run at the time, you could ask for 10 machines in 10 different states at the same time and it will all happen at the same time saving a lot of time in many cases, you can read the news while murphy does a good chunk of the tedious work.

Last but not least, you can proofread the dialogs in thumbnails and screens mode (as opposed to a statechart), you can also export the model as a self contained file, quite handy for things like translators so they can have contextual information of the texts (which helps to avoid typical transliteration mistakes)

This is just the tip of the capabilities of murphy, there are many others to mention in future posts :)

-Mat

A simple riddle...

While writting my next post on murphy it occurred me this simple riddle, can you take it?

"You will find it when you can't find it"

It's ok to post "I found it" but please don't post spoilers.
Need a hint? leave a comment... (c'mon, it's easy...)

-Mat

Wednesday, January 29, 2014

About time...!

It took a while, quite a while but it's finally happening, Murphy goes open source!
What would that be? well, it can do things like this:

No, it is not a flowchart drawing tool, those graphs were extracted from the apps automatically, how much effort does it takes? a few lines of python code.
So what is it? an experiment that I started a few years ago at work, it can do various things that are quite helpful like extract the model of an application, generate those graphs, report changes between different versions of an application, a nice interactive exploratory testing tool and a few other things.
More to come, just need to find the time to write it :)

-Mat

PS: no url yet, but soon, quite soon

Updated 15/3
Sources can be found at https://github.com/F-Secure/murphy, more information can be found in the later posts I published, I'll do a summary page about it soonish

Updated 20/3
A 4:11 minutes video can be watched at http://youtu.be/zUYmzYI_pvY