Wednesday, February 26, 2014

The fine print

I feel i've neglected to mention few things for the uninitiated in Murphy, mostly due to lack of time and the little I had was spent with this new cool feature I'm adding. But fear not as I plan to pay my debt now.

Murphy was born as my wish of getting rid the boring and tedious tasks of testing which could be automatically automated to a high degree (as redundant as that sounds), so it never had a clear direction nor idea of how, but of what to achieve.
As soon as I got the initial sparks a colleague pointed me into the model based approach, which at first I did not wanted to look deeper as to not to pollute my thoughts with preconceived ideas.

On my journey to create my tester automaton I deviated from what model based advocates and literates do, which is to focus in the test case generation, I saw many benefits could be harvested before reaching the state of 'mass generation of test cases' (like model extraction, model comparison, the interactive model and so), even as of today Murphy lacks the kind of traditional test case generation.

So I'll be disappointing model test based advocates and literates, Murphy cannot be considered it (in the traditional sense) however I hope at some point I could claim it does. It is not far from it but my energy at this moment goes into other aspects.

Got that out of my system, so let's go with other warnings :)

There has been a lot of different ideas and approaches tried out in the code, so the code has some debt to be paid, old stuff got forgotten in there, unpolished things and all kind of bad things that can happen when you experiment over the code trunk, heh, i'm totally to blame about it but it is not as bad as it sounds, just so you know if you dig into the code.

Even so it should be said that what is published is still a trimmed down of the original beast, I had to cut down parts that were not ready to be published, experiments in the middle, the self tests, even some functionality like the model comparison had a (bad) scissor cut (it is mostly working, but the real thing works better), I hope to rectify this in future releases, especially the model comparison thingy which is extremely useful.

And before I forget, let me warn you again, it will not work out of the box for every application, in fact very little has been tested with real life applications outside the company I work for, it is expected (to your disappointment) that it is likely not gonna work for you, I hope one day will.

So why bother? well, I did invest a lot into this just because I had fun and believe that there's room for a tool like this, my wish and hope is to inspire others to experiment and do things along the same lines, if I see at least another project inspired by this or get contributions and new cool ideas from the community then all the time I've spent was worth it.

If you reached this point and are reading this, you gotta be quite bored by now, go and give a download at https://github.com/F-Secure and play around with it! or tell me what cool things you're doing!

-Mat

Tuesday, February 25, 2014

And so it happened

Finally, it has been released!

First of all, big thanks to all who made this possible (specially my wife and kids)

Second, my very big disclaimer:

This software is *experimental*, incomplete and is not meant for end users, it is (at least at this moment) for thinkers and doers, it wont go down without a fight and getting your hands dirty.
Last but not least, it comes with no warranty of any kind, needless to say, use it at your own risk.

You can get murphy at https://github.com/F-Secure/murphy

Hope you enjoy poking around with it as much as I did

-Mat

Monday, February 24, 2014

De-teaser and what is coming to town...

Last post I hinted at what it was happening and let the imagination fly, now there's some details:


The Icon pointed by the arrow shows us that the application, after pressing the install button performed something in the hard drive, right click on it and we can see...


So we can see in details what actually happened, file by file, to the degree of details we want to filter.
This is not limited to what an app writes to the hard disk but also changes in the registry and network activity.
What is it used for? well the very first obvious application of it is that the crawler can then reset the whole thing, go to the select directory, leave the hard disk with less space than what the application needs and press the install button, so we can see what would happen if somebody tries to install having little hard disk space.
Same applies with network activity, it would be possible to cut off the network before performing (or during) an action that uses the network and record what happens, etc etc, you get the idea right?

I then experimented with another application, I was a bit surprised as to how eager was to inform thru the network what was happening with it (even before accepting any license terms...).



Cool stuff... like taking an x-ray picture of an app, it is still in quite an early stage but my proof of concepts did work as expected and looks promising. (sorry I did not had enough time for better screenshots)

In related news... nop, the code is not ready yet, buuuuuuuuuuuut keep an eye at https://github.com/F-Secure/murphy because either late today or tomorrow it will happen (if the caffeine is able to keep me up and running for the final checks...)

-Mat





Wednesday, February 19, 2014

An update, a teaser and a request for imagination

Wopsy, more time that I wanted since last update, was busy but good busy, wassup?

I had been itchy about some speed issues with Murphy and the vnc routines i'm using, to begin with there was some strange slowness with the networking code, given that I used out of the box components from python, took some measures and seemed that the urllib and built in web server from python has some weird flushing and networking issues that were slowing things down unnecessary (along with some bugs of my own).
One thing lead to another and to make a long story short I ended up writing down some easy to use yet fast socket routines along with a full vnc client, all in python.
Yes, included in it I wrote from scratch the vnc client library (includes raw, hextile and zlib decompression) and uses the web browser as the UI, this opens the door for doing something I wanted to include in murphy for a long time, a simple record & playback for the more simplest cases.
Buuut, nice tweaks that got along was that I included a protocol register, so you can do neat things like type vnc://192.168.56.1:5900 and it will open a web page with the vnc client pointing to whatever ip address you used in the vnc: protocol, neat! and to my very surprise performance is quite decent (no, you cant play fps games on it...)
The networking and vnc client are yet to be plugged with murphy, and I'm debating myself as to include in the same package or keep them as separate (but needed) library, i'm still undecided.

 As for the teaser, I started working into one ... let's say... interesting feature. It is still taking shape and I'm collecting opinions, suggestions and ideas... so here's a preview...


You have to pay attention to the details, I wont tell you (yet) what i'm doing, however It would be interesting to know what YOU think it will come.

So here's my request for imagination, what do you think it will soon murphy be able to do? what will you want it to do?

Hopefully in a day or two I'll find the time to do some writing about it.

-Mat

Monday, February 10, 2014

Ohio and beyond!

Wow, Mr Murphy was accepted in TAIC PART 2014!

March 31- April 4, Cleveland - Ohio, https://sites.google.com/site/icst2014/home

Thanks to Pekka Aho, Teemu Kanstren and Atif Memon for their work in the paper (mine was just a minor role, kudos should go to them)
If you're around, there is going to be many interesting papers presented, the Murphy one is related to the experience we did in F-Secure while creating it, interesting stuff for QE's

And... no, the source code isn't published yet, hopefully the guys here will finish with the final checks soon.

-Mat

PS: if you're into, there was another published paper in EESSMod 2013 - Experiences and Empirical Studies in Software Modeling at http://ceur-ws.org/Vol-1078/, a direct link to the paper is here again, related to the use of Murphy inside F-Secure

Wednesday, February 5, 2014

Testing in the cloud and writting murphy scripts

Murphy was designed to be able to run tests in a remote machine mainly, while it is still possible to run the tests locally there are many advantages to do it remotely.
Obvious advantages are that you can run as many parallel tests as machines you have access to, and the costs nowadays are dirty cheap (I remember something like 70 usd for an hour of 10.000 cpu's in Amazon or something along those lines)
Even when murphy extracts the model of an application, it does so by running the application to analyze in a remote machine, don't worry, you don't need a cloud to use it if you dont have, a virtual machine using VirtualBox, VmWare or KVM will do just fine.
If you're not familiar with it, there's a lot of magic you can do, like saving the state of the machine and restore it later and other very handy things.

Enough of the cloud, the idea should be somewhat clear already, let's go to the scripts.

While the main focus is to try to test as much as possible without writing test scripts it is still needed in some cases (hopefully less and less as murphy evolves), let's explain it over some sample model:


So how do we write a script over that? pretty simple, what we're looking at is the flow with images, if instead we look at the text view what we see is:


Test scripts are written in python so there's no limit to what you can and cannot do, you simply load a model and call methods. Let's suppose we want to write a test that launches the application and in the main dialog cancels the installation, that script is written as:

from murphy.tester import run_remote_test

def test_install_cancelled(worker):
    worker.In("Node 0").GoTo("Were Sorry You Did Not Installed Us")

if __name__ == '__main__':
    run_remote_test(test_install_cancelled, "abc/my_model.json")

That's it, that will assert everything, the test will fail if anything does not match with what you see in the flow.
Under the hood it has a GPS and will find the route from Node 0 to the installation cancelled dialog for you and execute all the needed intermediate steps, of course you can be verbose and explicitly state step by step, for example

worker.In("Node 0").Do("Launch application").Then("Cancel")

Trying out several invalid installation keys is a simple iterator with the values to test, the concept is still the same.
(sorry about the typo in the graph, I need to fix the example in the source code)

The names you see in the nodes are a best guess from the core extraction routines, but as everything is customization you can rename then to your own liking (specially if they're annoyingly long)

Those are the basics, it can get as complicated as the test cases you want to write.

As for automatic generation of test cases, it is a desired feature but absent at the moment, the idea is to export the model into another open source tool that takes care of that.

Ok then, I gotta think what the next post will be about.

-Mat

Sunday, February 2, 2014

Weekend experiments: android testing and murphy

Yeap, I put together some experiments I had been working on and got the initial support plugged into Murphy, here's a shot:


Still in early stage, but managed to get quite far quite fast, I hope having model extraction and test automation supported in android in a near future.
Android disclaimer: I'm focusing in native applications (java), so if you're developing web view based apps it wont help you there, I already know is a pain to adapt with pre 4.3 android, there's hope that 4.4 may make it easier.

-Mat