(7460) Automation of Multi-Threaded Testing using Byteman
Technical long talk 50 min
Wednesday, 2009-06-24, 14:30 - 15:20, Arena 7
- Andrew Dinn - JBoss, A Division of Red Hat (speaker)
- Jonathan Halliday - JBoss, A Division of Red Hat (co-speaker)
Rate this talk:
Multi-threaded applications are difficult to test automatically. Arbitrary variations in timing mean it is often hard to ensure test runs exhibit the sequence of events required by the test scenario. Conventional approaches to test development, such as writing stub and scaffolding code, do not address this problem. Enforcing timing constraints in test code requires a lot of design and implementation effort and is often difficult to validate without resorting to a debugger. The resulting test program often only tests parts of the application, in isolation and in modified form, and may require major rewriting if the application is changed. Clearly, there is a need for tools to help improve multi-threaded application test development and guarantee the reliability of test execution. Byteman is a bytecode instrumentation tool which simplifies the job of defining, implementing and validating multi-threaded tests by injecting scripted side-effects or trace output into the application code under test. This means whole applications can be tested without the need for rewriting or recompilation. Byteman is useful for testing anything from very simple to very complex multi-threaded applications. For example, it has been used successfully to automate testing of the crash recovery functions of the JBoss Web Service Transactions implementation. Byteman scripts provide a flexible, declarative framework for specifying how such side effects are introduced. Scripts comprise a sequence of Event Condition Action (ECA) rules. An event identifies a location in application code (or even in a JVM method). The rule is triggered when a thread reaches this location. Each rule defines a specific sequence of side effects or actions. These actions are only executed if the rule condition evaluates to true. Rules are parameterized by values derived from the arguments and local variables of their trigger method. This allows rule conditions to be highly selective and enables rule actions to directly manipulate objects currently in use by the triggering thread. Rule actions serve three main purposes. Firstly, they may simply generate trace output. This is useful to identify that the test has followed the desired code path or to display data values employed during execution. Secondly, rules may change the flow of control, e.g. by returning synthetic values or throwing synthetic exceptions. This is useful for simulating the conditions required in the test scenario. Finally, rules may be used to orchestrate the activities of independent threads, introducing orderings or synchronization points or otherwise delaying progress of a thread until other threads have completed related actions. Byteman provides a suite of built-in operations to simplify the tasks of execution tracing and thread orchestration. However, conditions and actions may combine or replace these built-in operations with arbitrary Java expressions, allowing rules to be tailored specifically to meet the needs of the application. This presentation will consider some of the problems which complicate the task of automating testing of multi-threaded applications. It will then introduce Byteman through a simple example program and a rule script which displays a specific timing dependency. The presentation will describe the Byteman scripting language in more detail and provide an overview of how Byteman has been implemented. It will conclude by showing how to customize the scripting language, adding to or replacing the standard built-in operations with new primitives specific to the application under test.