(7460) Automation of Multi-Threaded Testing using Byteman

Technical long talk 50 min

Wednesday, 2009-06-24, 14:30 - 15:20, Arena 7

• Andrew Dinn - JBoss, A Division of Red Hat (speaker)
• Jonathan Halliday - JBoss, A Division of Red Hat (co-speaker)

Topics

• Testing tools

Download the presentation

Abstract

Multi-threaded applications are difficult to test automatically. Arbitrary 
variations in timing mean it is often hard to ensure test runs exhibit the 
sequence of events required by the test scenario. Conventional approaches to 
test development, such as writing stub and scaffolding code, do not address this 
problem. Enforcing timing constraints in test code requires a lot of design and 
implementation effort and is often difficult to validate without resorting to a 
debugger. The resulting test program often only tests parts of the application, 
in isolation and in modified form, and may require major rewriting if the 
application is changed. Clearly, there is a need for tools to help improve 
multi-threaded application test development and guarantee the reliability of 
test execution.

Byteman is a bytecode instrumentation tool which simplifies the job of defining, 
implementing and validating multi-threaded tests by injecting scripted 
side-effects or trace output into the application code under test. This means 
whole applications can be tested without the need for rewriting or 
recompilation. Byteman is useful for testing anything from very simple to very 
complex multi-threaded applications. For example, it has been used successfully 
to automate testing of the crash recovery functions of the JBoss Web Service 
Transactions implementation.

Byteman scripts provide a flexible, declarative framework for specifying how 
such side effects are introduced. Scripts comprise a sequence of Event Condition 
Action (ECA) rules. An event identifies a location in application code (or even 
in a JVM method). The rule is triggered when a thread reaches this location. 
Each rule defines a specific sequence of side effects or actions. These actions 
are only executed if the rule condition evaluates to true. Rules are 
parameterized by values derived from the arguments and local variables of their 
trigger method. This allows rule conditions to be highly selective and enables 
rule actions to directly manipulate objects currently in use by the triggering 
thread. 

Rule actions serve three main purposes. Firstly, they may simply generate trace 
output. This is useful to identify that the test has followed the desired code 
path or to display data values employed during execution. Secondly, rules may 
change the flow  of control, e.g. by returning synthetic values or throwing 
synthetic exceptions. This is useful for simulating the conditions required in 
the test scenario. Finally, rules may be used to orchestrate the activities of 
independent threads, introducing orderings or synchronization points or 
otherwise delaying progress of a thread until other threads have completed 
related actions.

Byteman provides a suite of built-in operations to simplify the tasks of 
execution tracing and thread orchestration. However, conditions and actions may 
combine or replace these built-in operations with arbitrary Java expressions, 
allowing rules to be tailored specifically to meet the needs of the 
application.

This presentation will consider some of the problems which complicate the task 
of automating testing of multi-threaded applications. It will then introduce 
Byteman through a simple example program and a rule script which displays a 
specific timing dependency. The presentation will describe the Byteman scripting 
language in more detail and provide an overview of how Byteman has been 
implemented. It will conclude by showing how to customize the scripting 
language, adding to or replacing the standard built-in operations with new 
primitives specific to the application under test.