autocluster - a system for automated testing of clustered systems
This is a joint paper with Martin Schwenke of IBM OzLabs
In this talk we will describe a fully automated test environment for
clustered systems. The motivation for building this test environment
was to provide high quality automatic testing of IBMs 'SoFS' (Scale
out File Services) clustered NAS product.
At the core of this test environment is autocluster, a system that can
generate a pristine virtual test cluster from a base image in less
than a minute. A base image built from a distro ISO image can be
automatically built in about 15 minutes from a simple configuration
file, so you can be using autocluster within 30 minutes of downloading
it. The autocluster system uses templates to automatically configure
such complex elements as a clustered filesystem, clustered Samba,
multipath disk access and hierarchical storage management.
On top of autocluster we have built a full test environment that
provides automated testing of NAS functionality, including scripting of
Windows clients, domain controllers and automatic fail-over of cluster
nodes after simulated hardware failure.
Autocluster also makes it simple to create custom environments for
reproducing and debugging individual problems as they are reported.
The system uses KVM via the libvirt command-line utilities. Our
presentation will include demonstrations of how virtual clusters can
be created using autocluster and how those clusters can be used in