TestingServerHardware

Revision 4 as of 2005-11-01 18:20:43

Clear message

Summary

We need to do a better job of testing server hardware. To do this we need to:

  • Get up-to date hardware from IBM / HP / Sun / Apple / Dell to certify against Ubuntu Server 6.04
  • Create a comprehensive server test suite for hardware recognition and stress testing
  • Create an easy way to support and encourage community server testing for extra bug reports

Rationale

We need to guarantee that stuff works !!

Use cases

Sophia wants to help test Dapper Drake on her server hardware.

Jeff has some obscure raid hardware on his machine, and he wants to look up on the web and find out if it will work with Dapper Drake.

Roberta wants to buy a bunch of high end servers that are certified to work with Dapper.

Scope

We would like to certify on 25 machines from the above companies in the Dapper timeframe, and we would like to have community testing of as many server configurations as possible. We need a way to record and track user reports that is seprate from the way we do official server certification.

We would also like to have a large number of users test their own server hardware and report the results back to Canonical.

Implementation

We may want to produce install CD's tailored to specific "certified" hardware, these cd's would be avlalable for purchase by enterprise customers.

We will post the results of user testing to the internet so end users can see the results that other have found.

We need to investigate the details of using the single Harvard site for testing and certifiation certs. We will need infrastructure to connect to these servers remotely, so we can use them for testing.

  • Create a way to track community server test results (talk to ogra about intigration with hwdb)

Code

Create a test suite that runs in debian-installer rescue mode.

The Server Test Suite will ask the users for answers a few questions (which hardware does the system really have, vs. what the system thinks it has), email the result to us => done.

Test Suite Should Contain:

  • basic hardware recognition
  • Userspace tools for hardware configuration (raid controllers, etc)
  • Hot plug systems for blades (CPU, memory)
  • performance (memory bandwidth, disk, CPU, whatever)
  • include database workload or equivalent for high-level measurements
  • Burn-in, long term work load testing for stability
  • multi-system option for network testing (throughput, make sure that the network adapter doesn't go belly-up under load)
  • Install test, to make sure system can actually have system running.

Possible tools:

  • stress(1), package 'stress': tests I/O, CPU, memory
  • Bonnie for storage testing
  • iperf or netpipe-tcp for network stuff: evaluate the two
  • Module PCI tables for checking hardware support (needs code)
  • Stuff used for the test suite will need to produce udebs for use in d-i rescue mode

Test suite might take up to a week to run full burn-in tests, but will need to have a quicker test mode if users are going to test their own hardware and provide us with results.

Each test will run serially, with a final full stress test of all systems.

The inculuded test tools will need a very simple UI that will step through the tests and output the results in a format that can be reviewed by the testing team. Test suite should be automated, and allow for the test operator to perform some out-of-band tests to get details on failures. It should also provide an easy method of shipping the results back to Canonical.

Timeframe

It would be at least 6-8 weeks before hardware could start shipping from vendors to the hypothetical test centre. This means the server test suite needs to be completed by the end of the year.

Outstanding issues

It is difficult to reconcile Canonical's globally distributed development environment with Hardware Vendor's desire to ship hardware to a single location for testing purposes, however Harvard Computer Society has offered to support a testing facility, including staff to process inventory, run the testing suite , and catalog results. We would also have external IP access to the machines, plus external remote console for the systems (most/all of them?) that support ILO/LOM, etc.

Contact: Ivan Krstic <krstic@hcs.harvard.edu> ('krstic' on Launchpad)

Some hardware configurations may require non-distributable software to support (e.g. RAID). Malcolm will need to talk to vendors about being able to distribute those tools as packages in Ubuntu. In the cases where those tools are undistributable (which is the case with many of them), Malcolm will be petitioning vendors to spondor us creating custom CD images for their own distribution with their hardware.

BoF agenda and discussion

A different BOF needs to be scheduled to discuss improving/changing debian-installer's rescue mode to make server testing and rescue stuff less painful.