BareMetalTesting
Launchpad Entry: other-p-qa-baremetal-testing
Created:
Contributors:
Packages affected:
Summary
Test automation on bare-metal systems
Release Note
This is to enable test automation on bare-mental systems (i.e. without VM support).
Rationale
Too much of our test automation currently relies on KVM/Libvirt, making it difficult to impossible to run on systems that do not support KVM (Arm, low end NAS systems, etc) without a complete rewrite.
User stories
The Acme company wants to do some performance analysis testing on several different platforms to determine the best system that will meet their needs without breaking power/price budgets. They do not have the budget to deploy massive servers to each branch office, just to handle trivial server tasks (file storage, print serving, internal web document serving, etc). The systems they are considering need to be tested under load before a purchase determination can be made. Testing the software stack in a VM will not give them an accurate picture of how each system being considered will operate, and they want to be able to do 24/7 image deployment and load stress testing.
A new company is gearing up production on new servers for the market. These are low powered systems designed to provide low to medium server tasks on an extremely low power budget. The processor for these does not support vitualization, but the power consumption more than makes up for it, as a single 4U system with 20 4-core blades consumes less power than a single 8-core processor from a competing company that does support virtualization. They need to do extensive automated testing to help root out any hardware issues prior to production, part of which involves real-world environment testing. Having the ability to run automated tests already packaged for them would be a great benefit.
A software company, after months of testing software in an automated process based on virtualizing test systems. This has allowed them to speed up testing of their environment by a large factor with a lot of cost savings. However, when they finally release their product to the public, they are inundated with bug reports on real systems because of hardware/software configurations that their virtual environment couldn't account for. The hardware in question is relatively cheap and wouldn't have cost much to be able to add to their testing capabilities, averting a lot of these bug reports. The fixes to get the software running on these systems is also fairly quick, but public opinion has already spread like wildfire, diminishing their reputation.
Assumptions
Design
The environment should be modular so that as new types of systems and tests come online, they can be easily plugged in to the testing framework.
Implementation
This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:
UI Changes
Should cover changes required to the UI, or specific UI that is required to implement this
Code Changes
Code changes should include an overview of what needs to change, and in some cases even the specific details.
Migration
Include:
- data migration, if any
- redirects from old URLs to new ones, if any
- how users will be pointed to the new way of doing things, if necessary.
Test/Demo Plan
It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.
This need not be added or completed until the specification is nearing beta.
Unresolved issues
This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.
BoF agenda and discussion
Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.