TestingQA-ISOImages
|
Size: 3119
Comment:
|
← Revision 13 as of 2011-11-14 22:56:25 ⇥
Size: 4517
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 4: | Line 4: |
| The purpose of this page is to facilitate those not currently assisting Ubuntu Studio so that they can test QA ISO images by defining QA ISO image testing, explaining the goal of testing, providing links to the QA website, and outlining some basic testing steps. | The purpose of this page is to provide a very "top level" explanation of the QA ISO testing process to help those not currently testing QA ISO images to do so. |
| Line 9: | Line 9: |
| QA ISO testing is regular testing performed per the release schedule (see the [[https://wiki.ubuntu.com/PrecisePangolin/ReleaseSchedule|12.04 LTS 'Precise Pangolin' schedule]]). It is also called ''milestone'' testing because it occurs at the specific milestone during the release schedule. | QA ISO image testing ca be defined as scheduled testing done on specific [[http://en.wikipedia.org/wiki/ISO_image|ISO images]] of short availability for [[http://en.wikipedia.org/wiki/Quality_assurance|quality assurances (QA)]] purposes, i.e. to make sure the product is fit for purpose and mistakes are eliminated, using specific use case testing procedures. |
| Line 11: | Line 11: |
| Describing the process in broad terms, to test a QA ISO image a tester will download an ISO image, mount the image in VM or burn the image and install on spare hardware, and perform various tests. | = Purpose = The purpose of QA ISO image testing is to make sure the product is fit for it's intended purpose and mistakes are eliminated. |
| Line 13: | Line 14: |
| These tests may include, but are not limited to: * validate the ISO installed and the OS is operational * confirm key applications work * check that specific settings are correct. |
In the first case (fit for intended purpose) one could evaluate the ISO image by the following primary considerations: * can the ISO image be downloaded? * does the ISO image match the associated MD5 sum? * can the ISO image be installed? * is the operating system operational once installed? * do applications start? |
| Line 18: | Line 21: |
| = Purpose === The purpose of QA ISO testing is to validate the ISO images are installable but also to find as many bugs as possible throughout the release cycle. We perform this testing at multiple milestones so that bugs are found and reported as early as possible during the cycle to give more time to fix them. Testing is required throughout the cycle because new bugs may be introduced as changes are made. |
The second case (mistakes eliminated) one could evaluate the ISO image by the following secondary (and perhaps tertiary) considerations: * do applications behave as expected (i.e. not crash or have strange behavior)? * are settings correct and functional? |
| Line 21: | Line 25: |
| For more information about this see the [[https://wiki.ubuntu.com/Testing/ISO/Milestones|QA testing milestone wiki page]]. | The above are intended as examples of conditions that could be tested during QA and a definitive or complete list of required test cases. More information can be found at the [[http://testcases.qa.ubuntu.com/|Ubuntu QA test cases page]]. |
| Line 24: | Line 30: |
| The QA ISO images are only available during specific milestones (i.e. ''alpha1'', ''alpha2'', ''beta1'', etc) from the [[http://iso.qa.ubuntu.com/qatracker/build/ubuntustudio/all|QA ISO Testing wiki page]]. | QA ISO images are available at the [[http://iso.qa.ubuntu.com/|QA ISO testing tracker website]] only before [[https://wiki.ubuntu.com/Testing/ISO/Milestones|specific milestones]] per the release schedule ([[https://wiki.ubuntu.com/PrecisePangolin/ReleaseSchedule|reference 12.04 LTS 'Precise Pangolin' schedule]]). |
| Line 26: | Line 32: |
| These images will only be '''available for a few days''' at each milestone and all images need at least a single test performed or the next ISO image may not be available. Therefore it is critical for many testers to be involved to guarantee that all images are tested! | The images for each milestone will '''only be available for a few days for testing'''. If an image does not achieve all test it might not be release for the next milestone, therefore it is critical for many testers to be involved to guarantee that all images are tested! |
| Line 28: | Line 34: |
| Testers are encouraged to sign up for particular tests (e.g. Ubuntu Studio 64 bit images) and receive emails when images are available. | Ubuntu Studio specific ISO image for testing can be found at the [[http://iso.qa.ubuntu.com/qatracker/build/ubuntustudio/all|Ubuntu Studio QA ISO Testing wiki page]]. |
| Line 30: | Line 36: |
| = Test Location = Ubuntu Studio specific QA ISO images can be found at the [[http://iso.qa.ubuntu.com/qatracker/build/ubuntustudio/all|Ubuntu Studio QA ISO image page]]. Test reports are also filed at this page. |
= Process = ISO images can be tested with using a [[https://help.ubuntu.com/community/VirtualMachines|virtual machine (VM)]] or using spare computers (also know as ''bare iron'') and follow slightly different processes. |
| Line 33: | Line 39: |
| Again, testers are highly encouraged to sing up for specific tests for immediate notification of image availability. Testers signed up for tests will also receive links and additional download instructions in each email. | A generalized work flow for each is described below: == Testing with VM == * download or [[https://help.ubuntu.com/community/ZsyncCdImage|zsync]] ISO image * install virtual machine application * setup and mount ISO image in VM (e.g. [[https://wiki.ubuntu.com/Testing/VirtualBox|virtual box tutorial]]) * complete installation and test per required test cases == Testing on Spare Computer == * download or [[https://help.ubuntu.com/community/ZsyncCdImage|zsync]] ISO image * [[https://help.ubuntu.com/community/BurningIsoHowto|burn ISO image to CD or DVD]] * place CD or DVD in test machine and reboot machine * complete installation and test per required test cases '''NB: installing this way is a potentially destructive process, backup any important data before installation''' While VM has the benefit of ease, testing on bare iron is generally preferred for accurate testing. |
| Line 36: | Line 57: |
| procedure outline - https://wiki.ubuntu.com/Testing/ISO/Procedures | '''Sign Up''' - to receive email notification of Ubuntu Studio image/test availability go to the [[http://iso.qa.ubuntu.com/qatracker/build/ubuntustudio/all|Ubuntu Studio ISO image page]], click on your architecture, and sign up for a test. note: a QA account is required to sign up for tests. |
| Line 38: | Line 59: |
| virtual box instructions - https://wiki.ubuntu.com/Testing/VirtualBox | '''Read About the Process''' - you can read more about the process at the [[https://wiki.ubuntu.com/Testing/ISO/Procedures|ISO Testing wiki page]] and the [[http://qa.ubuntu.com/testing/iso-testing/|QA ISO testing page]]. |
| Line 40: | Line 61: |
| ask on irc channel - #ubuntustudio-devel | '''Additional Reference Material''' - visit [[https://wiki.ubuntu.com/Testing/ISO|ISO testing wiki page]] for additional information. |
| Line 42: | Line 63: |
| contact - scottalavender@gmail.com | '''Ubuntu Studio IRC Support''' - visit us on [[http://freenode.net/|Freenode]] in the #ubuntustudio-devel channel |
Abstract
The purpose of this page is to provide a very "top level" explanation of the QA ISO testing process to help those not currently testing QA ISO images to do so.
Testing QA ISO images is an important, pervasive, and a required step in the release process and everyone can help.
Definition
QA ISO image testing ca be defined as scheduled testing done on specific ISO images of short availability for quality assurances (QA) purposes, i.e. to make sure the product is fit for purpose and mistakes are eliminated, using specific use case testing procedures.
Purpose
The purpose of QA ISO image testing is to make sure the product is fit for it's intended purpose and mistakes are eliminated.
In the first case (fit for intended purpose) one could evaluate the ISO image by the following primary considerations:
- can the ISO image be downloaded?
- does the ISO image match the associated MD5 sum?
- can the ISO image be installed?
- is the operating system operational once installed?
- do applications start?
The second case (mistakes eliminated) one could evaluate the ISO image by the following secondary (and perhaps tertiary) considerations:
- do applications behave as expected (i.e. not crash or have strange behavior)?
- are settings correct and functional?
The above are intended as examples of conditions that could be tested during QA and a definitive or complete list of required test cases.
More information can be found at the Ubuntu QA test cases page.
Image Availability
QA ISO images are available at the QA ISO testing tracker website only before specific milestones per the release schedule (reference 12.04 LTS 'Precise Pangolin' schedule).
The images for each milestone will only be available for a few days for testing. If an image does not achieve all test it might not be release for the next milestone, therefore it is critical for many testers to be involved to guarantee that all images are tested!
Ubuntu Studio specific ISO image for testing can be found at the Ubuntu Studio QA ISO Testing wiki page.
Process
ISO images can be tested with using a virtual machine (VM) or using spare computers (also know as bare iron) and follow slightly different processes.
A generalized work flow for each is described below:
Testing with VM
download or zsync ISO image
- install virtual machine application
setup and mount ISO image in VM (e.g. virtual box tutorial)
- complete installation and test per required test cases
Testing on Spare Computer
download or zsync ISO image
- place CD or DVD in test machine and reboot machine
- complete installation and test per required test cases
NB: installing this way is a potentially destructive process, backup any important data before installation
While VM has the benefit of ease, testing on bare iron is generally preferred for accurate testing.
Get Involved
Sign Up - to receive email notification of Ubuntu Studio image/test availability go to the Ubuntu Studio ISO image page, click on your architecture, and sign up for a test. note: a QA account is required to sign up for tests.
Read About the Process - you can read more about the process at the ISO Testing wiki page and the QA ISO testing page.
Additional Reference Material - visit ISO testing wiki page for additional information.
Ubuntu Studio IRC Support - visit us on Freenode in the #ubuntustudio-devel channel
UbuntuStudio/TestingQA-ISOImages (last edited 2011-11-14 22:56:25 by 17)