KernelSRU-EC2

Differences between revisions 4 and 6 (spanning 2 versions)
⇤ ← Revision 4 as of 2011-01-05 20:07:27 →
Size: 4934
Editor: pool-173-64-203-126
Comment: ## page was copied from QATeam/KernelSRU
← Revision 6 as of 2011-01-05 20:27:13 → ⇥
Size: 4277
Editor: pool-173-64-203-126
Comment: initial version
Deletions are marked like this. Additions are marked like this.
Line 4: Line 4:
= 1. Context = ---- /!\ '''Edit conflict - other version:''' ----
=== Coverage ===
 - At least test m1.small (i386) and m1.large (amd64) in us-east-1
 - more ideal would be testing across all 4 regions and better coverage across instance types
=== Testing the kernel ===
Note: The following tests are all part of the "QA Regression Test Suite", as defined at the Test Automation Sprint.
==== Autotest ====
1. SSH into it
Line 6: Line 13:
Today, testing a package, a fortiori the kernel, may be slow because of the waiting period that compensate the lack of rigorous test suites to regression-test changes, lack of hardware for QA’ing, we are mainly relying on community for testing, the amount of patches that get rolled can be important,... All of this can lead to a bug with a know fix going unaddressed in Ubuntu for several months. 2. sudo apt-get install build-essential make (kernel.org)
Line 8: Line 15:
We need to shorten the delay of validation for Kernel SRUs. 3. {{{wget http://test.kernel.org/releases/0.12.0/autotest-client-0.12.0.tar.bz2}}} (kernel.org autotest client)
Line 10: Line 17:
 1. tar -xjf the beast
 1. (autotest-1) run {{{autotest-0.12.0/bin/autotest tests/dbench/control > test-dbench.out 2>&1}}}
 1. (autotest-2) run {{{autotest-0.12.0/bin/autotest tests/tbench/control > test-tbench.out 2>&1}}}
 1. (autotest-3) run {{{autotest-0.12.0/bin/autotest tests/signaltest/control > test-signaltest.out 2>&1}}}
 1. (autotest-4) run {{{sudo autotest-0.12.0/bin/autotest tests/tiobench/control > test-tiobench.out 2>&1}}}
 1. (autotest-4) run {{{sudo autotest-0.12.0/bin/autotest tests/real_time_tests/control > test-real_time_tests.out 2>&1}}}
4. save the logs: {{{tar -cjf autotest.tar.bz2 test*.out}}}
Line 11: Line 25:
= 2. Processes to update a Stable Release = 5. from your machine, grab the results tarball: {{{scp ubuntu@<ec2-instance>:autotest.tar.bz2}}}
Line 13: Line 27:
The following current processes exist to publish an update to a stable release:
  * General SRU process: https://wiki.ubuntu.com/StableReleaseUpdates
  * Kernel SRU: https://wiki.ubuntu.com/KernelTeam/KernelUpdates
  * Security Updates: https://wiki.ubuntu.com/SecurityTeam/UpdateProcedures		 ==== QA Regression Testing ====
Line 18: Line 29:
== The Update Process == /!\: We should call this something else, since it's confusing to call it the "QA Regression Testing". It's now only one test in the "QA Regression Test Suite"
1. SSH into it
2. bzr branch lp:qa-regression-testing
 1. cd qa-regression-testing/scripts
 1. run the scripts:
  * {{{python test-kernel-aslr-collisions.py -v }}}
  * {{{python test-kernel.py -v}}}
  * {{{sudo python test-kernel-root-ops.py -v}}}
  * {{{sudo python test-kernel-security.py -v (requires execstack, libcap2-bin)}}}
Line 20: Line 39:
To achieve a high level of quality, the process is broken down into the following stages:
 * Packaging
 * Testing
 * Publication		 ==== LTP =====
Line 25: Line 41:
In the context of Kernel SRUs we are interested by how to achieve steps 2 and 3 and being able to achieve that on short delay. 1. SSH into it
Line 27: Line 43:
=== Testing an SRU === 2. {{{sudo apt-get install ltp}}}
Line 29: Line 45:
The tasks to be fulfilled are:
 * Review the test case and reproduce the issue (Community + Kernel Team)
 * build in a clean build environment (Integrated to the build process)
 * verify the package still installs
 * verify the package upgrades cleanly
 * verify the package still functions properly
 * verify the bug is fixed using the provided test case
 * verify that it introduces no regression		 3. run {{{sudo /usr/lib/ltp/tools/runltp -f ltplite -l /var/cache/ltp/results/runltplite.log -o /var/cache/ltp/results/runltplite-results.txt}}}
Line 38: Line 47:
Today we are relying on the community to achieve theses tasks. The community usually do, to some level, whole or part of tasks 1, 3, 4, 5 and 6, rarely 7.
Step 2 (build) is done automatically by the build environment when the package is published to -proposed.
In any case, we need a human intervention to test the fix and above all that is introduces no regression.		 ---- /!\ '''Edit conflict - your version:''' ----
=== Coverage ===
 - At least test m1.small (i386) and m1.large (amd64) in us-east-1
 - more ideal would be testing across all 4 regions and better coverage across instance types
=== Testing the kernel ===
Note: The following tests are all part of the "QA Regression Test Suite", as defined at the Test Automation Sprint.
==== Autotest ====
1. SSH into it
Line 42: Line 56:
==== The HW Cert Lab / Kernel Automated Tests ====
Using the wide array of hardware in the cert lab to test proposed SRUs is an effective way of addressing the problem of hardware diversity encountered with Kernel SRU.
Kernel Automated Tests, ensures that at minimum, the system boots and can be online. From that point the user can update the system and bring it back to normal operational conditions.		 2. sudo apt-get install build-essential make (kernel.org)
Line 46: Line 58:
Basically, the process of Kernel Automated Tests does the following:
 * We poll -proposed once a day and check if there are changes,
 * If something has changed, we install the system and all everything in -proposed
 * Then the automated tests run against that		 3. {{{wget http://test.kernel.org/releases/0.12.0/autotest-client-0.12.0.tar.bz2}}} (kernel.org autotest client)
Line 51: Line 60:
=== Publication ===  1. tar -xjf the beast
 1. (autotest-1) run {{{autotest-0.12.0/bin/autotest tests/dbench/control > test-dbench.out 2>&1}}}
 1. (autotest-2) run {{{autotest-0.12.0/bin/autotest tests/tbench/control > test-tbench.out 2>&1}}}
 1. (autotest-3) run {{{autotest-0.12.0/bin/autotest tests/signaltest/control > test-signaltest.out 2>&1}}}
 1. (autotest-4) run {{{sudo autotest-0.12.0/bin/autotest tests/tiobench/control > test-tiobench.out 2>&1}}}
 1. (autotest-4) run {{{sudo autotest-0.12.0/bin/autotest tests/real_time_tests/control > test-real_time_tests.out 2>&1}}}
4. save the logs: {{{tar -cjf autotest.tar.bz2 test*.out}}}
Line 53: Line 68:
In the current SRU process, the patched version of the package will be published to -proposed then to -updates once it is tested and validated. 5. from your machine, grab the results tarball: {{{scp ubuntu@<ec2-instance>:autotest.tar.bz2}}}
Line 55: Line 70:
= 3. Regular cadence micro-releases = ==== QA Regression Testing ====
Line 57: Line 72:
== Proposal == /!\: We should call this something else, since it's confusing to call it the "QA Regression Testing". It's now only one test in the "QA Regression Test Suite"
1. SSH into it
2. bzr branch lp:qa-regression-testing
 1. cd qa-regression-testing/scripts
 1. run the scripts:
  * {{{python test-kernel-aslr-collisions.py -v }}}
  * {{{python test-kernel.py -v}}}
  * {{{sudo python test-kernel-root-ops.py -v}}}
  * {{{sudo python test-kernel-security.py -v (requires execstack, libcap2-bin)}}}
Line 59: Line 82:
The proposal is to introduce regular cadence “bug fix/HW enablement” releases and a non-resetting bake window. ==== LTP =====
Line 61: Line 84:
We would have such a release every 2 weeks (frequency to by defined). During this period, if a security fix is released, it's added to the release in -proposed without resetting the the bake window, ensuring that we can engage on a strict ETA. 1. SSH into it
Line 63: Line 86:
Fixes that are not verified at the end of the first week are removed from the update. The author of the report is responsible of the verification. Testing will be performed during the second week. 2. {{{sudo apt-get install ltp}}}
Line 65: Line 88:
The main constraint is that most of the regression testing must be automated. 3. run {{{sudo /usr/lib/ltp/tools/runltp -f ltplite -l /var/cache/ltp/results/runltplite.log -o /var/cache/ltp/results/runltplite-results.txt}}}
Line 67: Line 90:
At a minimum we need to ensure that:
 * The system boots
 * X comes up for the desktop or network comes up for the servers
 * Updates can be pulled and installed with update-manager or apt


== Pros/Cons ==

 * Pros:
  * Strict ETA
  * Ability to use the Automated Kernel Testing
  * Testing limited to the fix but still need regression testing over a large range of setup
  * Keep the work close to the Ubuntu Tree

 * Cons:
  * Do not rely on the community for testing.
  * Testing limited to the internal capacity
  * Strongly rely on the quality and completeness of the QA Regression Test Suite


== Rollback ==

TBD

== Procedure ==

|| When || What || Who ||
|| Packaging || <<BR>> || <<BR>> ||
|| <<BR>> || - Provide a fix<<BR>>- Nominate for SRU<<BR>>- Write a test case <<BR>>- Document any potential regression || Dev team ||
|| Testing || <<BR>> || <<BR>> ||
|| <<BR>> || - Validate the SRU and mandatory information<<BR>>- Publication to -proposed || SRU Reviewing Team ||
|| <<BR>> || Testing version in -proposed || <<BR>> ||
|| <<BR>> || - Test that the patch fixes the issue and check for inadvertent side effect || - Reporter <<BR>>- Community ||
|| <<BR>> || - Run Kernel Automated Testing in Cert Lab || HW Cert Team / Platform QA ||
|| Publishing || <<BR>> || <<BR>> ||
|| <<BR>> ||Publication to -updates || SRU Team ||		 ---- /!\ '''End of edit conflict''' ----

Contents

  1. Coverage
  2. Testing the kernel
  3. Coverage
  4. Testing the kernel

Warning /!\ Edit conflict - other version:

Coverage

  • - At least test m1.small (i386) and m1.large (amd64) in us-east-1 - more ideal would be testing across all 4 regions and better coverage across instance types

Testing the kernel

Note: The following tests are all part of the "QA Regression Test Suite", as defined at the Test Automation Sprint.

Autotest

1. SSH into it

2. sudo apt-get install build-essential make (kernel.org)

3. wget  http://test.kernel.org/releases/0.12.0/autotest-client-0.12.0.tar.bz2 (kernel.org autotest client)

  1. tar -xjf the beast

  2. (autotest-1) run autotest-0.12.0/bin/autotest tests/dbench/control > test-dbench.out 2>&1

  3. (autotest-2) run autotest-0.12.0/bin/autotest tests/tbench/control > test-tbench.out 2>&1

  4. (autotest-3) run autotest-0.12.0/bin/autotest tests/signaltest/control > test-signaltest.out 2>&1

  5. (autotest-4) run sudo autotest-0.12.0/bin/autotest tests/tiobench/control > test-tiobench.out 2>&1

  6. (autotest-4) run sudo autotest-0.12.0/bin/autotest tests/real_time_tests/control > test-real_time_tests.out 2>&1

4. save the logs: tar -cjf autotest.tar.bz2 test*.out

5. from your machine, grab the results tarball: scp ubuntu@<ec2-instance>:autotest.tar.bz2

QA Regression Testing

/!\: We should call this something else, since it's confusing to call it the "QA Regression Testing". It's now only one test in the "QA Regression Test Suite" 1. SSH into it 2. bzr branch lp:qa-regression-testing

  1. cd qa-regression-testing/scripts
  2. run the scripts:

    • python test-kernel-aslr-collisions.py -v 

    • python test-kernel.py -v

    • sudo python test-kernel-root-ops.py -v

    • sudo python test-kernel-security.py -v (requires execstack, libcap2-bin)

==== LTP =====

1. SSH into it

2. sudo apt-get install ltp

3. run sudo /usr/lib/ltp/tools/runltp -f ltplite -l /var/cache/ltp/results/runltplite.log -o /var/cache/ltp/results/runltplite-results.txt

Warning /!\ Edit conflict - your version:

Coverage

  • - At least test m1.small (i386) and m1.large (amd64) in us-east-1 - more ideal would be testing across all 4 regions and better coverage across instance types

Testing the kernel

Note: The following tests are all part of the "QA Regression Test Suite", as defined at the Test Automation Sprint.

Autotest

1. SSH into it

2. sudo apt-get install build-essential make (kernel.org)

3. wget  http://test.kernel.org/releases/0.12.0/autotest-client-0.12.0.tar.bz2 (kernel.org autotest client)

  1. tar -xjf the beast

  2. (autotest-1) run autotest-0.12.0/bin/autotest tests/dbench/control > test-dbench.out 2>&1

  3. (autotest-2) run autotest-0.12.0/bin/autotest tests/tbench/control > test-tbench.out 2>&1

  4. (autotest-3) run autotest-0.12.0/bin/autotest tests/signaltest/control > test-signaltest.out 2>&1

  5. (autotest-4) run sudo autotest-0.12.0/bin/autotest tests/tiobench/control > test-tiobench.out 2>&1

  6. (autotest-4) run sudo autotest-0.12.0/bin/autotest tests/real_time_tests/control > test-real_time_tests.out 2>&1

4. save the logs: tar -cjf autotest.tar.bz2 test*.out

5. from your machine, grab the results tarball: scp ubuntu@<ec2-instance>:autotest.tar.bz2

QA Regression Testing

/!\: We should call this something else, since it's confusing to call it the "QA Regression Testing". It's now only one test in the "QA Regression Test Suite" 1. SSH into it 2. bzr branch lp:qa-regression-testing

  1. cd qa-regression-testing/scripts
  2. run the scripts:

    • python test-kernel-aslr-collisions.py -v 

    • python test-kernel.py -v

    • sudo python test-kernel-root-ops.py -v

    • sudo python test-kernel-security.py -v (requires execstack, libcap2-bin)

==== LTP =====

1. SSH into it

2. sudo apt-get install ltp

3. run sudo /usr/lib/ltp/tools/runltp -f ltplite -l /var/cache/ltp/results/runltplite.log -o /var/cache/ltp/results/runltplite-results.txt

Warning /!\ End of edit conflict

QATeam/KernelSRU-EC2 (last edited 2011-01-06 11:19:54 by pc-2-14-161-190)