Contributing to Ubuntu - 2 - Ubuntu Quality
by Simon Quigley
For the past nine months, I have done a couple forms of QA for the Ubuntu project and flavors. In this blog post, I plan on highlighting some common practices and how I contributed.
ISO QA Test
Download the image
For demonstration purposes, I'll grab the Lubuntu daily image, which as I'm writing this, is the Yakkety daily image. First, navigate to cdimage.ubuntu.com, it should look similar to the below screenshot:
Since in this case we are testing Lubuntu, select lubuntu/, the page should look like this after loading:
Lubuntu is a special case.
They have desktop images and alternate images1.
The desktop image is the same across all flavors, while the alternate image uses the Debian installer and requires less RAM to run.
In this case, let's test the alternate image, so select daily/, the page should look like this after loading:
In the above screenshot, I have noted a few things, here is the purpose of each directory2:
20160425/ - I am writing this on April 26, 2016 (20160426), but this directory exists to ensure we have an image from the previous day for testing purposes and in the odd case that something gets messed up.
20160426/ - These are today's images
current/ - All of the images go through some automatic testing before proceeding to this directory, these are the last images that passed these tests.
pending/ - This is the directory that holds the images that are either currently testing or that haven't passed the tests, and this is usually the same files as 20160426/.
We are going to grab the image from the current/ directory.
The page for current/ should look like the following screenshot3:
Scroll down and you should see the following:
Listed are three architectures: amd64 (which is 64-bit), i386 (which is 32-bit), and PowerPC (old Macintosh machines).
Other Ubuntu flavors usually have only amd64 and i386, but Lubuntu has a PowerPC image as well.
We have six (6) files for each architecture:
*.OVERSIZED- irrelevant right now, you do not need to worry about it.*.iso- the image file.*.iso.zsync- the zsync file for the image.*.jigdo- the Jigdo file.*.list- the list file.*.template- the jigdo template file.
I will show you how to utilize two (2) of the six (6) files in this directory to download the images.
Ensure the zsync package is installed on your system before continuing.
To download the image, yes, you can use your web browser to download the image.
But there is a better way, using zsync.
zsync allows you to download the image, but when the image changes, it will automatically merge the changes with the existing image that you have.
Since it is the same directory and image name for the whole development cycle, you can safely use one link until the development release gets released.
I'll create a directory to put the images:
$ mkdir daily-images && cd daily-images
Copy the zsync link for your architecture and paste it into the terminal, for example:
$ zsync http://cdimage.ubuntu.com/lubuntu/daily-live/current/yakkety-desktop-amd64.iso.zsync
When you press Enter, the image will download.
When this is done, you should have the image ready to go.
Getting set up with the ISO QA Tracker
Navigate to iso.qa.ubuntu.com in your browser, you should see the following:
On the left side, click the Log in button.
You should be brought to a page that looks like this:
If you already have an Ubuntu One account, log in with your credentials and press the Log In button.
If you don't have an account, create one.
I don't plan on going much into detail on this, so let's move on.
You should be brought to a screen that looks similar to this:
Click the Yes, log me in button.
You should be brought to the following screen, if you have a black bar at the top like in the image, you have logged in correctly:
Towards the bottom, you should see the Yakkety Daily suite3. Click on that and you should get to a screen with various options for images. Scroll down and you should get a screen similar to this:
Since I downloaded the amd64 desktop image earlier, I will select Lubuntu Desktop amd64.
You should get to a screen that looks like this:
Here we have four test cases to choose from:
Install (auto-resize)- requires an existing installation.Install (entire disk)- very generic.Install (manual partitioning)- same as entire disk except you get to manually partition everything.Live Session- testing the live instance.
Because it's the most generic, I'll select Install (entire disk).
When you click on that, you should get a screen similar to this:
If you scroll down to the bottom, you should see something similar to this:
Filling it out will be covered in a bit, but it's good to know what the submission form looks like.
Executing the test case
A lot of the test cases are done in virtual machines, but if you have hardware to spare, that's better, but it's not essential. I'm not going to cover how to use a virtual machines, but you can find good guides below. I prefer KVM because the Linux kernel has built-in support for it, it's a lot more integrated, and it's completely free software. I've used VirtualBox in the past, it's a great program, but it's proprietary, and the kernel modules can be a bit fidgety to get working after a kernel upgrade.
Before you begin, read over the test case you plan on executing, there may be some important instructions that you need to watch out for.
When you are ready to start, on the test case, select the In Progress radio box and press the Submit result button.
Complete the test exactly how the test case says it should be done.
During the installation, if you encounter an error that is critical enough that you cannot proceed, first look at the Bugs to look for.
Is there a bug that describes your problem?
If not, search Launchpad for the bug.
If it's not there, identify the culprit package and file a bug against it.
If you have trouble doing this, join the #ubuntu-quality channel using your IRC client or the IRC client provided below and ask for help.
When you are done, on the test case page, click the button to edit your test case result:
You should be brought to a page similar to this:
If you had no trouble installing it using the instructions, select the Passed radio box.
For the bugs, from what I have seen, any bugs in the system, NOT just the installation bugs, should be reported.
So look at the release notes of the most recent milestone for the flavor you are testing, and go through bug reports.
Confirm as much as you can, when you do, mark the bug as Me too and list the bugs under the Bugs header in the following format:
XXXXX, YYYYY, ZZZZZ
Be careful, the tracker likes to insert extra commas and sometimes mangle this a bit.
Always check this field before updating your result.
When you have all the bugs cited on this page, select the Update result button.
If you then scroll down to the bottom, you should see your Launchpad ID with a green checkmark beside it and links to the bugs you listed.
Congrats!
You now completed that test case!
Package QA Test
The package QA tests exist to ensure that all packages are thoroughly tested for the release. Ensure you have an installation of the daily image for that flavor. Before you begin, ensure you have an Ubuntu One account set up. Also ensure you have read the previous section as I will not repeat some information that is explained above.
Navigate to packages.qa.ubuntu.com in your browser, you should see the following:
Click on Yakkety Daily3 and you should be brought to a screen similar to this:
Since as of the time of writing, Xubuntu is the only flavor with package test cases, click on Xubuntu Desktop and you should be brought to a screen similar to this:
At this point, open the test case and complete like normal, making sure to follow instructions exactly how they are listed.
Instead of checking the release notes, go to the Launchpad bugs page for each package and check against all of the bugs.
You can find the bugs for each package at the following URL, replacing PACKAGE with the respective package name:
https://bugs.launchpad.net/ubuntu/+source/PACKAGE
After this is done, congrats!
You just completed a Package QA test!
Write a manual test case
The manual test cases are not automatically generated, they are hand-written for that application. I will tell you how to write a test case but I'm not actually going to write one because that would be too tedious for me to put in this article.
To keep track of the manual test cases that need writing, we have bugs for each test case and details on what to do to it.
I've done this a few times before, so I know these already, but read over the instructions for writing a test case before you begin.
First, assign the bug to yourself and mark it as In Progress.
Then read over the bug report and responses to ensure you know what you are doing.
Next, ensure the program is installed by default in the flavor the bug report specifies.
If this is not the case, mark the bug as Incomplete and state that it is not in the flavor you specified.
If it is there, you can continue.
After that, grab the source for the Ubuntu Manual Test Cases.
Ensure you have bzr installed and run:
$ bzr branch lp:ubuntu-manual-tests
Then cd into the ubuntu-manual-tests directory just created.
Go into the testcases/packages/ directory and another subdirectory if applicable.
Open your favorite text editor and create a file with the name of the package you are creating a test case for, no need for any numbers.
Using the guide above, write the test case and verify it.
Below I have embedded a video by Nicholas Skaggs if you prefer to view a video:
When you are done, make sure you are directly in the ubuntu-manual-tests directory, and execute the following command4:
$ bzr add * && bzr commit --fixes lp:BUG#
Then push to a Bazaar branch in Launchpad:
$ bzr push lp:~/ubuntu-manual-tests/bugfix#####
Then go to the Bazaar page for the Ubuntu Manual Testcases project and find your branch.
Then submit a merge request detailing your test case.
Congrats! You wrote a manual test case!
Further questions?
Below if you are on my website, I have linked an IRC client for you to use if you do not have your own.
Type in a nickname, press start, and press Enter to speak in the channel.
If you have an IRC client, go to #ubuntu-quality on irc.freenode.net.
If you prefer not to use IRC, we have a mailing list that you can use if you wish.
Good luck and I hope to see you around! ☺