As KARL3 is developed, we provide a test site where OSI evaluates deliverables and reports bugs. This process is called “UAT”: User Acceptance Testing.
This document explains the process and provides details (URLs, usernames and passwords, bug reporting procedures) for conducting UAT.
The KARL3 project is scheduled into units called “milestones”. A milestone is usually around a week long and contains a number of “screens” (Login screen, List Communities screen, etc.) that are scheduled for completion and evaluation.
Each of these screens have a scorecard that measures success. The scorecard is called the “UAT” (user acceptance test) and lists all the business rules for that screen. The set of UATs for a milestone is the definition of the milestone itself and the success criteria.
Note
The buck stops with Nat on approving updates to the scorecard and flagging the scorecard as “Confirmed”. He will consolidate OSI’s internal discussions and be the liason for testing.
To manage and report on this UAT process, we create “bug reports” in Launchpad for each UAT in a milestone. They are denoted as a UAT by using the “uat” tag on the bug report. Further, we use a tag as the screen identifier to know which screen the UAT is associated with. This system of cross-referencing via tags is how we layer project management atop Launchpad’s (rather limited) PM facilities.
Details about each of these UAT bug reports in Launchpad:
The UAT bug is assigned a milestone that schedules that screen’s evaluation to a milestone.
The UAT bug has two tags: one for “uat” that makes it a UAT bug report, and onther for the screen id (e.g. “show-login”).
The screen id helps see what existing bugs are open for that screen.
The body of the issue points to a page in the docs with all the business rules.
The UAT bug is assigned to OSI and owned by them. The status of the UAT bug reflects percent completion of evaluation:
o Mark its status as “Confirmed” to say that Nat and Paul agree on the scorecard.
o “In Progress” says evaluation has started.
o “Fix Committed” when all bugs have been reported.
o “Fix Released” when OSI confirms all bugs are closed. (Note: We might never get around to this step, it’s a lot of extra work for OSI to confirm bug closing.)
To discuss the scorecard’s business rules, use comments on the UAT issue. This keeps an historical record of the discussion and the decisions regarding the scorecard. Again, this is the authoritative specification, trumping all emails, phone calls, and hearsay.
Note
Use bookmarks and multiple tabs to avoid re-opening the same screens repeatedly. For example, do an advanced search of all the uat’s in M3 and bookmark that (with a friendlier bookmark title.) Bookmark the advanced bug report URL, etc.
As you progress through the items in the UAT “scorecard” and find a problem, report a bug. Simple tip: bookmark the URL for adding an advanced bug.
Filing a bug is relatively simple:
On the next screen you have the option to do additional categorizing by clicking on the icon to the left of New under Status. The icon looks like an “eject” button.