To improve quality, there will be a greater emphasis on testing. As a team, we are responsible for the quality of FX and in addition to the usual unit and automated testing, we will be performing weekly sanity tests.
Every Monday morning for about an hour or two at most, every team members will sanity test the release. Team members will catch up on all changes from the different components that make up FX and ensure that they can build the product (or test using the latest Hudson build). Monday morning testing is a time bounded exercise that should not interfere with regular work but will give the team a sense of over all quality in the product
Testing will involve running for Ensemble8, Modena, toys and other applications on Windows, Mac and Linux and the other platforms. The goal is to catch regressions quickly at the time they go in. For example, a recent change to better support the Input Method on Linux caused selected text to be deleted every time a text control lost focus. This is the kind of thing that should be found immediately by anyone running the Linux platform.
Weekly Code Freeze / Push Rules / Milestone Week
Code will be frozen every Monday at 1:00 AM Pacific Time. The repo will remain effectively locked for most changes until 1:00 PM Pacific Time. During that time, only fixes for "stopper" bugs or test failures found during sanity testing are permitted. For some committers that are not located in the US time zones, this may result in "no push Mondays" where changesets will need to wait until the very late in the day or the next day (Tuesday) to be pushed.
Monday 1:00 AM Pacific -- repo is locked, testing can begin as soon as a build is available (or you can build it yourself)
Monday 1:00 PM Pacific -- repo is unlocked, changesets can be pushed normally again
During the 1 or 2 weeks before the rampdown for a release, and during the week before other published milestones, we need to be extra careful about destabilizing the release. During this time you will need additional approval prior to pushing the changeset.
The idea behind ramping down is to stabilize the code base but not stop all changes and disrupt work. It is important for the leads to understand which changes are going into a release in order to manage risk. We will be more restrictive as we approach the ship date.
Fixing Bugs Found as a Result of Sanity Testing
Serious bugs found during sanity testing are almost always regressions. When a "stopper" bug is found, the default action is that the change set that caused the problem will be reverted. Obviously, each particular case needs to be evaluated and the engineer who made the change will be consulted. If reverting the fix is likely to make things more unstable, then reverting is not the best answer. If it is a milestone week, then extra care will need to be taken when any changes are made (either reverting or fixing).
Monday 10:00 AM Pacific - the deadline for fixing a bug, if no plan is in place for a fix, the change set will be reverted
Bugs found as a result of sanity testing that are not stoppers are expected to be fixed before the next sanity test pass.
The following table describes sanity test assignments (* for has machine, ~ for has Vbox):
Freescale i.MX6 (EGL)
Freescale i.MX6 (sw 32-bit)
Freescale i.MX6 (sw 16-bit)
|David H||*||*E(O)||~||T(S)* ||*||*||*||*||*|
|E(C)||Ensemble8 Controls (Controls, Charts)|
|E(G)||Ensemble8 Graphics (Graphics 2D, Graphics 3D, Scene Graph)|
|E(O)||Ensemble8 Other (Animation, Language, Layout, Media,)|
|T(S)||Toys - run-subset.sh; for embedded, use run-core.sh|
|T(C)||Toys - run-controls.sh|
|Dev Build Full (use dev-build-full.sh, runs visual tests)|