Running tests and pushing changes
Read about the Submit repo here.
In order to push a change to the HotSpot source code you need to complete all the steps in the list below. This is done in order to ensure code quality and reduce the risk of introducing bugs into the code base.
- You must be a Committer in the JDK project
- You need a non-JEP JBS issue for tracking
- Your change must have been available for review at least 24 hours to accommodate for all time zones
- Your change must have been approved by two Committers out of which at least one is also a Reviewer
- Your change must have passed through the hs tier 1 testing provided by the submit-hs repository with zero failures
- You must run all relevant testing to make sure your actual change is working
- You must be available the next few hours, and the next day and ready to follow up with any fix needed in case your change causes problems in later tiers
There is a notion of trivial changes that can be pushed sooner than 24 hours. It should be clearly stated in the review mail that the intention is to push as a trivial change. How to actually define "trivial" is decided on a case-by-case basis but in general it would be things like fixing a comment, or moving code without changing it. Backing out a change is also considered trivial as the change itself in that case is generated by mercurial.
Please note that the submit repository will only run a set of smoke-tests to ensure your change compiles and runs on a variety of platforms. It will not do any targeted testing on the particular code you have changed. Running through the submit repository is only the minimum requirement. You must also make sure your change works as expected before pushing using targeted testing. Consider writing a few JTREG tests for your change, or some unit tests using the GTest framework. Including the new tests (in the right places) with your push to the submit repository will ensure your tests will be run as part of your testing on all platforms and in the future. Look for tier1 in test/hotspot/jtreg/TEST.groups to see which tests and directories that are included in the submit repo testing.
Pushing a change is fairly straight forward. Make sure your commit has a proper description. The JBS bug id and the Rewiewed-by lines are mandatory.
hg pull -u --rebase
When you feel confident enough, push using
As noted in the list above, you are expected to be around after having pushed a change in case there are any issues with it. A change that causes failures in later tiers may be backed out if a fix can not be provided fast enough, or if the developer is not responsive when noticed about the failure. Note that #7 above should be interpreted as "it is a really bad idea to push a change the last thing you do before bedtime, or the day before going on vacation".
Test analysis and bug handling
This can mean a few different things:
- Any failure in tier 1 is considered critical. A single failure, no matter how isolated or trivial, will cause the entire tier to signal failure. As long as there are tier 1 failures, every developer that pushes a change must dig into the results to see what caused the failure. Human nature dictates that after a while one will assume that the failure is the same as it was in the previous run, and at that point tier 1 testing is useless. Unless the results are checked every single time we will not notice when a new failure appears.
- Any failure that is blocking integration is considered critical. Integration blockers cause delays in propagation of other changes. People outside of the HotSpot team might be waiting for a change to propagate to master. The performance team needs HotSpot changes to get into promoted builds on a regular basis. In short, integration blockers stop others from doing their work.
- Any failure that causes multiple unrelated tests to fail is considered critical. Bugs with a large damage area will hide other bugs and make it harder to fix those other bugs once they are found. Tests that cause massive failures in the nightly will seriously reduce our test coverage.
Handling a critical failure
It is required that anyone who pushes a change monitors the tier 1 testing and the following nightly, or designates this requirement to someone who can also fix possible regressions. It is considered exceptionally bad engineering to push a change at the end of the day, or on the last day before going on vacation or otherwise becoming unavailable. Critical failures must be handled with high urgency, thus the engineer who caused the failure is best suited to do so.
Backing a change out is easy. Anyone can do it.
There can never be any ownership involved when it comes to backing out a change. It doesn't matter who made the change or why it fails, if it needs to be backed out it should be backed out. Whoever is on site first should do it. Usually there is some synchronization before doing the actual work to avoid duplicate work.
Use mercurial to create a backout changeset:
A backout is considered a trivial change so a single Reviewer is enough and you can push immediately when reviewed. See How to backout a change for details on bug management in JBS related to backing out changes.
The nightly HotSpot testing starts at 8 pm PT each weeknight (Mon - Fri). To clarify what this means here are a few relevant timezones. The nightly snapshot in this example would be referred to as the Wednesday snapshot.
|AEST||Thursday May 12, 1:00 pm|
|IST||Thursday May 12, 9:30 am|
|CET||Thursday May 12, 5:00 am|
|UTC||Thursday May 12, 4:00 am|
|EST||Wednesday May 11, 11:00 pm|
|PST||Wednesday May 11, 8:00 pm|
- cpu (C+R) - will have to be decided on a case by case basis
- os (R)
- os_cpu (C)
- adlc (C)
- aot (C)
- asm (R)
- c1 (C)
- ci (C)
- classfile (R)
- code (C)
- compiler (C)
- gc (G)
- interpreter (R)
- jvmci (C)
- libadt (C)
- logging (R)
- memory (R+G)
- oops (R)
- opto (C)
- precompiled (R)
- prims (R)
- runtime (R)
- services (R)
- trace (R)
- utilities (R)
The HotSpot team owns two components in JBS: hotspot and core-svc.
|Team||JBS component||JBS query|
|Compiler||hotspot / compiler|
component = hotspot AND Subcomponent = compiler
|GC||hotspot / gc|
component = hotspot AND Subcomponent = gc
hotspot / runtime
hotspot / jfr
component = hotspot AND Subcomponent in (runtime, jfr)
hotspot / jvmti
hotspot / svc
core-svc / debugger
core-svc / tools
core-svc / java.lang.instrument
tools / hprof
( (component = hotspot AND Subcomponent in (jvmti, svc)) OR
(component = tools AND Subcomponent = hprof) )
|Monitor and Management|
hotspot / svc-agent
core-svc / java.lang.management
core-svc / javax.management
( (component = hotspot AND Subcomponent = svc-agent) OR
|Test||hotspot / test|
component = hotspot AND Subcomponent = test
integration_blocker label should be used to indicate that a bug is present in a repository but not present upstreams. When filing a new bug found in hotspot testing, the default should be to add the integration_blocker label. If it is verified (when filing or later) that the new failure is already present upstreams from the repository where it was found the label should be removed and an explicit comment must be added to the bug explaining why the bug is not an integration blocker. A failure seen in a project repo is considered escaped when found in
jdk/hs. A failure seen in
jdk/hs is considered escaped if seen in
jdk/jdk or if older failures are found from a date before the last integration to
testbug label should be used for bugs in tests and test infrastructure. There is no fundamental difference between a test bug and a product bug in the eyes of the gatekeeper. Both are considered integration blockers and both should be handled asap when appearing in the testing.
When a change is identified that causes a regression and the best way to handle it is to back out the change, anyone can do so. It will still go through the standard code review process, but is considered a trivial change and thus it requires only one Reviewer and will avoid the 24h code review window. The idea here is to save time by not having a broken change hindering others. There is also the rationale that the change itself is automatically created by hg, and reviewed by the person who is performing the backout, so only one additional reviewer is required.
There are two parts to this task, how to do the bookkeeping in JBS, and how to do the actual backout in mercurial.
How to work with JBS when a change is backed out
- Close the original JBS issue and mark it with "Fix Failed", which is an alternative on the "Verify" action.
- If the intention is to fix the change and submit it again, create a redo-issue to track that the work still needs to be done. Clone the original JBS issue and use prefix [REDO] on the summary.
- Make sure relevant information is brought to the clone.
- Create a backout-issue:
- Alternative 1 - a regression is identified directly. Create a Sub-Task to the redo-issue with the same summary, but prefix with [BACKOUT].
- Alternative 2 - an investigation issue is created, and during the investigation backing out the change is identified as the best solution.
- Use the investigation issue for the backout.
- Change summary to the same as the issue to back out and prefix with [BACKOUT].
- Link the redo-issue and the backout-issue.
- Alternative 3 - no redo issue was created. Create a backout-issue with the same summary, but prefix with [BACKOUT].
- Link the backout-issue and the original issue.
How to work with mercurial when a change is backed out
In order to backout a change, the
hg backout command is recommended, which essentially applies the anti delta of the change. Make sure you perform the backout in the most upstream repostiory the change has escaped to.
To remove noise from our testing, tests that are expected to fail are quarantined or excluded. A quarantined test is removed from standard runs, but run in a separate job. Excluded tests are not run.
Test quarantines are done in the file
ProblemList.txt while exclusions shall be done in the source code using
@ignore. This enables adhoc runs on local machines, it makes it possible to go back in time and get expected results, and (for good and bad) by being part of the standard code review process the visibility increases. For each quarantine and exclude, there should of course be a corresponding bug, which is referenced in the quarantine.
Quarantine jtreg tests
ProblemList.txt and exclude the test in the known issues ignore list.
Example where MyTest.java is excluded on windows, tracked by bug JDK-4711:
Exclude jtreg tests
@Ignore with a bug reference in the test case, to prevent the test from being run.
Example where MyTest.java is excluded, tracked by bug JDK-4711:
Dealing with JBS bugs for test exclusion
With the quarantine/exclude mechanism in the source code, a check-in into the repository is needed, which means a unique JBS issue and a code review is needed. This is a good thing since it makes the test problems visible.
- Code review: Since it is a trivial change, it only needs a review from one official Reviewer and don't need to wait 24 hour before commit.
- JBS issue: A JBS issue is created for the bug. Create a subtask for the test exclusion checkin.
Any rule, any process has exceptions. And in some cases it is better to live with a potential failure and integrate anyway. It should of course be used in rare cases only, for example when it is more important to continue to run the test, and thus at the same time accept the failure. A typical example is an intermittent failure like an intermittent crash (say once a month) in a large system test, or if more info is needed, possibly diagnostics are added to the test, and the risk of not running it is high.
Label the bug with hs-nightly-quarantine-exception.
There are a few ways to figure out if a problem has already escaped a repository:
- The easiest way is of course if you know which change that caused the problem. Just have a look at the hg log of the upstream repository to see if the change is in there.
- Searching JBS can give information as well. See if there is more than one bug filed for the issue. If it escaped it is likely that someone else has found it and maybe filed a new bug.
If a main bug is targeted to a release and the fix is pushed to a different release, then a backport bug is automatically created. Usually this is a "good thing", e.g., when you are really backporting a fix to an earlier release, but not always... If the main bug is targeted to a later release (due to schedule planning), but someone finds the time to fix that bug in the current release, then the bug should be retargeted to the current release before pushing the fix. However, sometimes we forget to do that.
Here is how to fix that:
- Reopen the backport bug that was created automatically
Use a comment like the following (in the reopen dialog):
- Change the 'Fix Version/s' from '10' to '11'.
- Close the backport bug as "Not an Issue".
- Clean up the main bug
Copy the open push notification comment from the backport bug to the main bug, e.g.:
Add a comment like the following to the main bug:
- Reset the main bug 'Fix Version/s' from '11' to '10'.
- Resolve the main bug as "Fixed" in build "team" or in build "master" depending on where the fix was pushed. Pushes to 'jdk/jdk' are fixed in build "master" and pushes to other repos are fixed in build "team".
There are several ways to find out if a change has been integrated to
jdk/jdk or not. The easiest way is probably to look at the mercurial log in
jdk/jdk and see if your change is there:
Or, if you don't have a recent clone of
jdk/jdk available, look at