- Testing and Performance
- Overview of testing process
- Testing and Performance tracking
- Testing baseline
- Testing criteria
- Performance baseline
- Push to staging
- Stabilization Phase
- Functional Requirements
- Hardware Architecture
- Current red flags
- Milestones and tasks
This document covers Oracle's effort in supporting the integration of IBM and SAP's PowerPC/AIX port into the OpenJDK master repository. Oracle engineers are needed to participate in this effort since there are a significant number of proposed changes in shared code, which will directly affect all JREs on the current list of supported platforms.
The goal of the PPC/AIX Project is to have a working AIX and Linux PPC 64bit port of OpenJDK. Project completion is marked by no unresolved regressions, either performance or functional on any platform that OpenJDK supports. A complete fix isn't required for each regressions discovered during the porting project, but a plan must be in place to track each regression via Oracle's bug tracking system.
Oracle will create a staging repository which is owned by the PPC/AIX Project and will contain the fixes that have been reviewed and approved. The PPC/AIX Project is responsible for ensuring that the forest is up-to-date with the latest changes from Master. Oracle will create a private Hudson instance for the staging repository, which will build and test the changes. Oracle will periodically test the changes in the staging repository and give feedback to the PPC/AIX Project as appropriate. Oracle, SAP and IBM will jointly determine when the staging forest is ready for integration. All discussions related to this Project should cross-post to the mailing list.
The PPC specific portions of the port will have been properly tested on systems where the results are satisfactory to SAP and IBM. The assumption is that IBM and SAP have workloads that properly test the PPC/AIX port, including the PPC code generation portions for which expertise at Oracle is limited.
Testing and Performance
SAP and IBM are responsible for running JCK, JTReg, and any other public appropriate test suites. A summary of results should be provided when a changeset is submitted for review.
Results from additional non-public test suites may be required before a changeset is pushed. In cases where Oracle tests are not public, it is Oracle's responsibility to run tests while a changeset is being reviewed. In the case of failed tests, or unacceptable performance Oracle will provide sufficient information to debug the problem possibly as a targeted test case, suggested fix, or general resolution advice.
Overview of testing process
An overview of the steps for integration:
- A patch for review will be sent to the mailing alias as well as being cross-post to the appropriate subcomponent mailing list by one of the developers of the PPC/AIX Project
- Oracle will create a BugID to track the patchset and any associated changes with the patch
- At least two Reviewers one of who should be an official JDK8 Reviewer will review the changes with feedback if appropriate. It's not required that both reviewers be Oracle employees.
- Feedback will be addressed before the patch is committed to the staging repository
- Regression bugs will be filed against these builds as appropriate
Testing and Performance tracking
If a test or performance regression is found by Oracle, the PPC/AIX Project will be notified via an E-mail sent to the mailing list with the following pieces of information:
- Test regression
- An Oracle-provided BugID
- Description of test
- Stack trace of failure
- If test is in open, then a reference to the test
- Performance regression
- An Oracle-provided BugID
- Name of benchmark if public
- Percentage regression
All communication will be done via the mailing list along with the appropriate OpenJDK mailing list for each subcomponent (compiler, runtime, gc, libraries, etc).
The following provides additional details for each of the test suites.
- JCK - Oracle and IBM/SAP
- The appropriate JCK must pass at the same rate as the unmodified source while in early access
- JTReg - Oracle and IBM/SAP
- The test suite must pass at the same rate as the unmodified source.
- Any existing platform-specific tests should be modified to run on the new platform or vacuously pass, as appropriate.
- If practical, unit or regression tests must be added to exercise any new functionality or changes in behavior. The tests must meet the following criteria:
- Pass if the behavior is as expected and fail otherwise.
- Be contained in the changeset which introduces the behavior.
- Be designed to run across multiple platforms if the source change applies to multi-platform code.
- Pass at initial push.
- Internal tests - Oracle, SAP, IBM
- All tests must pass at the same rate as before the change was made
Oracle recommends that SAP or IBM will test on at least one supported platform.
Testing criteria are measured and compared against the previous staging repository. The fixed baseline is based on the staging repository when the changes from mainline are merged with the staging repository. Once a baseline has been established, any failure as reported by Oracle that are new are tracked and sent to the ppcaix mailing list for discussion and fixing.
The following provides details for the performance aspects of this document, There must be no regression in performance on supported platforms with any change made by SAP or IBM.
- Minimal set of applications include but not limited to the following:
- Any potential negative performance impact must be understood, fixed and accepted before the port can be complete
Push to staging
An Oracle BugID is required for all changesets, which will be provided by Oracle. Once all testing has been satisfied and changes have been approved, the changeset can be pushed into the staging repository.
Once all changes have been integrated into the staging repository, a final round of testing will be performed. This testing will include a run of all tests specified in this document as well as other tests that Oracle will deem necessary. The testing will cover functional as well performance with the goal of the PPC/AIX project being that no regressions will be unresolved before merging into the main project.
A complete architectural review is required to ensure that the changes SAP and IBM propose fit into the overal architecture of HotSpot. Other areas of concern include coding style, overall code quality, ease of extensibility and other code metrics.
Updates and reviews to the architecture should be made by members of the PPC/AIX Porting Project at Architecture of the OpenJDK PPC Port .
Current red flags
|Issue||Current Status||Resolution||Date Resolved|
|Waiting on SQE resources||Waiting on SQE manager response|
Milestones and tasks
These dates are calendar weeks. There are significant requirements in terms of resources required as such this plan cannot start until those resources are available.
Confidence level maps to the following:
= Low effort or high confidence in meeting the expected effort schedule
= Medium effort or medium confidence in meeting the expected effort schedule
= High effort or low confidence in meeting the expected effort schedule
|Milestone||Task||Who||Effort (expected)||Effort (pessimistic)||Dependency||Target Date||Confidence Level (, , )||Completed||Comments|
|M1||Setup||~1 day (M1.1 + 1.2 + 1.3)||~1 week (M1.1 + 1.2 + 1.3)||When JEP is Funded|
Funded: 2013/06/10 notification
|M1.1||Staging repository setup||Oracle||Few hours||1 day||At request of Lead||When JEP is Funded||2013/06/10|
|M1.2||JIRA CPU and OS updates||Oracle||Few hours||1 day||After M1.1||July 2013||2013/07/12|
OS value "aix" added. New CPU field unnecessary.
The following fields should be updated. OS would have a new field 'aix' with optionally adding version post fix such as 'aix_7.1' or 'aix_6'. The CPU would have a new field 'ppc64'.
|M1.3||Hudson instance on staging server||Oracle||Few hours||1 day||After M1.1||July 2013||2013/08/01|
|M2||Review and approve initial set of changes||~4 weeks||~6 weeks||After M1||2013/11/28||The work can happen mostly in parallel, with M2.1 and M2.2 taking roughly the same time to complete as M2.3.|
|M2.1||Review changes for a working linux_ppc64 hotspot port||Oracle, SAP, IBM||~2 weeks||~3 weeks||After M1.3||Mid-Late August||2013/08/20||The initial 8 changes (1-8) that get a working linux_ppc64 port|
|M2.2||Review changes for a working aix_ppc64 hotspot port||Oracle, SAP, IBM||~2 weeks||~3 weeks||After M1.3||Mid-Late August||2013/09/06||The next set of patches (9-13) to get a working aix_ppc64 port|
|M2.3||Basic libraries porting effort||Oracle, SAP, IBM||~2 weeks||~4 weeks||After M1.3||Early-October||2013/11/28||Enabling the build and basic functionality of the complete JDK on Linux and AIX.|
|M3||Review and integrate PPC64/C2 and advanced HotSot changes and the remaining AIX class library changes.||~34 weeks (M3.1 + 3.2 + 3.3 + 3.4)||~43 weeks (M3.1 + 3.2 + 3.3 + 3.4)||After M2||None of the sub milestones are dependent on each other, they can happen in any order as the PPC/AIX project members see fit. While the milestones are not dependent on each other, there are resource constraints that will not allow them to happen in parallel.|
|M3.1||HotSpot changes for the PPC64 C2 Port||Oracle, SAP, IBM||~6 weeks||~7 weeks||After M2.2||December||2013/12/13||Basic changes for a running C2 Port on PPC64 (~10 changes).|
|M3.2||Remaining HotSpot changes for PPC64.||Oracle, SAP, IBM||~12 weeks||~15 weeks||After M2.2||February 2014||2014/01/22||Additional C2 and runtime/GC changes which fix remaining problems and improve the performance.|
|M3.3||Complete and stabilize class library port.||Oracle, SAP, IBM||~12 weeks||~15 weeks||After M2.2||February 2014||2014/01/24||Add class library functionality which is missing on AIX and improve the implementation.|
|M3.4||Performance evaluation||Oracle, SAP, IBM||~4 weeks||~6 weeks||After M3.3||March 2014||All performance criteria resolved|
|M4||Stabilization/post integration bug fixing||Oracle, SAP, IBM||~4 weeks||~8 weeks||After all milestones||March 2014||Extended testing on the completed integration will testing and performance regressions resolved|
|M5||Final integration||Oracle, SAP, IBM||< 1 week||1 week||After stabilization period||March 2014||2014/03/27||Verification of all integrated changes, including a functional satisfaction report by SAP and IBM|
These checkpoints are meant as a way to review the current status of the port, adjusting the schedule or adding resources as needed.
|After completion of M2.2 or M2.3|
|After completion of M3.2|
|After completion of M3.4|
|After final integration|