TCK Coverage Document for JSR nnn: Wombat API (Optional Package for J2SE)
Assertion: A specific statement of functionality or behavior derived from a specification. A testable assertion is one that can be validated in an implementation by testing.
Test: A binary application or script comprised of one or more Test Cases.
Test Case: A single set of test inputs, execution conditions, and expected results developed to verify an implementation's conformance with a specific assertion.
Assertion Breadth Coverage: The ratio of all assertions tested by at least one test case to the total number of testable assertions defined by the specification.
Assertion Depth Coverage: For all assertions tested, the ratio of the number of these test cases to the total number of test cases needed to completely test the assertions (assertions may require multiple test cases to adequately verify conformance to the specification).
Method Coverage: The ratio of the number of methods directly exercised by test cases to the total number of methods defined by the specification.
Total testable assertions: 426, identified through specification markup.
Tests: 304 tests comprising 627 test cases.
Assertion Breadth Coverage: 70%.
Assertion Depth Coverage: Approximately 50% (estimated by sampling.) Assertion Breadth Coverage varies by package and ranges from 30% to 95%. Assertion Breadth Coverage of the critical Wombat data exchange protocol is estimated to be 90%.
Method Coverage (static): 94%, measured by Oracle's API Coverage Tool.
Method Coverage (runtime against the RI): 97%, measured by the XYZ code coverage utility
API Signature Coverage: 100% for all defined public and protected members - validated by the Oracle's Sig Test tool
The TCK was run using representative configurations of the Reference Implementation on the following platforms:
Code quality was demonstrated through use of code reviews and inspections. The User's Guide was modeled on Oracle-supplied TCK User's Guide template
Documentation instructions and Configuration Interviews were exhaustively verified and tested
Justification of Adequacy
As the first major release of this specification, the coverage is good and consistent with the coverage of similar JSRs at this level of spec maturity. Coverage variance is due to effects of late spec changes and the availability/cost of test development resources. Emphasis was given to covering critical spec areas as identified by the Spec Lead and Expert Group. We plan to increase coverage in weak areas during the next major specification update release.