Return to the top of the Spec Lead Guide
The Spec Lead's TCK obligations under the Process
A high-quality Technology Compatibility Kit (TCK) is critical to ensuring
that implementations of Java technologies are compatible with their
specifications and with
each other. The JCP's Process
Document specifies certain high-level requirements
that TCKs must meet, and charges the Executive
Committee with the responsibility for reviewing the quality of TCKs.
Specifically, the Process Document states that the TCK submitted as part
of a JSR's Final
Draft must meet the following requirements:
- Include documentation explaining how to run the TCK and interpret the test
- Include Compatibility Requirements explaining what is necessary,
in addition to passing the tests, to be compatible.
- Provide a mechanism for
running the tests automatically.
- Define a Test Appeals Process.
- Provide 100% signature-test coverage to
ensure that all of the APIs required by the Specification are completely
correctly implemented and that
no unspecified APIs are included
in the JSR's
- Provide a TCK Coverage Document explaining the criteria used
to measure TCK test coverage, and justifying the level of coverage
provided and the overall quality of the TCK.
What is compatibility?
The term compatibility when used within the JCP refers to a measure
of an implementation's conformance to a specification. It does not
refer to application portability, though the likelihood of such portability
is increased when implementations are compatible.
Compatibility provides assurance that:
- Implementations match the specification.
- Implementations all meet (at least) a minimum level of quality.
can write to a specification rather than an implementation.
What is conformance testing?
Conformance testing is different from unit-testing or QA. Conformance tests
- Test what is specified and only what is specified (implementation-specific
features are off-limits.)
- "Don’t care” about
performance, efficiency, usability, security, robustness, or even
quality unless these are specified (usually they aren’t.)
- Make no assumptions about the environment in which the implementation
- Assume no knowledge of implementation internals (black-box testing
The components of a typical TCK
A TCK should contain at least the following components:
- The test suite: conformance tests designed to demonstrate and verify
implementation compliance to its specification.
- Test harness: a tool to automate test selection, scheduling, execution,
and the reporting test results.
- Documentation: explaining how to configure and run the test suite.
The TCK documentation should also specify the Compatibility
Requirements and define the Test Appeals Process.
- Exclude list: a means to exclude invalid tests from being run.
Some TCKs, particularly those targeting the Java EE and Java ME platforms, may also require a "connector" module to enable the TCK to communicate with the implementation to be tested. In these cases the TCK documentation must provide detailed instructions on how to develop such a module, since typically it must be developed by the implementer.
What makes a good TCK?
A high-quality TCK is more than "lots of tests" (high coverage.) You
must also consider:
- Are the tests focused where it's most important?
- Are the tests correct?
- Are they efficient?
- Are they portable?
- Are they documented?
- Are they maintainable?
What should you test?
It is impossible to “completely test” any non-trivial Specification.
You must therefore consciously decide what not to test. You should focus
your test development efforts:
- Where the risk of incompatibility is greatest.
- Where implementation
- Where implementers may have an incentive to “tweak” the
code (or even to cheat.)
- Where the consequences of incompatibility would be greatest. (For
example, leading to security breaches or application breakage.)
you will get the biggest bang for the buck. (For example, fundamental
classes, mainline code, etc.)
The JCP's Process Document requires you to deliver a Test Coverage
Document that explains how you measured TCK test coverage
and that justifies the level of coverage you provide. In order to estimate
your level of coverage you will need to identify normative requirements
(Test Assertions) within the Specification and then for each area of
the specification (class, functional area, chapter…)
measure the breadth of your assertion coverage (the percentage of assertions
that are tested) and estimate the depth of that coverage (how thoroughly
the assertions are tested.) For a detailed explanation of the theory and usage of Test Assertions see the Test Assertion Guidelines document published by OASIS.
Use the Sample Test Coverage Document to get started.
The Compatibility Requirements specify the conditions that implementations
must satisfy – in
addition to passing the TCK tests – to be compatible. The Process
Document requires that at a minimum these requirements include the
- implementations must fully implement the Specification,
including all required
functionality, whether or not the TCK actually tests these
- implementations must not modify, subset, superset, or otherwise
extend the implementation name space.
Typically many other requirements are also included, for example:
- Implementations must be able to pass all the tests in all configurations.
- The tests and test harness should not be modified.
You may find it helpful to start with the Simplified
TCK User's Guide Template.
The Process Document requires that TCKs test to ensure that all of the
APIs required by the Specification are completely and correctly implemented
no un-Specified APIs
are included in the JSR's namespace. (This process is called signature-testing,
and may be most easily performed by means of the Signature
Test tool that Oracle makes available for free download.
The Test Appeals Process
The Spec Lead is responsible for defining a test-appeals process enabling
implementers to assert that a test is invalid. Possible grounds for
such challenges are:
The appeals process should identify who can make test challenges, how
such challenges should be submitted, how challenges are addressed,
and how successful challenges are managed.
Because of a bug in the test (for example, a logic
error or an incorrect interpretation of the specification)
- Because of a bug in the specification (for example,
its requirements are contradictory)
- Because the test is biased towards a particular implementation.
Planning TCK development
TCKs are difficut to produce, so it's important to begin early.
You will need to define the level of coverage desired and to
allocate adequate resources
and time. Creating a good TCK will require nearly the same amount of
time as developing the Reference Implementation.
For more information about the TCK planning and development process
see the TCK Project
Planning and Development Guide.
For more information
Additional documentation and tools to aid in the process of TCK development
can be found at http://jcp.org/en/resources/tdk.
The PMO welcomes suggestions and requests from the Spec Leads for improvements
of this guide and the process: firstname.lastname@example.org