Return to the top of the Spec Lead Guide

The Spec Lead's TCK obligations under the Process

A high-quality Technology Compatibility Kit (TCK) is critical to ensuring that implementations of Java technologies are compatible with their specifications and with each other. The JCP's Process Document specifies certain high-level requirements that TCKs must meet, and charges the Executive Committee with the responsibility for reviewing the quality of TCKs. Specifically, the Process Document states that the TCK submitted as part of a JSR's Final Draft must meet the following requirements:

• Include documentation explaining how to run the TCK and interpret the test results.
• Include Compatibility Requirements explaining what is necessary, in addition to passing the tests, to be compatible.
• Provide a mechanism for running the tests automatically.
• Define a Test Appeals Process.
• Provide 100% signature-test coverage to ensure that all of the APIs required by the Specification are completely and correctly implemented and that no unspecified APIs are included in the JSR's namespace.
• Provide a TCK Coverage Document explaining the criteria used to measure TCK test coverage, and justifying the level of coverage provided and the overall quality of the TCK.

What is compatibility?

The term compatibility when used within the JCP refers to a measure of an implementation's conformance to a specification. It does not refer to application portability, though the likelihood of such portability is increased when implementations are compatible. Compatibility provides assurance that:

• Implementations match the specification.
• Implementations all meet (at least) a minimum level of quality.
• Developers can write to a specification rather than an implementation.

What is conformance testing?

Conformance testing is different from unit-testing or QA. Conformance tests typically:

• Test what is specified and only what is specified (implementation-specific features are off-limits.)
• "Don’t care” about performance, efficiency, usability, security, robustness, or even quality unless these are specified (usually they aren’t.)
• Make no assumptions about the environment in which the implementation is run.
• Assume no knowledge of implementation internals (black-box testing only.)

The components of a typical TCK

A TCK should contain at least the following components:

• The test suite: conformance tests designed to demonstrate and verify implementation compliance to its specification.
• Test harness: a tool to automate test selection, scheduling, execution, and the reporting test results.
• Documentation: explaining how to configure and run the test suite. The TCK documentation should also specify the Compatibility Requirements and define the Test Appeals Process.
• Exclude list: a means to exclude invalid tests from being run.

Some TCKs, particularly those targeting the Java EE and Java ME platforms, may also require a "connector" module to enable the TCK to communicate with the implementation to be tested. In these cases the TCK documentation must provide detailed instructions on how to develop such a module, since typically it must be developed by the implementer.

What makes a good TCK?

A high-quality TCK is more than "lots of tests" (high coverage.) You must also consider:

• Are the tests focused where it's most important?
• Are the tests correct?
• Are they efficient?
• Are they portable?
• Are they documented?
• Are they maintainable?

What should you test?

It is impossible to “completely test” any non-trivial Specification. You must therefore consciously decide what not to test. You should focus your test development efforts:

• Where the risk of incompatibility is greatest.
• Where implementation is difficult.
• Where implementers may have an incentive to “tweak” the code (or even to cheat.)
• Where the consequences of incompatibility would be greatest. (For example, leading to security breaches or application breakage.)
• Where you will get the biggest bang for the buck. (For example, fundamental classes, mainline code, etc.)

Measuring coverage

The JCP's Process Document requires you to deliver a Test Coverage Document that explains how you measured TCK test coverage and that justifies the level of coverage you provide. In order to estimate your level of coverage you will need to identify normative requirements (Test Assertions) within the Specification and then for each area of the specification (class, functional area, chapter…) measure the breadth of your assertion coverage (the percentage of assertions that are tested) and estimate the depth of that coverage (how thoroughly the assertions are tested.) For a detailed explanation of the theory and usage of Test Assertions see the Test Assertion Guidelines document published by OASIS.

Use the Sample Test Coverage Document to get started.

Compatibility Requirements

The Compatibility Requirements specify the conditions that implementations must satisfy – in addition to passing the TCK tests – to be compatible. The Process Document requires that at a minimum these requirements include the following:

• implementations must fully implement the Specification, including all required interfaces and functionality, whether or not the TCK actually tests these features.
• implementations must not modify, subset, superset, or otherwise extend the implementation name space.

Typically many other requirements are also included, for example:

• Implementations must be able to pass all the tests in all configurations.
• The tests and test harness should not be modified.

You may find it helpful to start with the Simplified TCK User's Guide Template.

Signature testing

The Process Document requires that TCKs test to ensure that all of the APIs required by the Specification are completely and correctly implemented and that no un-Specified APIs are included in the JSR's namespace. (This process is called signature-testing, and may be most easily performed by means of the Signature Test tool that Oracle makes available for free download.

The Test Appeals Process

The Spec Lead is responsible for defining a test-appeals process enabling implementers to assert that a test is invalid. Possible grounds for such challenges are:

• Because of a bug in the test (for example, a logic error or an incorrect interpretation of the specification)
• Because of a bug in the specification (for example, its requirements are contradictory)
• Because the test is biased towards a particular implementation.

Planning TCK development

TCKs are difficut to produce, so it's important to begin early. You will need to define the level of coverage desired and to allocate adequate resources and time. Creating a good TCK will require nearly the same amount of effort and time as developing the Reference Implementation.

For more information about the TCK planning and development process see the TCK Project Planning and Development Guide.

For more information

Additional documentation and tools to aid in the process of TCK development can be found at http://jcp.org/en/resources/tdk.