This document defines the plan for building interoperability tests and evaluates the coverage of the test suite against the Encrypted Media Extensions specification [[!ENCRYPTED-MEDIA]].

This document is not yet maintained.

Useful links

Objective

For the EME specification to exit the Candidate Recommendation stage, two independent implementations as detailed in the CR Exit Criteria (Public Permissive version 3) document will be required to demonstrate each feature of the specification. The test suite is expected to demonstrate implementations beyond the Clear Key system required by the specification. There is no preliminary interoperability or implementation report yet.

Plan

Current status and next steps are summarized below.

Existing Tests

The currently checked in tests require the Clear Key CDM and exercise a range of EME requirements. These tests are grouped here by general category.

There is an existing Pull Request (w3c/web-platform-tests #3214 ) to add WebIDL checks as well.

Next Steps

  1. Integrate the WebIDL tests.
  2. Assess the coverage of the current ClearKey tests combined with the WebIDL ones.
  3. Create new tests to address the gaps in prioritized tiers:
    1. To test core functionality in commerical usage today.
    2. Tier 2: Address other prioritized needs.
    3. Tier 3: Any remaining testing gaps.

Remarks

All occurrences of InvalidAccessError in existing test files should be replaced by TypeError (changed in the spec in February 2016). The table below assumes that this has already been fixed.

ACTION: TBD - tests must still be collected and added.

Test coverage

Section Name Test(s) Covered Assignee

How to update the table

How to update the list of sections

To update the rows that compose the coverage table, run the following steps:

  1. Open the Encrypted Media Extensions specification in your favorite browser
  2. Run the following code in a console window to build the internal representation used to render the table above.
  3. Copy the result and paste it in the toc.js file in the same folder as the source of this document.
  4. Refresh this document.
// List of section titles to skip
// (because they don't contain testable assertions per se)
var sectionsToIgnore = [
  'Introduction',
  'Byte Stream Formats',
  'Examples',
  'Acknowledgments',
  'Revision History',
  'References',
  'Event Summary',
  'Track Buffers'
];


// Custom forEach function for querySelectorAll results
var forEach = function (array, callback, scope) {
  for (var i = 0; i < array.length; i++) {
    callback.call(scope, array[i], i);
  }
};


// Parse the table of contents and extract the sections of interest
var extractTocRecursively = function (tocEntry, section) {
  forEach(tocEntry.querySelectorAll('ul > li'), function (subTocEntry) {
    if (subTocEntry.parentNode.parentNode !== tocEntry) {
      return;
    }

    // Extract link and main title (skipping the section number in the "span")
    var link = subTocEntry.querySelector('a');
    var name = link.firstChild.nextSibling.textContent;
    if (sectionsToIgnore.includes(name)) {
      return;
    }

    var subSection = {
      number: link.firstChild.textContent.trim(),
      name: name,
      url: link.getAttribute('href').substring(1)
    };

    if (!section.children) {
      section.children = [];
    }
    section.children.push(subSection);
    extractTocRecursively(subTocEntry, subSection);
  });
};

var toc = { name: 'Table of Contents' };
extractTocRecursively(document.querySelector('#toc'), toc);


// Complete the extracted structure with attributes, methods, constructors
// definitions.
var addObjectPropertiesRecursively = function (section) {
  if (section.children) {
    section.children.forEach(addObjectPropertiesRecursively);
    return;
  }
  forEach(document.querySelectorAll(
    '#' + section.url + ' > dl:not(.switch) > dt'),
    function (dfn) {
      var type = dfn.parentNode.getAttribute('class') || 's';
      var subSection = {
        name: dfn.innerText,
        url: dfn.getAttribute('id'),
        type: type.substring(0, type.length - 1)
      }
      if (!section.children) {
        section.children = [];
      }
      section.children.push(subSection);
    }
  );
};
addObjectPropertiesRecursively(toc);


console.log(JSON.stringify(toc, null, 2));
        

How to update the list of tests

To update the list of tests and have them link to the section(s) of the specification that they check, edit the tests.js file that sits along the source of this document. Nothing magic there, that's all the result of a bit of elbow grease. A given test may check more than one section in the specification.

How to update the coverage info

To update the information displayed in the last column, edit the coverage.js file that sits along the source of this document. The keys of the coverage object are the ID of the sections, and the values are rough estimate of the test coverage in percent, or an object with a coverage key that sets the coverage percentage, a comments key that lists possibles comments about the coverage (in an array of strings), and an assignee key that sets the person responsible for updating the test suite to cover the section.

If a section does not appear in the coverage object, a question mark will appear in the last column to indicate that coverage is unknown at this stage.