"Media Capture from DOM Elements"

1. Introduction

This document describes an extension to both HTML media elements and the HTML canvas element that enables the capture of the output of the element in the form of streaming media.

The captured media is formed into a MediaStream ([mediacapture-streams]), which can then be consumed by the various APIs that process streams of media, such as [WEBAUDIO], or [WEBRTC].

2. HTML Media Element Media Capture Extensions

This section defines a method captureStream() on HTMLMediaElements.

Both MediaStream and HTMLMediaElement expose the concept of a track. Since there is no common type used for HTMLMediaElement, this document uses the term track to refer to either a VideoTrack or an AudioTrack. MediaStreamTrack is used to identify the media in a MediaStream.

partial interface HTMLMediaElement {
    MediaStream captureStream();
};

2.1. Methods

captureStream()

captureStream() method produces a real-time capture of the media that is rendered to the media element.

The captured MediaStream comprises of MediaStreamTracks that render the content from the set of VideoTrack.selected (for VideoTracks, or other exclusively selected track types) or AudioTrack.enabled (for AudioTracks, or other track types that support multiple selections) tracks from the media element. If the media element does not have a selected or enabled tracks of a given type, then no MediaStreamTrack of that type is present in the captured stream.

A <video> element can therefore capture a video MediaStreamTrack and any number of audio MediaStreamTracks. An <audio> element can capture any number of audio MediaStreamTracks. In both cases, the set of captured MediaStreamTracks could be empty.

Unless and until there is a track of given type that is selected or enabled, no MediaStreamTrack of that type is present in the captured stream. In particular, if the media element does not have a source assigned, then the captured MediaStream has no tracks. Consequently, a media element with a ready state of HAVE_NOTHING produces no captured MediaStreamTrack instances. Once metadata is available and the selected or enabled tracks are determined, new captured MediaStreamTrack instances are created and added to the MediaStream.

A captured MediaStreamTrack ends when playback ends (and the ended event fires) or when the track that it captures is no longer selected or enabled for playback. A track is no longer selected or enabled if the source is changed by setting the src or srcObject attributes of the media element. The steps in stop() are performed on the MediaStreamTrack when it ends.

The set of captured MediaStreamTracks change if the source of the media element changes. If the source for the media element ends, a different source is selected.

If the selected VideoTrack or enabled AudioTracks for the media element change, an addtrack event with a new MediaStreamTrack is generated for each track that was not previously selected or enabled; and a removetrack event is generated for each track that ceases to be selected or enabled. A MediaStreamTrack MUST be ended prior to being removed from the MediaStream.

Since a MediaStreamTrack can only end once, a track that is enabled, disabled and re-enabled will be captured as two separate tracks. Similarly, restarting playback after playback ends causes a new set of captured MediaStreamTrack instances to be created. Seeking during playback without changing track selection does not generate events or cause a captured MediaStreamTrack to end.

The MediaStreamTracks that comprise the captured MediaStream become muted or not muted as the tracks they capture change state. At any time, a media element might not have active content available for capture on a given track for a variety of reasons:

• Media playback could be paused.
• A track might not have content for the current playback time if that time is either before the content of that track starts or after the content ends.
• A MediaStreamTrack that is acting as a source could be muted or not enabled.
• The contents of the track might become inaccessible to the current origin due to cross-origin protections. For instance, content that is rendered from an HTTP URL can be subject to a redirect on a request for partial content, or the enabled or selected tracks can change to include cross-origin content.

Absence of content is reflected in captured tracks through the muted attribute. A captured MediaStreamTrack MUST have a muted attribute set to true if its corresponding source track does not have available and accessible content. An event named mute is raised on the MediaStreamTrack when content availability changes.

What output a muted capture produces as a result will vary based on the type of media: a VideoTrack ceases to capture new frames when muted, causing the captured stream to show the last captured frame; a muted AudioTrack produces silence.

Whether a media element is actively rendering content (e.g., to a screen or audio device) has no effect on the content of captured streams. Muting the audio on a media element does not cause the capture to produce silence, nor does hiding a media element cause captured video to stop.

Captured audio from an element with an effective playback rate other than 1.0 MUST be time-stretched. An unplayable playback rate causes the captured audio track to become muted.

3. HTML Canvas Element Media Capture Extensions

The captureStream() method is added to the HTMLCanvasElement. The resulting CanvasCaptureMediaStreamTrack provides methods that allow for controlling when frames are sampled from the canvas.

partial interface HTMLCanvasElement {
    MediaStream captureStream(optional double frameRequestRate);
};

interface CanvasCaptureMediaStreamTrack : MediaStreamTrack {
    readonly attribute HTMLCanvasElement canvas;
    void requestFrame();
};