Streams and MediaStreamTracks

1. Introduction

Streams ([streams]) are designed to provide real time streams of data with powerful semantics (e.g. built-in backpressure and queuing) to allow users to build higher-level abstractions. MediaStreamTracks ([getusermedia]) are opaque handles to Real-Time video/audio being transported in the browser. This document describes the ways in which ReadableStreams can be created out of a MediaStreamTrack.

Please see the Readme/Explainer in the repository for use cases and more rationale.

2. MediaStreamTrack API extension

partial interface MediaStreamTrack {
  // |any| should be ReadableStream, but that is not an idl type.
  [CallWith=ScriptState] readonly attribute any readable;
};

readable, of type any, readonly

Constructs a ReadableStream out of the MediaStreamTrack following the MediaStreamTrack lifetime. A ReadableStreamReader created out of this will produce VideoFrames.

3. VideoFrame

typedef (Uint8Array or FrozenArray<Uint8Array>) VideoFrameDataArray;

interface VideoFrame {
  readonly attribute VideoFrameDataArray data;
  readonly attribute unsigned long width;
  readonly attribute unsigned long height;
  readonly attribute PixelFormat format;

  readonly attribute DOMHighResTimeStamp timecode;
};

data, of type VideoFrameDataArray, readonly

Consider using Web IDL §ArrayBufferView ("used to represent objects that provide a view on to an Web IDL §ArrayBuffer.") -- it might expose too many format combinations though.

width, of type unsigned long, readonly

Actual horizontal dimension of the data in the data object, in pixels.

height, of type unsigned long, readonly

Actual vertical dimension of the data in the data object, in pixels.

format, of type PixelFormat, readonly

This attribute specifies the concrete pixel format of data; rgba is equivalent to the one of HTML Canvas 2D Context §imagedata.

timecode, of type DOMHighResTimeStamp, readonly

The difference between the timestamp of the first generated chunk of data in VideoFrame and the timestamp of the first chunk in the first VideoFrame produced by this reader. Note that the timecode in the first produced VideoFrame does not need to be zero.

3.1. PixelFormat

enum PixelFormat {
  "rgba",
  "yuv420",
};

rgba

Specifies one-dimensional data array in RGBA order, as integers in the range 0 to 255. This is the same format as the one in HTML Canvas 2D Context §imagedata.

yuv420

4. Examples

4.1. VideoFrame reading and casting onto a <canvas>

// Assuming |theCanvas| and |theStream| exist already.

let context = theCanvas.getContext("2d");

let track = theStream.getVideoTracks()[0];

track.readable.pipeTo(new WritableStream({
  write(videoFrame) {
    console.assert(videoFrame.format == "rgba");
    if (videoFrame.format != "rgba")
      return;

    theCanvas.width  = videoFrame.width;
    theCanvas.height = videoFrame.height;
    context.putImageData(videoFrame, 0, 0);
  }
  , close() {
    console.log("All data successfully read!");
  }
  , abort(e) {
    console.error("Uh, oh, something went wrong: ", e);
  }
}));