1. Introduction
Streams
([streams]) are designed to provide real time streams of
data with powerful semantics (e.g. built-in backpressure and queuing) to allow
users to build higher-level abstractions. MediaStreamTracks
([getusermedia]) are opaque handles to Real-Time video/audio being
transported in the browser. This document describes the ways in which ReadableStreams
can be created out of a MediaStreamTrack
.
Please see the Readme/Explainer in the repository for use cases and more rationale.
2. MediaStreamTrack
API extension
partial interface MediaStreamTrack { // |any| should be ReadableStream, but that is not an idl type. [CallWith=ScriptState] readonly attribute any readable; };
, of type any, readonlyreadable
- Constructs a
ReadableStream
out of theMediaStreamTrack
following theMediaStreamTrack
lifetime. AReadableStreamReader
created out of this will produceVideoFrames
.
3. VideoFrame
typedef (Uint8Array or FrozenArray<Uint8Array>)VideoFrameDataArray
; interfaceVideoFrame
{ readonly attribute VideoFrameDataArray data; readonly attribute unsigned long width; readonly attribute unsigned long height; readonly attribute PixelFormat format; readonly attribute DOMHighResTimeStamp timecode; };
format
== rgba
, VideoFrame
is just an HTML Canvas 2D Context §imagedata with a timecode
. How to represent that in WebIDL?
, of type VideoFrameDataArray, readonlydata
-
Consider using Web IDL §ArrayBufferView ("used to represent objects that provide a view on to an Web IDL §ArrayBuffer.") -- it might expose too many format combinations though.
, of type unsigned long, readonlywidth
- Actual horizontal dimension of the data in the
data
object, in pixels.
, of type unsigned long, readonlyheight
- Actual vertical dimension of the data in the
data
object, in pixels.
, of type PixelFormat, readonlyformat
- This attribute specifies the concrete pixel format of
data
;rgba
is equivalent to the one of HTML Canvas 2D Context §imagedata.
, of type DOMHighResTimeStamp, readonlytimecode
- The difference between the timestamp of the first generated chunk of data
in
VideoFrame
and the timestamp of the first chunk in the firstVideoFrame
produced by this reader. Note that thetimecode
in the first producedVideoFrame
does not need to be zero.
3.1. PixelFormat
enum PixelFormat
{
"rgba",
"yuv420",
};
rgba
- Specifies one-dimensional data array in RGBA order, as integers in the range 0 to 255. This is the same format as the one in HTML Canvas 2D Context §imagedata.
yuv420
4. Examples
4.1. VideoFrame reading and casting onto a <canvas>
// Assuming |theCanvas| and |theStream| exist already. let context = theCanvas.getContext("2d"); let track = theStream.getVideoTracks()[0]; track.readable.pipeTo(new WritableStream({ write(videoFrame) { console.assert(videoFrame.format == "rgba"); if (videoFrame.format != "rgba") return; theCanvas.width = videoFrame.width; theCanvas.height = videoFrame.height; context.putImageData(videoFrame, 0, 0); } , close() { console.log("All data successfully read!"); } , abort(e) { console.error("Uh, oh, something went wrong: ", e); } }));