"MediaStream Image Capture"

1. Introduction

The API defined in this document captures images from a photographic device referenced through a valid MediaStreamTrack. The produced image can be in the form of a Blob (see takePhoto() method) or as a ImageBitmap (see grabFrame()). Picture-specific settings can be optionally provided as arguments that can be applied to the device for the capture.

2. Image Capture API

The User Agent must support Promises in order to implement the Image Capture API. Any Promise object is assumed to have a resolver object, with resolve() and reject() methods associated with it.

[Constructor(MediaStreamTrack track)]
interface ImageCapture {
   readonly attribute MediaStreamTrack videoStreamTrack;

   Promise<Blob>              takePhoto();
   Promise<ImageBitmap>       grabFrame();
};

This document defines a number of new constraints for a MediaStreamTrack that can be applied in order to make its behavior more suitable for taking pictures. The use of these constraints is via the standard getUserMedia() and applyConstraints() methods. They will modify the behavior of the track and its source independently of the usage of the videoStreamTrack with the ImageCapture object.

2.1. Attributes

videoStreamTrack, of type MediaStreamTrack, readonly

The MediaStreamTrack passed into the constructor.

2.2. Methods

ImageCapture(MediaStreamTrack track)

Parameter	Type	Nullable	Optional	Description
track	MediaStreamTrack	✘	✘	The MediaStreamTrack to be used as source of data. This will be the value of the videoStreamTrack attribute. The MediaStreamTrack passed to the constructor MUST have its kind attribute set to "video" otherwise a DOMException of type NotSupportedError will be thrown.

takePhoto()

takePhoto() produces the result of a single photographic exposure using the video capture device sourcing the videoStreamTrack and including any settings previously configured, returning an encoded image in the form of a Blob if successful. When this method is invoked:

1. If the readyState of videoStreamTrack provided in the constructor is not live, return a promise rejected with a new DOMException whose name is InvalidStateError.
2. Otherwise it MUST queue a task, using the DOM manipulation task source, that runs the following steps:
   1. Gather data from videoStreamTrack into a Blob containing a single still image. The method of doing this will depend on the underlying device.
      Devices MAY temporarily stop streaming data, reconfigure themselves with the appropriate photo settings, take the photo, and then resume streaming. In this case, the stopping and restarting of streaming SHOULD cause onmute and onunmute events to fire on the track in question.
   2. If the UA is unable to execute the takePhoto() method for any reason (for example, upon invocation of multiple takePhoto() method calls in rapid succession), then the UA MUST return a promise rejected with a new DOMException whose name is UnknownError.
   3. Return a resolved promise with the Blob object.

grabFrame()

grabFrame() takes a snapshot of the live video being held in videoStreamTrack, returning an ImageBitmap if successful. grabFrame() returns data only once upon being invoked. When this method is invoked:

1. If the readyState of videoStreamTrack provided in the constructor is not live, return a promise rejected with a new DOMException whose name is InvalidStateError.
2. Otherwise it MUST queue a task, using the DOM manipulation task source, that runs the following steps:
   1. Gather data from videoStreamTrack into an ImageBitmap object. The width and height of the ImageBitmap object are derived from the constraints of videoStreamTrack.
      The result of grabFrame() is affected by any options set by setOptions() if those are reflected in videoStreamTrack.
   2. Returns a resolved promise with a newly created ImageBitmap object.
   3. If the UA is unable to execute the grabFrame() method for any reason (for example, upon invocation of multiple grabFrame()/takePhoto() method calls in rapid succession), then the UA MUST return a promise rejected with a new DOMException whose name is UnknownError.

3. Extensions

3.1. MediaTrackSupportedConstraints dictionary

MediaTrackSupportedConstraints is extended here with the list of constraints that a User Agent recognizes for controlling the photo capabilities. This dictionary can be retrieved using MediaDevices getSupportedConstraints() method.

partial dictionary MediaTrackSupportedConstraints {
  boolean whiteBalanceMode = true;
  boolean colorTemperature = true;
  boolean exposureMode = true;
  boolean exposureCompensation = true;
  boolean iso = true;
  boolean redEyeReduction = true;
  boolean focusMode = true;
  boolean pointsOfInterest = true;

  boolean brightness = true;
  boolean contrast = true;
  boolean saturation = true;
  boolean sharpness = true;
  boolean imageHeight = true;
  boolean imageWidth = true;
  boolean zoom = true;
  boolean fillLightMode = true;
};

3.1.1. Members

whiteBalanceMode, of type boolean, defaulting to true

Whether white balance mode constraining is recognized.

colorTemperature, of type boolean, defaulting to true

Whether color temperature constraining is recognized.

exposureMode, of type boolean, defaulting to true

Whether exposure constraining is recognized.

exposureCompensation, of type boolean, defaulting to true

Whether exposure compensation constraining is recognized.

iso, of type boolean, defaulting to true

Whether ISO constraining is recognized.

redEyeReduction, of type boolean, defaulting to true

Whether red eye reduction configuration is recognized.

focusMode, of type boolean, defaulting to true

Whether focus mode constraining is recognized.

pointsOfInterest, of type boolean, defaulting to true

Whether points of interest are supported.

brightness, of type boolean, defaulting to true

Whether brightness constraining is recognized.

contrast, of type boolean, defaulting to true

Whether contrast constraining is recognized.

saturation, of type boolean, defaulting to true

Whether saturation constraining is recognized.

sharpness, of type boolean, defaulting to true

Whether sharpness constraining is recognized.

imageHeight, of type boolean, defaulting to true

Whether the image height configuration is recognized.

imageWidth, of type boolean, defaulting to true

Whether the image width configuration is recognized.

zoom, of type boolean, defaulting to true

Whether configuration of the zoom level is recognized.

fillLightMode, of type boolean, defaulting to true

Whether configuration of the fill light mode is recognized.

3.2. MediaTrackCapabilities dictionary

MediaTrackCapabilities is extended here with the capabilities specific to image capture. This dictionary is produced by the UA via getCapabilities() and represents the supported ranges and enumerations of the supported constraints.

partial dictionary MediaTrackCapabilities {
  sequence<MeteringMode>  whiteBalanceMode;
  MediaSettingsRange      colorTemperature;
  sequence<MeteringMode>  exposureMode;
  MediaSettingsRange      exposureCompensation;
  MediaSettingsRange      iso;
  sequence<boolean>       redEyeReduction;
  sequence<MeteringMode>  focusMode;

  MediaSettingsRange      brightness;
  MediaSettingsRange      contrast;
  MediaSettingsRange      saturation;
  MediaSettingsRange      sharpness;
  MediaSettingsRange      imageHeight;
  MediaSettingsRange      imageWidth;
  MediaSettingsRange      zoom;
  sequence<FillLightMode> fillLightMode;
};

3.2.1. Members

whiteBalanceMode, of type sequence<MeteringMode>

A sequence of supported white balance mode MeteringModes.

colorTemperature, of type MediaSettingsRange

This range reflects the supported correlated color temperature to be used for the scene white balance calculation.

exposureMode, of type sequence<MeteringMode>

A sequence of supported MeteringMode settings for exposure calculation.

exposureCompensation, of type MediaSettingsRange

This reflects the supported range of exposure compensation. The supported range can be, and usually is, centered around 0 EV.

iso, of type MediaSettingsRange

This reflects the permitted range of ISO values.

redEyeReduction, of type sequence<boolean>

A sequence of booleans indicating whether camera supports red eye reduction - on meaning true/active.

focusMode, of type sequence<MeteringMode>

A sequence of supported MeteringMode settings for focus mode.

brightness, of type MediaSettingsRange

This reflects the supported range of brightness setting of the camera. Values are numeric. Increasing values indicate increasing brightness.

contrast, of type MediaSettingsRange

This reflects the supported range of contrast. Values are numeric. Increasing values indicate increasing contrast.

saturation, of type MediaSettingsRange

This reflects the permitted range of saturation setting. Values are numeric. Increasing values indicate increasing saturation.

sharpness, of type MediaSettingsRange

This reflects the permitted sharpness range of the camera. Values are numeric. Increasing values indicate increasing sharpness, and the minimum value always implies no sharpness enhancement or processing.

imageHeight, of type MediaSettingsRange

This reflects the image height range supported by the UA.

imageWidth, of type MediaSettingsRange

This reflects the image width range supported by the UA.

zoom, of type MediaSettingsRange

This reflects the zoom value range supported by the UA.

fillLightMode, of type sequence<FillLightMode>

A sequence of supported fill light mode FillLightModes settings.

3.3. MediaTrackConstraintSet dictionary

MediaTrackConstraintSet dictionary is used for both reading the current status with getConstraints() and for applying a set of constraints with applyConstraints().

partial dictionary MediaTrackConstraintSet {
  MeteringMode       whiteBalanceMode;
  ConstrainDouble    colorTemperature;
  MeteringMode       exposureMode;
  ConstrainDouble    exposureCompensation;
  ConstrainDouble    iso;
  ConstraintBoolean  redEyeReduction;
  MeteringMode       focusMode;
  ConstraintBoolean  pointsOfInterest;

  ConstrainDouble    brightness;
  ConstrainDouble    contrast;
  ConstrainDouble    saturation;
  ConstrainDouble    sharpness;
  ConstrainDouble    imageHeight;
  ConstrainDouble    imageWidth;
  ConstrainDouble    zoom;
  FillLightMode      fillLightMode;
};

3.3.1. Members

whiteBalanceMode, of type MeteringMode

See white balance mode constrainable property.

colorTemperature, of type ConstrainDouble

See color temperature constrainable property.

exposureMode, of type MeteringMode

See exposure constrainable property.

exposureCompensation, of type ConstrainDouble

See exposure compensation constrainable property.

iso, of type ConstrainDouble

See iso constrainable property.

redEyeReduction, of type ConstraintBoolean

See red eye reduction constrainable property.

focusMode, of type MeteringMode

See focus mode constrainable property.

pointsOfInterest, of type ConstraintBoolean

See points of interest constrainable property.

brightness, of type ConstrainDouble

See brightness constrainable property.

contrast, of type ConstrainDouble

See contrast constrainable property.

saturation, of type ConstrainDouble

See saturation constrainable property.

sharpness, of type ConstrainDouble

See sharpness constrainable property.

imageHeight, of type ConstrainDouble

See image height constrainable property. The UA MUST select the closest height value of this constraint if it supports a discrete set of height options.

imageWidth, of type ConstrainDouble

See image width constrainable property. The UA MUST select the closest width value of this constraint if it supports a discrete set of width options.

zoom, of type ConstrainDouble

See zoom constrainable property.

fillLightMode, of type FillLightMode

See fill light mode constrainable property.

3.4. MediaTrackSettings dictionary

When the getSettings() method is invoked on a video stream track, the user agent must return the extended MediaTrackSettings dictionary, representing the current status of the underlying user agent.

partial dictionary MediaTrackSettings {
  MeteringMode      whiteBalanceMode;
  double            colorTemperature;
  MeteringMode      exposureMode;
  double            exposureCompensation;
  double            iso;
  boolean           redEyeReduction;
  MeteringMode      focusMode;
  sequence<Point2D> pointsOfInterest;

  double        brightness;
  double        contrast;
  double        saturation;
  double        sharpness;
  double        zoom;
  double        imageHeight;
  double        imageWidth;
  FillLightMode fillLightMode;
};

3.4.1. Members

whiteBalanceMode, of type MeteringMode

Current white balance mode setting.

colorTemperature, of type double

Color temperature in use for the white balance calculation of the scene. This field is only significant if whiteBalanceMode is manual.

exposureMode, of type MeteringMode

Current exposure mode setting. Acceptable values are of type MeteringMode.

exposureCompensation, of type double

Current exposure compensation setting. A value of 0 EV is interpreted as no exposure compensation.

iso, of type double

Current camera ISO setting.

redEyeReduction, of type boolean

Current camera red eye reduction configuration setting.

focusMode, of type MeteringMode

Current focus mode setting.

pointsOfInterest, of type sequence<Point2D>

A sequence of Point2Ds in use as points of interest for other settings, e.g. Focus, Exposure and Auto White Balance.

brightness, of type double

This reflects the current brightness setting of the camera.

contrast, of type double

This reflects the current contrast setting of the camera.

saturation, of type double

This reflects the current saturation setting of the camera.

sharpness, of type double

This reflects the current sharpness setting of the camera.

zoom, of type double

This reflects the current zoom setting of the camera.

imageWidth, of type double

This reflects the current image height.

imageHeight, of type double

This reflects the current image width.

fillLightMode, of type FillLightMode

This reflects the current fill light mode setting (flash).

3.5. Constrainable Properties

Many of these fields mirror hardware capabilities that are hard to define since can be implemented in a number of ways. Moreover, hardware manufacturers tend to publish vague definitions to protect their intellectual property. The following definitions are assumed for individual settings and are provided for information purposes:

1. White balance mode is a setting that cameras use to adjust for different color temperatures. Color temperature is the temperature of background light (usually measured in Kelvin). This setting can usually be automatically and continuously determined by the implementation, but it’s also common to offer a manual mode in which the estimated temperature of the scene illumination is hinted to the implementation. Typical temperature ranges for popular modes are provided below:

   Mode	Kelvin range
   incandescent	2500-3500
   fluorescent	4000-5000
   warm-fluorescent	5000-5500
   daylight	5500-6500
   cloudy-daylight	6500-8000
   twilight	8000-9000
   shade	9000-10000

2. Exposure is the amount of time during which light is allowed to fall on the photosensitive device. Auto-exposure mode is a camera setting where the exposure levels are automatically adjusted by the implementation based on the subject of the photo.
3. Exposure Compensation is a numeric camera setting that adjusts the exposure level from the current value used by the implementation. This value can be used to bias the exposure level enabled by auto-exposure, and usually is a symmetric range around 0 EV (the no-compensation value).
4. The ISO setting of a camera describes the sensitivity of the camera to light. It is a numeric value, where the lower the value the greater the sensitivity. This value should follow the [iso12232] standard.
5. Red Eye Reduction is a feature in cameras that is designed to limit or prevent the appearance of red pupils ("Red Eye") in photography subjects due prolonged exposure to a camera’s flash.
6. Focus mode describes the focus setting of the capture device (e.g. auto or manual).
7. Points of interest describe the metering area centers used in other settings, e.g. exposure, each one being a Point2D.

   A Point2D Point of Interest is interpreted to represent a pixel position in a normalized square space (|{x,y} ∈ [0.0, 1.0]|). The origin of coordinates |{x,y} = {0.0, 0.0}| represents the upper leftmost corner whereas the |{x,y} = {1.0, 1.0}| represents the lower rightmost corner: the x coordinate (columns) increases rightwards and the y coordinate (rows) increases downwards. Values beyond the mentioned limits will be clamped to the closest allowed value.

8. [LIGHTING-VOCABULARY] defines brightness as "the attribute of a visual sensation according to which an area appears to emit more or less light" and in the context of the present API, it refers to the numeric camera setting that adjusts the perceived amount of light emitting from the photo object. A higher brightness setting increases the intensity of darker areas in a scene while compressing the intensity of brighter parts of the scene. The range and effect of this setting is implementation dependent but in general it translates into a numerical value that is added to each pixel with saturation.
9. Contrast is the numeric camera setting that controls the difference in brightness between light and dark areas in a scene. A higher contrast setting reflects an expansion in the difference in brightness. The range and effect of this setting is implementation dependent but it can be understood as a transformation of the pixel values so that the luma range in the histogram becomes larger; the transformation is sometimes as simple as a gain factor.
10. [LIGHTING-VOCABULARY] defines saturation as "the colourfulness of an area judged in proportion to its brightness" and in the current context it refers to a numeric camera setting that controls the intensity of color in a scene (i.e. the amount of gray in the scene). Very low saturation levels will result in photos closer to black-and-white. Saturation is similar to contrast but referring to colors, so its implementation, albeit being platform dependent, can be understood as a gain factor applied to the chroma components of a given image.
11. Sharpness is a numeric camera setting that controls the intensity of edges in a scene. Higher sharpness settings result in higher contrast along the edges, while lower settings result in less contrast and blurrier edges (i.e. soft focus). The implementation is platform dependent, but it can be understood as the linear combination of an edge detection operation applied on the original image and the original image itself; the relative weights being cotrolled by this sharpness.
    Brightness, contrast, saturation and sharpness are specified in [UVC].
12. Image width and image height are...
    The supported resolutions are managed segregated e.g. imageWidth and imageHeight values/ranges to prevent increasing the fingerprinting surface and to allow the UA to make a best-effort decision with regards to actual hardware configuration vis-a-vis requested constraints.
13. Zoom is a numeric camera setting that controls the focal length of the lens. The setting usually represents a ratio, e.g. 4 is a zoom ratio of 4:1. The minimum value is usually 1, to represent a 1:1 ratio (i.e. no zoom).
14. Fill light mode describes the flash setting of the capture device (e.g. auto, off, on).

4. MediaSettingsRange

interface MediaSettingsRange {
    readonly attribute double max;
    readonly attribute double min;
    readonly attribute double step;
};

4.1. Attributes

max, of type double, readonly

The maximum value of this setting

min, of type double, readonly

The minimum value of this setting

step, of type double, readonly

The minimum difference between consecutive values of this setting.

5. FillLightMode

enum FillLightMode {
  "unavailable",
  "auto",
  "off",
  "flash",
  "torch"
};

5.1. Values

unavailable

This source does not have an option to change fill light modes (e.g., the camera does not have a flash)

auto

The video device’s fill light will be enabled when required (typically low light conditions). Otherwise it will be off. Note that auto does not guarantee that a flash will fire when takePhoto() is called. Use flash to guarantee firing of the flash for takePhoto() method.

off

The source’s fill light and/or flash will not be used.

flash

This value will always cause the flash to fire for takePhoto() method.

torch

The source’s fill light will be turned on (and remain on) while the source videoStreamTrack is active

6. MeteringMode

enum MeteringMode {
  "none",
  "manual",
  "single-shot",
  "continuous"
};

6.1. Values

none

This source does not offer focus/exposure/white balance mode. For setting, this is interpreted as a command to turn off the feature.

manual

The capture device is set to manually control the lens position/exposure time/white balance, or such a mode is requested to be configured.

single-shot

The capture device is configured for single-sweep autofocus/one-shot exposure/white balance calculation, or such a mode is requested.

continuous

The capture device is configured for continuous focusing for near-zero shutter-lag/continuous auto exposure/white balance calculation, or such continuous focus hunting/exposure/white balance calculation mode is requested.

7. Point2D

A Point2D represents a location in a two dimensional space. The origin of coordinates is situated in the upper leftmost corner of the space.

dictionary Point2D {
  double x = 0.0;
  double y = 0.0;
};

7.1. Members

x, of type double, defaulting to 0.0

Value of the horizontal (abscissa) coordinate.

y, of type double, defaulting to 0.0

Value of the vertical (ordinate) coordinate.

8. Examples

8.1. Update camera zoom and takePhoto()

<html>
<body>
<video autoplay></video>
<img>
<input type="range" hidden>
<script>
  var imageCapture;

  navigator.mediaDevices.getUserMedia({video: true})
    .then(gotMedia)
    .catch(err => console.error('getUserMedia() failed: ', err));

  function gotMedia(mediastream) {
    const video = document.querySelector('video');
    video.srcObject = mediastream;

    const track = mediastream.getVideoTracks()[0];
    imageCapture = new ImageCapture(track);
    imageCapture.getPhotoCapabilities()
      .then(photoCapabilities => {
        // Check whether zoom is supported or not.
        if (!photoCapabilities.zoom.min && !photoCapabilities.zoom.max) {
          return;
        }

        // Map zoom to a slider element.
        const input = document.querySelector('input[type="range"]');
        input.min = photoCapabilities.zoom.min;
        input.max = photoCapabilities.zoom.max;
        input.step = photoCapabilities.zoom.step;
        input.value = photoCapabilities.zoom.current;
        input.oninput = function(event) {
          imageCapture.setOptions({zoom: event.target.value});
        }
        input.hidden = false;
      })
      .catch(err => console.error('getPhotoCapabilities() failed: ', err));
  }

  function takePhoto() {
    imageCapture.takePhoto()
      .then(blob => {
        console.log('Photo taken: ' + blob.type + ', ' + blob.size + 'B');

        const image = document.querySelector('img');
        image.src = URL.createObjectURL(blob);
      })
      .catch(err => console.error('takePhoto() failed: ', err));
  }
</script>
</body>
</html>

8.2. Repeated grabbing of a frame with grabFrame()

<html>
<body>
<canvas></canvas>
<button onclick="stopGrabFrame()">Stop frame grab</button>
<script>
  const canvas = document.querySelector('canvas');

  var interval;
  var track;

  navigator.mediaDevices.getUserMedia({video: true})
    .then(gotMedia)
    .catch(err => console.error('getUserMedia() failed: ', err));

  function gotMedia(mediastream) {
    track = mediastream.getVideoTracks()[0];
    var imageCapture = new ImageCapture(track);
    interval = setInterval(function () {
      imageCapture.grabFrame()
        .then(processFrame)
        .catch(err => console.error('grabFrame() failed: ', err));
    }, 1000);
  }

  function processFrame(imgData) {
    canvas.width = imgData.width;
    canvas.height = imgData.height;
    canvas.getContext('2d').drawImage(imgData, 0, 0);
  }

  function stopGrabFrame(e) {
    clearInterval(interval);
    track.stop();
  }
</script>
</body>
</html>

8.3. Grabbing a Frame and Post-Processing

<html>
<body>
<canvas></canvas>
<script>
  const canvas = document.querySelector('canvas');

  var track;

  navigator.mediaDevices.getUserMedia({video: true})
    .then(gotMedia)
    .catch(err => console.error('getUserMedia() failed: ', err));

  function gotMedia(mediastream) {
    track = mediastream.getVideoTracks()[0];
    var imageCapture = new ImageCapture(track);
    imageCapture.grabFrame()
      .then(processFrame)
      .catch(err => console.error('grabFrame() failed: ', err));
  }

  function processFrame(imageBitmap) {
    track.stop();

    // |imageBitmap| pixels are not directly accessible: we need to paint
    // the grabbed frame onto a <canvas>, then getImageData() from it.
    const ctx = canvas.getContext('2d');
    canvas.width = imageBitmap.width;
    canvas.height = imageBitmap.height;
    ctx.drawImage(imageBitmap, 0, 0);

    // Read back the pixels from the <canvas>, and invert the colors.
    const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height);

    var data = imageData.data;
    for (var i = 0; i < data.length; i += 4) {
      data[i]     = 255 - data[i];     // red
      data[i + 1] = 255 - data[i + 1]; // green
      data[i + 2] = 255 - data[i + 2]; // blue
    }
    // Finally, draw the inverted image to the <canvas>
    ctx.putImageData(imageData, 0, 0);
  }
</script>
</body>
</html>

9. Legacy API

The previous thing was:

interface LegacyImageCapture {
  Promise<LegacyPhotoCapabilities> getPhotoCapabilities();
  Promise<void>                    setOptions(optional LegacyPhotoSettings photoSettings);
};

interface LegacyPhotoCapabilities {
  readonly attribute MeteringMode       whiteBalanceMode;
  readonly attribute MediaSettingsRange colorTemperature;
  readonly attribute MeteringMode       exposureMode;
  readonly attribute MediaSettingsRange exposureCompensation;
  readonly attribute MediaSettingsRange iso;
  readonly attribute boolean            redEyeReduction;
  readonly attribute MeteringMode       focusMode;

  readonly attribute MediaSettingsRange brightness;
  readonly attribute MediaSettingsRange contrast;
  readonly attribute MediaSettingsRange saturation;
  readonly attribute MediaSettingsRange sharpness;
  readonly attribute MediaSettingsRange imageHeight;
  readonly attribute MediaSettingsRange imageWidth;
  readonly attribute MediaSettingsRange zoom;
  readonly attribute FillLightMode      fillLightMode;
};

dictionary LegacyPhotoSettings {
  MeteringMode      whiteBalanceMode;
  double            colorTemperature;
  MeteringMode      exposureMode;
  double            exposureCompensation;
  double            iso;
  boolean           redEyeReduction;
  MeteringMode      focusMode;
  sequence<Point2D> pointsOfInterest;

  double        brightness;
  double        contrast;
  double        saturation;
  double        sharpness;
  double        zoom;
  double        imageHeight;
  double        imageWidth;
  FillLightMode fillLightMode;
};