Keyboard accessibility in web applications: part 2

By Mihai Sucan

Introduction

In the first part of the article we added keyboard shortcuts to a simple paint application. Now we will implement a way to move the pointer inside the drawing canvas using keyboard shortcuts.

At this point, the web application cannot be used entirely via the keyboard; it still requires a pointing device for the actual drawing. Users who really need keyboard-based drawing will most likely have mouse keys activated in their operating system of choice (check this page out for details of the Windows implementation, for example).

For educational and experimentation purposes, we can implement keyboard-based drawing inside our paint tool. Compared to a system-wide mouse keys implementation, this has the advantage of being tied directly to our web application, opening a range of possible improvements applicable only to our use-case.

To implement mouse keys in the paint application we have to add support for extending the application functionality from external code. We must make sure any extended functionality can be accessed from the keyboard shortcuts configuration file.

To make it easier to follow along with the code walkthrough presented below, download the full code example and follow along with it as you read the article. The code discussed in this article is all found inside the app4 directory.

Extending the paint application

Let's begin by adding support for removable actions to the paint application. Here is a snippet from the updated Painter class:

function Painter () {
  // ...
  /**
   * Holds all the removable functionality from the paint application.
   *
   * @type Object
   * @see Painter#actionAdd Add a new action.
   * @see Painter#actionRemove Remove an action.
   */
  this.actions = {};

  // ...
  this.actionAdd = function (id, func, overwrite) {
    if (typeof id != 'string' || typeof func != 'function' || (this.actions[id] && !overwrite)) {
      return false;
    }

    this.actions[id] = new func(_self);

    return this.actions[id] ? true : false;
  };

  // ...
  this.actionRemove = function (id) {
    if (!id || !_self.actions[id]) {
      return false;
    }

    if (typeof _self.actions[id].actionRemove == 'function') {
      _self.actions[id].actionRemove();
    }

    delete _self.actions[id];

    return true;
  };

  // ...
};

The new code allows us to add new actions, which are in fact functions that return an object holding any properties and methods you want. Any new object is added to the Painter actions with the given ID. When the action is removed, the actionRemove() method is called, if the action implements it. This allows you to execute any custom "uninstall" procedures you like.

There is not much we can do with that, for now. Let's tie things together by allowing the keyboard shortcuts configuration to call methods from installed actions. This can be done in the ev_keyhandler() function:

// ...
var gkey = PainterConfig.keys[ev.kid_];
if (!gkey) {
  return;
}

ev.kobj_ = gkey;

// Execute the associated action.
if (gkey.action) {
  var action = _self.actions[gkey.action];
  if (action) {
    var func = action[ev.type];
    if (typeof func == 'function') {
      func(ev);
    }
  }
}
// ...

This update allows us to include an action property for each configured keyboard shortcut, just like we have the tool property for tool activation. When the user presses the key combination, the method having the same name as the event type will be invoked, from the action object.

With the above code in place, we can now do something really simple like this:

PainterInstance.actionAdd('test', function (app) {
 // ...
 this.keydown = function (ev) { ... };
 this.keypress = function (ev) { ... };
 // ...
});

Into the PainterConfig object we can add a keyboard shortcut that invokes the test action:

var PainterConfig = {
  keys: {
    T: { action: 'test', prop1: 'val1' }
  }
};

From this code you can see that the key T is associated with the action with an ID of test. Now we've implemented the change on the ev_keyhandler() function, the keydown(), keypress() and the keyup() methods from the test action are invoked when their respective events are fired. Additionally, by adding the kobj_ property to the DOM Event object, any method from the test action can access the keyboard shortcut configuration and any of its properties. This allows us to pass custom parameters to the action method.

Mouse keys implementation

To implement mouse keys support, we need to always store the last mouse location in an object, to allow keyboard-based movements to continue from the same place.

During typical use, the event flow involves several mousemove events to the desired position, then mousedown to start drawing, then several mousemove events until the destination is reached, and finally a mouseup event for ending the drawing operation. We need to store the state of the mouse button - if it is down or not. The mouse button is considered to be down/active between a mousedown and a mouseup event. This is needed by a mouse keys implementation in order to determine and synthetically alter the state of the mouse button and generate any required mouse events when the user presses a key.

Here are the updates needed in the ev_canvas() function:

function Painter () {
  // ...

  /**
   * Holds the last recorded mouse coordinates and the button state (if it's 
   * down or not).
   *
   * @private
   * @type Object
   */
  this.mouse = {x: 0, y: 0, buttonDown: false};

  // ...
  function ev_canvas (ev) {
    // ...

    /*
     * If the mouse is down already, skip the event.
     * This is needed to allow the user to go out of the drawing canvas, release 
     * the mouse button, then come back and click to end the drawing operation.
     * Additionally, this is needed to allow extensions like mouse keys to 
     * perform their actions during a drawing operation, even when a real mouse 
     * is used. For example, allow the user to start drawing with the keyboard 
     * (press 0) then use the real mouse to move and click to end the drawing 
     * operation.
     */
    if (_self.mouse.buttonDown && ev.type == 'mousedown') {
      return false;
    }

    // Don't overwrite any existing x_ / y_ property.
    // These properties might be added by other functions.
    if (typeof ev.x_ == 'undefined') {
      if (typeof ev.layerX != 'undefined') { // Firefox
        ev.x_ = ev.layerX;
        ev.y_ = ev.layerY;
      } else if (typeof ev.offsetX != 'undefined') { // Opera
        ev.x_ = ev.offsetX;
        ev.y_ = ev.offsetY;
      }

      // Update the current mouse position only for mouse events.
      // Other events do not provide accurate mouse coordinates.
      switch (ev.type) {
        case 'mousedown':
        case 'mousemove':
        case 'mouseup':
          _self.mouse.x = ev.x_;
          _self.mouse.y = ev.y_;
      }
    }

    if (ev.type == 'mousedown') {
      _self.mouse.buttonDown = true;
    }

    // Call the event handler of the active tool.
    var func = _self.tool[ev.type];
    if (typeof func != 'function') {
      return false;
    }

    res = func(ev);

    if (ev.type == 'mouseup') {
      _self.mouse.buttonDown = false;
    }
  };

  // ...
};

Here we've added a new property object to the Painter class - the mouse property - to hold information about the mouse coordinates and the button state. The updated ev_canvas() function tries to determine the mouse coordinates if the DOM Event object does not already have the two x_ and y_ properties. Both properties will be added by the mouse keys implementation. The new ev_canvas() function updates the mouse coordinates and the button state, depending on each event type.

The above completes the updates necessary to the main code - check out the fourth index.js update.

In the current tools, we have set an inner variable named started to true on mousedown, to track if the mouse button is being clicked, so their mousemove event handler can perform the drawing operation. We can drop that now, and reuse the new mouse.buttonDown property from the application object. Check out the minor changes in the fourth tools.js update.

The Painter API provides us with sufficient pointer information, and an extension mechanism for adding custom keyboard shortcuts - we can now start implementing mouse keys support. Let's create a new mousekeys.js file, which is added into the index.html file (to see all of these in action, fire up index.html from the app4 folder in the code download). Here is the basic code structure for mouse keys:

/**
 * @class The MouseKeys action.
 *
 * @param {Painter} app Reference to the main paint application object.
 */
function PaintMouseKeys (app) {
  var canvas = app.buffer.canvas,
      mouse  = app.mouse;

  /**
   * Holds the current mouse movement speed in pixels.
   *
   * @private
   * @type Number
   */
  var speed = 1;

  /**
   * Holds the current mouse movement acceleration, taken from the 
   * configuration.
   *
   * @private
   * @type Number
   * @see PainterConfig.mousekeys_accel The mouse keys acceleration setting.
   */
  var accel = PainterConfig.mousekeys_accel;

  if (!canvas || !canvas.parentNode) {
    return false;
  }

  /**
   * Holds a reference to the DOM element representing the pointer on top of the 
   * canvas element.
   *
   * @private
   * @type Element
   */
  var pointer = document.createElement('div');
  if (!pointer) {
    return false;
  }
  pointer.id = 'mousekeysPointer';
  pointer.style.display = 'none';
  canvas.parentNode.appendChild(pointer);

  function pointerMousemove (ev) { ... };
  function dispatch (type, ev) { ... };

  this.keydown = function (ev) { ... };
  this.keypress = function (ev) { ... };
  this.actionRemove = function () { ... };

  canvas.addEventListener('mousemove', pointerMousemove, false);
};

The PaintMouseKeys class object holds the main methods: keydown(), keypress() and actionRemove(). The two private variables, speed and accel, hold information about the speed and the acceleration of the mouse pointer. The longer the user holds down a key, the faster the pointer will move. While a key is being held down the speed value increases based on the accel value.

The pointer DOM element is added in the parent node of the <canvas> element because it is needed for providing the user with a visual indicator of where the pointer is moving. Since this script cannot move the real mouse pointer, which might not even exist, we need our own virtual pointer. The pointerMousemove() updates the position of the pointer element when the mouse moves:

/**
 * Track the virtual pointer coordinates, by updating the position of the 
 * pointer element. This allows the keyboard users to see where 
 * they moved the virtual pointer.
 *
 * @param {Event} ev The DOM Event object.
 */
function pointerMousemove (ev) {
  if (typeof ev.x_ == 'undefined' || !ev.kobj_ || !ev.kobj_.action || ev.kobj_.action != 'mousekeys') {
    if (pointer.style.display == 'block') {
      pointer.style.display = 'none';
    }
    return;
  }

  pointer.style.top  = ev.y_ + 'px';
  pointer.style.left = ev.x_ + 'px';
};

The function above will only update the pointer location if the mousemove event contains the x_ and y_ coordinates, and only if the event was generated by the mousekeys action. If the event is not a synthetic one, the pointer element is hidden. We only want it visible when the keyboard is used.

You need to decide how you want the users to interact with the drawing canvas using their keyboard. I decided for this demo to use the number keys. Therefore, key 2 should move the cursor to the south (towards the bottom of the image), key 4 to the west (left), key 6 to the east (right), and key 8 should move the pointer towards the north (top of the image). The rest of the keys, 1, 3, 7 and 9, should move the pointer in diagonal directions (eg 1 for south-west). If the user also holds down the Shift key in addition to a direction, the mouse pointer moves faster.

Having defined what we want for the pointer movement (the mousemove event) we need a key to generate the mousedown and the mouseup events. For this demo, when the user presses the 0 key the synthetic mousedown event is dispatched. Pressing it again would generate the mouseup event. As such, this key allows the user to toggle the "mouse is down" state to true or false, by alternating the two events. When the mouse is down, any mousemove event will perform the drawing operation associated with the active tool. The drawing tools do not need any code change since the mouse events are all fired synthetically.

Configuration file update

Another important technical point is that we do not want to modify the script when we change the keyboard shortcuts. Here is the modification for the PainterConfig object:

var PainterConfig = {
  /**
   * The mouse keys movement acceleration.
   *
   * @type Number
   * @see PaintMouseKeys The MouseKeys extension.
   */
  mousekeys_accel: 0.1,

  /**
   * Keyboard shortcuts associated to drawing tools and other actions.
   *
   * @type Object
   * @see PaintTools The object holding all the drawing tools.
   */
  keys: {
    0: { action: 'mousekeys', param: 'Toggle' },
    1: { action: 'mousekeys', param: 'SouthWest' },
    2: { action: 'mousekeys', param: 'South' },
    3: { action: 'mousekeys', param: 'SouthEast' },
    4: { action: 'mousekeys', param: 'West' },
    6: { action: 'mousekeys', param: 'East' },
    7: { action: 'mousekeys', param: 'NorthWest' },
    8: { action: 'mousekeys', param: 'North' },
    9: { action: 'mousekeys', param: 'NorthEast' },
    L: { tool: 'line' },
    P: { tool: 'pencil' },
    R: { tool: 'rect' }
  }
};

We have added the virtual pointer acceleration we want, and the new keyboard shortcuts for the mousekeys action. The param property specifies the action what to do. This is where adding the kobj_ property to the DOM Event object in the ev_keyhandler() becomes useful. The keydown() and keypress() methods from the mouse keys implementation can use the param property to determine the mouse movement direction associated to the keyboard shortcut. As such, we do not include any of the keyboard shortcuts at all in the implementation, only in the configuration file.

When the Shift key is held down, the system generates different keys for the numbers on the number pad. We need to take them into consideration in the keyboard shortcuts configuration:

lib.extend(PainterConfig.keys, {
  'Shift Insert':   PainterConfig.keys['0'],
  'Shift End':      PainterConfig.keys['1'],
  'Shift Down':     PainterConfig.keys['2'],
  'Shift PageDown': PainterConfig.keys['3'],
  'Shift Left':     PainterConfig.keys['4'],
  'Shift Right':    PainterConfig.keys['6'],
  'Shift Home':     PainterConfig.keys['7'],
  'Shift Up':       PainterConfig.keys['8'],
  'Shift PageUp':   PainterConfig.keys['9']
});

We use the extend() function to add the new keys, as duplicates of the numbers. With this we have completed the update needed for the configuration file.

The event handlers

Now we will continue with the MouseKeys action implementation. Here is the code for the keydown() event handler:

this.keydown = function (ev) {
  speed = 1;
  accel = PainterConfig.mousekeys_accel;

  if (pointer.style.display == 'none') {
    pointer.style.display = 'block';
    pointer.style.top  = mouse.y + 'px';
    pointer.style.left = mouse.x + 'px';
  }

  if (!ev || !ev.kobj_ || ev.kobj_.param != 'Toggle') {
    return false;
  }

  var type = mouse.buttonDown ? 'mouseup' : 'mousedown';
  dispatch(type, ev);

  return true;
};

The keydown() method always resets the speed and the mouse acceleration to the default values. This ensures that the speed gained from the previous key press is not reused in another key press for mouse movement. Otherwise, the mouse movement would become too fast quite quickly.

The pointer element is always made visible, such that the user can see where the virtual pointer is located when he/she starts using the keyboard for drawing.

The kobj_ property object is used to determine which action needs to be performed. If the keyboard shortcut object holds the param property with the value Toggle, we perform the action of the key 0, as discussed above. We dispatch a mouseup event if the mouse button is down, otherwise we dispatch the mousedown event. The mouse.buttonDown boolean property is the one being updated by the ev_canvas() function as previously described.

The implementation for the rest of the number keys used for dispatching synthetic mousemove events is held in the keypress() event handler in the mousekeys action:

this.keypress = function (ev) {
  if (!ev || !ev.kobj_ || !ev.kobj_.param) {
    return false;
  }

  if (ev.shiftKey) {
    speed += speed * accel * 3;
  } else {
    speed += speed * accel;
  }

  var w = canvas.width,
      h = canvas.height,
      x = mouse.x,
      y = mouse.y,
      step = Math.ceil(speed);

  switch (ev.kobj_.param) {
    case 'SouthWest':
      x -= step;
      y += step;
      break;
    case 'South':
      y += step;
      break;
    case 'SouthEast':
      x += step;
      y += step;
      break;
    case 'West':
      x -= step;
      break;
    case 'East':
      x += step;
      break;
    case 'NorthWest':
      x -= step;
      y -= step;
      break;
    case 'North':
      y -= step;
      break;
    case 'NorthEast':
      x += step;
      y -= step;
      break;
    default:
      return false;
  }

  if (x < 0) {
    x = 0;
  } else if (x > w) {
    x = w;
  }

  if (y < 0) {
    y = 0;
  } else if (y > h) {
    y = h;
  }

  mouse.x = x;
  mouse.y = y;

  dispatch('mousemove', ev);

  return true;
};

The implementation for the movement keys has been put in the keypress() event handler because this event is always repeated while the user holds down a key. The keydown event is dispatched only once.

The code is rather simple: increase the movement speed based on the acceleration (with a variation if the Shift key is down), and update the mouse coordinates based on the param property of the keyboard shortcut object. At the end, the synthetic mousemove event is dispatched. The mouse coordinates are those automatically updated by the ev_canvas() function.

Here is the code that performs the actual synthetic event dispatching:

function dispatch (type, ev) {
  var ev_new = document.createEvent('MouseEvents');

  ev_new.initMouseEvent(type,
      ev.bubbles,  ev.cancelable,
      ev.view,     0,
      0,           0,
      0,           0,
      ev.ctrlKey,  ev.altKey,
      ev.shiftKey, ev.metaKey,
      0,           ev.relatedTarget);

  // Make sure the new coordinates are passed to the event handlers.
  ev_new.x_ = mouse.x;
  ev_new.y_ = mouse.y;

  // Make sure the event handlers can check this is a synthetic event.
  // This is needed by the pointerMousemove() function.
  ev_new.keyCode_  = ev.keyCode_;
  ev_new.key_      = ev.key_;
  ev_new.kid_      = ev.kid_;
  ev_new.kobj_     = ev.kobj_;

  canvas.dispatchEvent(ev_new);
};

This function creates a new mouse event of the given type. The new mouse event will share several properties (eg the active key modifiers) with the current keyboard event. Several new properties are added: the new mouse coordinates and the keyboard-related properties. The new event is dispatched to the buffer <canvas> element.

When the real mouse is used, typically the following execution chain is activated:

The mouse event is fired by the browser and the ev_canvas() event handler is invoked, which determines the mouse coordinates on the <canvas> element and updates the mouse object to hold them and the button state
The event handler of the active tool is invoked to perform any drawing operations

When the mouse keys are used, typically the following execution chain is activated:

The keyboard event is fired by the browser and handled by the KeyboardEventListener class instance from the minimal JavaScript library.
That code invokes the ev_keyhandler() function from the paint application, which in turn invokes the event handler within the mousekeys action.
The event handlers in the mouse keys implementation, as explained, also dispatch a synthetic mouse event, which in turn is handled by the ev_canvas() function and by the active tool.
For synthetic mousemove events, the pointerMousemove() function updates the location of the pointer element.

To wrap it all up, we include the actionRemove() method and perform the actual addition of the action in the Painter instance:

function PaintMouseKeys (app) {
  // ...

  /**
   * Handles action removal. This will remove the pointer DOM element and the 
   * canvas event listener.
   */
  this.actionRemove = function () {
    canvas.parentNode.removeChild(pointer);
    canvas.removeEventListener('mousemove', pointerMousemove, false);
  };
};

window.addEventListener('load', function () {
  // Add the MouseKeys action to the Painter instance.
  if (window.PainterInstance) {
    PainterInstance.actionAdd('mousekeys', PaintMouseKeys);
  }
}, false);

That's all! Go ahead and test the updated application inside the app4 folder - and the documentation, found in app4/doc/ - in the code download.

In the next part

In the third and final part of the article we will look into the cross-browser compatibility layer we use for dealing with the browser differences in the DOM keyboard events handling.

If you want to learn more, stick around at DevOpera and also follow the upcoming major changes to PaintWeb - the open-source web application this article is based on.

This article is licensed under a Creative Commons Attribution, Non Commercial - Share Alike 2.5 license.