Symbol users with Speech, Language and Literacy Difficulties

This draft is out of date, an update that addresses new research and new technologies is being worked on at Symbol users with Speech, Language and Literacy Difficulties.

Introduction

Voice Output Communication Aids (VOCAs) are devices that have been developed to allow pictures, symbols or words to be used with speech synthesis. The devices may be specialist items with built in systems carrying the symbols. They may also be a generic computer, tablet or mobile with specialist software or apps. The symbols are activated by single or multiple touches, eyes dwelling on a chosen image or manual use of a keyboard or switches. An external input device usually allows for step by step scanning up and down rows and columns of symbols on a grid. Symbols and word choices are personalised to suit the user, representing known objects, actions and happenings.

Challenges for people with cognitive disabilities using symbols

The users of Alternative and Augmentative Communication (AAC) based on images or pictographs as symbols tend to have no speech or language, very unintelligible speech or difficulties expressing themselves and/or may need reading support. Individuals may also have severe mobility and dexterity disabilities and/or cognitive impairments. Depending on the skills of the user, the environment and task in hand, symbols may be used to represent a phrase or whole sentence of speech or be made up of individual parts of speech to aid sentence making. Where literacy skills are a challenge the use of online communication becomes an issue due to the lack of accurate symbol to text or text to symbol sets that can be used/translate across all symbol systems.

Memory

Symbol users may be unable to cope with large amounts of online material. This depends on their ability but may limit the degree to which they can cope with content that is text based.

Reasoning and executive function

Symbol users may find it hard to plan a route through web pages unless navigation is clear.

They may have other physical and cognitive impairments affecting their understanding and use of navigation controls.

AAC users may also be overwhelmed by the amount of interactions required to complete tasks.

Attention-related limitations

Symbol users may find large amounts of text reduces attention but symbol based content helps concentration.

Language and auditory perception related functions

Symbol users may find dense text based content with audio output confusing whereas audio with symbols may be helpful.

Literacy-related functions

Symbol users may not comprehend the meaning of text based content and may fail to act correctly when warnings or other interactive items appear on the web.

Knowledge

Symbol users may fail to recognize images, such as symbols or icons that are not in their known set and may lack the ability to use the web as it has been intended, failing to find understandable icons or imagery that provides easy to use navigation or content.

Proposed solutions

The strategy of mapping symbol sets to a common vocabulary may allow developers to incorporate helpful symbols knowing these can be automatically presented to the user in their own symbol language. This schema would also allow symbol users to use their own symbols in an edit box or form knowing it can be recognised by other symbol users with the automatic translation. The primary solutions for the mapping symbols for symbol users could be divided into three different user scenarios. For all these scenarios, the symbols presented to the symbol users could be alternative and adaptive based on the users' preference.

  • Adaptive Symbols in the Web Content;
  • Adaptive Symbols in the User Agent;
  • Adaptive Symbols in the Web Authoring Tools.

Using proposed syntax

One option for interoperable symbol mapping uses the proposed syntax: aria-concept = "uri" to reference target concept URI like the following example.

<img aria-concept="http://wordnet.org/somepage#girlnode" src="girlwithbow.gif"/>

This aria-concept syntax could take advantage of the Concept Coding Framework (CCF). Due to the lack of standardised encoding schemes and common practises, it is difficult to reference to and exchange symbols as an alternative or complement character based texts. Therefore, the CCF provides a common framework to link and map these symbols (currently Blissymbolics and ARASAAC symbols, and eventually other alternative representations) based on the concept coding. For example, the presentation of cat symbol (concept coding: cc-cat-1001) in this scenario would be as follows: for the cat concept: cc-cat-1001, there are different symbols to represent this concept across different symbols providers.

  • If user prefer ARASSC symbols:
    <img aria-concept=" http://www.conceptcoding.org/ontologies/2011/09/01/ConcetpDefinitions#cc-cat-1001" src=" http://www.catedu.es/arasaac/repositorio/thumbs/10/200/9/9881.png"/>
  • If user prefer Bliss symbols:
    <img aria-concept=" http://www.conceptcoding.org/ontologies/2011/09/01/ConcetpDefinitions#cc-cat-1001" src=" http://www.symbolforwindows.eu/images/Bls_cat.gif"/>
  • or other symbols

Mapping symbols

In order to map the symbols from different dictionaries, one of the approaches could be the use of an ontology and Semantic Web technologies to enable interoperability of symbols datasets.

An ontology is a formal specification of a shared conceptualization. In Semantic Web, a vocabulary can be considered as a special form of (usually light-weight) ontology, or sometimes also merely as a collection of URIs with a (usually informally) described meaning. Linked Data is a term used to describe a recommended best practice for exposing, sharing, and connecting pieces of data, information, and knowledge on the Semantic Web using URIs and RDF (The Resource Description Framework, W3C recommend metadata data model). The Concept Encoding Framework Working Group provides the multilingual and multi-modal lexical ontology (The Lexicon Model for Ontologies of the CCF to be made available in a Linked Open Data (LOD) format. It also allows users to search symbols based on different concepts and metadata, such as language, symbol datasets, localization. The following example demonstrates using the RDFa to represent the mapping between symbols based on concept coding.

  • The ARASAAC monitor symbol:
    <div xmlns="http://www.w3.org/1999/xhtml" prefix=" rdf:http://www.w3.org/1999/02/22-rdf-syntax-ns# ccf: http://www.conceptcoding.org/Ontologies/2014/07/ccf-owl-ns# rdfs: http://www.w3.org/2000/01/rdf-schema#">
    <div typeof="ccf:RepresentationItem" about="http://www.conceptcoding.org/ontologies/2011/09/01/ArasAssistiveOntologgy#ARAS274521001">
    <div property="ccf:example" content=""></div>
    <div rel="ccf:concept" resource="http://www.conceptcoding.org/ontologies/2011/09/01/ArasAssistiveOntologgy#ARAS1104789601000"></div>
    <div property="ccf:representationData" content="file:representations/aras/monitor_(2).png"></div>
    <div rel="ccf:representationType" resource="http://www.conceptcoding.org/ontologies/2011/09/01/ArasAssistiveOntologgy#aras"></div>
    </div>
    </div>
  • The concept monitor presented in CCF Linked Data:
    <div xmlns="http://www.w3.org/1999/xhtml" prefix=" df: http://www.w3.org/1999/02/22-rdf-syntax-ns# ccf: http://www.conceptcoding.org/Ontologies/2014/07/ccf-owl-ns# rdfs: http://www.w3.org/2000/01/rdf-schema#">
    <div typeof="ccf:ConceptCode" about="http://www.conceptcoding.org/ontologies/2011/09/01/ConceptCodeDefinitions#cc-monitor-1001">
    <div property="ccf:lemma" content="monitor"></div>
    </div>
    </div>
  • Base Concept:
    <div xmlns="http://www.w3.org/1999/xhtml" prefix=" rdf: http://www.w3.org/1999/02/22-rdf-syntax-ns# ccf: http://www.conceptcoding.org/Ontologies/2014/07/ccf-owl-ns# rdfs: http://www.w3.org/2000/01/rdf-schema#">
    <div typeof="ccf:RepresentationItem" about="http://www.conceptcoding.org/ontologies/2011/09/01/BaseReferenceOntology#EN-3721-1001">
    <div property="ccf:example" content=""></div>
    <div rel="ccf:representationType" resource="http://www.conceptcoding.org/ontologies/2011/09/01/BaseReferenceOntology#en"></div>
    <div rel="ccf:concept" resource="http://www.conceptcoding.org/ontologies/2011/09/01/BaseReferenceOntology#BRO-110478960-1000"></div>
    <div property="ccf:representationData" content="monitor, monitors"></div>
    </div>
    </div>
  • Mapping (BRO-114078960-1000 and ARAS-110478960-1000 based on concept cc-monitor-1001)
    <div xmlns="http://www.w3.org/1999/xhtml" prefix="rdf: http://www.w3.org/1999/02/22-rdf-syntax-ns# ccf: http://www.conceptcoding.org/Ontologies/2014/07/ccf-owl-ns# rdfs: http://www.w3.org/2000/01/rdf-schema#" >
    <div typeof="ccf:Mapping" about="http://www.conceptcoding.org/Ontologies/2011/09/01/StandardBridge#9900145925429353419727587434">
    <div rel="ccf:relatesConcept" resource="http://www.conceptcoding.org/ontologies/2011/09/01/BaseReferenceOntology#BRO-110478960-1000"></div>
    <div rel="ccf:relatesConceptCode" resource="http://www.conceptcoding.org/ontologies/2011/09/01/ConceptCodeDefinitions#cc-monitor-1001"></div>
    <div rel="ccf:relatesConcept" resource="http://www.conceptcoding.org/ontologies/2011/09/01/ArasAssistiveOntologgy#ARAS-110478960-1000"></div>
    </div>
    </div>

This approach mainly requires the symbol dataseat providers to publish the symbols and their concepts as "Linked Open Data". As one of four principles in Linked Data, the URI naming for the concepts could provide the target concept link described in the first solution. It could also provide the alternative same concept symbols based on preferred properties, such as localization, language and colour etc.