Using the AIDA Language to Formally Organize Scientific Claims
Tobias Kuhn
Sixth International Workshop on Controlled Natural Language (CNL 2018)
Maynooth, Ireland
27 August 2018
These Slides: https://tinyurl.com/cnl2018kuhn
Scientific Papers:
Optimized for Reading Single Work
Scientific Papers:
Bad for Getting More General View
Scientific Papers:
Unused Potential of Software/Databases
We Have: Network of Publications
typical edge: paper cites paper
We Need: Network of Results
typical edge: study supports hypothesis
We Need: Network of Knowledge
typical edge: gene causes disease
Networks of Results/Knowledge
Letting researchers communicate their findings in a way that
- allows for machine-interpretable representations
- is simple and intuitive
- is flexible and practical
- is general
AIDA Approach
We introduced the approach of AIDA Sentences in earlier work:
- Controlled Natural Language
- English sentences that are Atomic, Independent, Declarative, and Absolute
Kuhn, Barbano, Nagy, and Krauthammer. Broadening the Scope of Nanopublications. In Proceedings of the 10th Extended Semantic Web Conference (ESWC). Springer, 2013.
AIDA Sentence: Definition
- Atomic: a sentence describing one thought that cannot be further broken down in a practical way
- Independent: a sentence that can stand on its own, without external references like "this effect" or "we"
- Declarative: a complete sentence ending with a full stop that could (at least in theory) be true or false
- Absolute: a sentence describing the core of a claim ignoring the (un)certainty about its truth and ignoring how it was discovered (without "probably" or "evaluation showed that")
AIDA Sentences: Examples
"A combination of system and searcher biases lead search engine users to settle on the incorrect answer to yes/no-questions around half of the time."(from 10.1145/2484028.2484053)
"Teenagers reply on average faster to emails than adults.''
(from 10.1145/2736277.2741130)
"Deep learning is a powerful and accurate method for automatic speech recognition.''
(from 10.1109/ASRU.2011.6163930, 10.1109/MSP.2012.2205597, and 10.1109/ICASSP.2013.6639347)
Linking AIDA Sentences
Informal, Semi-formal, Formal
Linking Informal, Semi-formal, and Formal Statements
Related Controlled Natural Languages
Basic English (~1930): designed to improve human communication in politics, economy, and science.
Other CNLs (ACE, CLEF) proposed for scientific results: focus on precision
AIDA is unique as a CNL for knowledge representation that focuses on expressivity instead of precision.
Research Questions
About AIDA sentences:
- How easily can researchers write them?
- How easily can they be automatically extracted?
- Are they really general and cover everything?
- Do researchers like them?
- Can they be easily linked and formalized?
- ...
Previous Studies
First study on manual creation of AIDA sentences from abstracts by untrained researchers
Second study on automatic creation of AIDA sentences from texts with a simple algorithm
In both studies, about 70% of the created AIDA sentences were perfectly accurate
Alzheimer's Case Study
Manually representing the main statements of meta-review as AIDA sentences:
- Cochrane Library report on "Cholinesterase inhibitors for Alzheimer's disease"
- Summarizes evidence and findings from various publications on the topic
- Led to 62 AIDA sentences
Alzheimer's Case Study: Results
Open Access Case Study
Representing the statements of another meta-review:
- On Open Access citation advantage
- General AIDA sentence: "Open Access publications receive on average more citations than similar publications that are not Open Access."
- More specific ones, e.g. "Open Access publications in astronomy and physics receive ..."
- Result: AIDA sentences for 70 publications
User Study
AIDA sentences in the classroom setting:
- Course entitled "Knowledge and Media"
- AIDA sentences used for summarizing papers (2015-2017, 20 papers each year)
- Should help students remember and understand the main content of the papers
- Questionnaire about AIDA at end of course
User Study Questions
1. AIDA Sentences: Were the AIDA sentences, as presented during the lectures and on the slides, helpful for you to understand and remember the content of the papers?
- Yes, the AIDA sentences were helpful.
- Maybe. I am not sure whether the AIDA sentences were helpful.
- No, the AIDA sentences were not helpful.
2. AIDA sentences compared to classical text summaries: Did you find the AIDA sentences, as presented during the lectures and on the slides, to be more or less useful than classical text summaries?
- I found the AIDA sentences to be more useful than classical text summaries.
- I found the AIDA sentences to be about as useful as classical text summaries.
- I found the AIDA sentences to be less useful than classical text summaries.
User Study Results
70% found AIDA sentences to be helpful;
only 1.6% found them not helpful
User Study Results
45% prefer AIDA sentences;
only 3.6% prefer classical summaries
Data: 659 AIDA Sentences
Previous work:
- Manual extraction study: 51 AIDA sentences
- Automatic extraction study: 189 AIDA sentences
Case studies:
- Alzheimer's: 62 AIDA sentences
- Open Access: 70 AIDA sentences
Personal collection:
Linking and Network Study
Analyzing the effect of simple post-hoc partial formalization of AIDA sentences.
Use of DBpedia Spotlight: text annotation tool producing links to DBpedia (Wikipedia pages).
Linking and Network Study Results: Accuracy
Manual evaluation of a random sample of 10% of the resulting annotations (173 out of 1726):
94.2% correct
Example:The treatment of Alzheimer's disease with one of the three cholinesterase inhibitors donepezil, galantamine or rivastigmine has a higher probability of at least one adverse event of anorexia before the end of the treatment as compared to a placebo treatment.
Network: AIDA Sentences and Papers
332 network components: largest covers 10%
Network: Existing Concepts Added
167 network components: largest covers 24%
Network: DBpedia Spotlight Links Added
66 network components: largest covers 48%
Conclusion
Results indicate that approach is promising:
- Students found AIDA sentences useful
- AIDA sentences can be automatically connected to Linked Data identifiers at good accuracy
- Linking seems to lead to a dense and broad network of scientific findings
Nanopublications:
Publishing AIDA Sentences with Provenance
