gilmore-csc-talk

• Collect diverse types of data
• Must aggregate, link data across space, time, individual identities
• Data not spatially uniform
• Time series not uniformly sampled, different sampling intervals

• Aggregating big data about individuals poses privacy risks

• Most data collected are from WEIRD populations (Henrich, Heine, and Norenzayan 2010)
• Studies underpowered (Maxwell 2004)
• Findings hard to reproduce/replicate

"We have empirically assessed the distribution of published effect sizes and estimated power by extracting more than 100,000 statistical records from about 10,000 cognitive neuroscience and psychology papers published during the past 5 years…False report probability is likely to exceed 50% for the whole literature. In light of our findings the recently reported low replication success in psychology is realistic and worse performance may be expected for cognitive neuroscience."

(Szucs and Ioannidis 2016)

• Still collect data in non-electronic formats
• Even electronic formats not readily shareable
• Vital metadata (geo-, participant-level) often not collected
• "Reproducible" workflows not standard practice
• Results have limited robustness and generalizability
• Misunderstanding/agreement about what reproducibility means

• Methods reproducibility refers to the provision of enough detail about study procedures and data so the same procedures could, in theory or in actuality, be exactly repeated.
• Results reproducibility (previously described as replicability) refers to obtaining the same results from the conduct of an independent study whose procedures are as closely matched to the original experiment as possible.

(Goodman, Fanelli, and Ioannidis 2016)

• Robustness refers to the stability of experimental conclusions to variations in either baseline assumptions or experimental procedures.
• Generalizability refers to the persistence of an effect in settings different from and outside of an experimental framework.

(Goodman, Fanelli, and Ioannidis 2016)

• Reproducible methods
• Reproducible results
• Robust fundings
• Generalizable findings

• A.D. = After Databrary

Gilmore, R. O. (2016). From big data to deep insight in developmental science. Wiley Interdisciplinary Reviews: Cognitive Science, 7(2), 112–126. https://doi.org/10.1002/wcs.1379

• National Children's Study (NCS)
• U.K. Life Study

• National Children's Study (NCS)
• U.K. Life Study

• We don't agree about who owns data
  • Participants
  • Us
  • Penn State
  • The taxpayer
• Minimal rewards for data sharing
• Post hoc sharing hard, time-consuming, expensive

• Why share if no one will reuse, build upon?
• Journals don't always encourage/support/mandate publication of data, detailed methods
  • Guidelines for Transparency and Openness Promotion from the Center for Open Science (COS)
• Building community consensus better than centralized mandates

http://datausa.io

• Numeric, text-based measures miss/reduce complexity of behavior
• Video captures and preserves it
• Replications can fail due to methodological differences
• Methods sections can't possibly report essential details
• Video captures and preserves them

Gilmore et al. talk

• Video record all tasks, measures, and behaviors
• Share the recordings
• Share all questionnaires, tasks, displays
• Share statistical, computational, data workflows
• Prepare to share from the beginning
• Seek permission to share data

• But, not impossible

• Contribute to public good
• Aid discovery
• Curiosity
• Help institution
• How can we better capitalize on the value the public places on our work?

• Are your preferred standardized/normed tasks open, non-proprietary?
• Can you share the questions/items with colleagues?
• Are the data underlying the norms openly available?

• Wechlser
• Autism Diagnostic Observation Schedule (ADOS)
• MacArthur CDI
• Achenbach System of Empirically Based Assessment (ASEBA), e.g., CBCL, etc.

• Wechlser
• Autism Diagnostic Observation Schedule (ADOS)
• MacArthur CDI
• Achenbach System of Empirically Based Assessment (ASEBA), e.g., CBCL, etc.
• Difficult to find, aggregate across studies, and reuse/reanalyze data drawn from most measures (Gilmore 2016)

• Recruiting – larger, more diverse samples
• Data collection – more data types, allow linkage across levels
• Data curation/management – easy/automatic, standardized formats
• Data sharing – PI controls when, permission levels

• Data mining, visualization, linking
• Search, filter by participant characteristics, tasks/measures, geo/temporal factors
• Analysis in the "cloud"
• Automatic versioning, history

• App/web service (MeeSearch.com)
• Linking researchers with participants (or parents)
• Participants own/control their data, determine level of sharing (like datawallet.io)
• Lab, computer/smart-phone based, survey tasks
• Data visualizations, dashboard
• 1,000+ psych pool/semester, 500K PSU alumni, 1M friends

• Shall we build it?
• After all…