Introduction

Each effect size is nested within an experiment which is in turn nested within a paper (this includes unpublished reports, theses, and the likes). It can be assumed that effect sizes within these nested structures are not independent. Here we explore whether and how accounting for this possible correlation affects both a random effects base model and a moderator analysis. As example we chose InWordDB.

Base Model

Standard random effects model, no moderators. First we run the model without accounting for any hierarchical structure (as reported in the publication by Bergmann & Cristia 2015; note that differences in effect size estimation are due to an updated dataset here that also includes nonnative studies, as compared to the paper).

inworddb <- droplevels(all_data[all_data$short_name=="inworddb", ])

StandardMod <- rma(g_calc, g_var_calc, data = inworddb)
summary(StandardMod)
## 
## Random-Effects Model (k = 286; tau^2 estimator: REML)
## 
##    logLik   deviance        AIC        BIC       AICc  
## -149.5440   299.0881   303.0881   310.3931   303.1307  
## 
## tau^2 (estimated amount of total heterogeneity): 0.1148 (SE = 0.0125)
## tau (square root of estimated tau^2 value):      0.3389
## I^2 (total heterogeneity / total variability):   81.37%
## H^2 (total variability / sampling variability):  5.37
## 
## Test for Heterogeneity: 
## Q(df = 285) = 1347.9098, p-val < .0001
## 
## Model Results:
## 
## estimate       se     zval     pval    ci.lb    ci.ub          
##   0.1822   0.0230   7.9242   <.0001   0.1371   0.2273      *** 
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Two Level Model: Paper

We first add a level for the paper a given effect size was reported in. These effect sizes presumably stem from a batch of studies that were conducted in the same lab in a very similar fashion and by the same set of experimenters, introducing possible correlations.

PerPaperMod <- rma.mv(g_calc, g_var_calc, random = ~ 1 | short_cite, data = inworddb)
summary(PerPaperMod)
## 
## Multivariate Meta-Analysis Model (k = 286; method: REML)
## 
##    logLik   Deviance        AIC        BIC       AICc  
## -324.8584   649.7168   653.7168   661.0218   653.7594  
## 
## Variance Components: 
## 
##             estim    sqrt  nlvls  fixed      factor
## sigma^2    0.0426  0.2063     64     no  short_cite
## 
## Test for Heterogeneity: 
## Q(df = 285) = 1347.9098, p-val < .0001
## 
## Model Results:
## 
## estimate       se     zval     pval    ci.lb    ci.ub          
##   0.1480   0.0285   5.1928   <.0001   0.0922   0.2039      *** 
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Three Level Model: Paper and Experiment

Nested within paper, we introduce a level for experiment number. Experiments can report several effect sizes, for example when infants are run in conditions; slight variations of the same study which are presumed to be even more similar than effect sizes within a paper. A caveat is that conventions on what counts as experiment and what counts as conditions within an experiment might differ across papers.

PerExpPaperMod <- rma.mv(g_calc, g_var_calc, random = ~ factor(expt_num) |  short_cite, data = inworddb)
summary(PerExpPaperMod)
## 
## Multivariate Meta-Analysis Model (k = 286; method: REML)
## 
##    logLik   Deviance        AIC        BIC       AICc  
## -244.3049   488.6098   494.6098   505.5673   494.6952  
## 
## Variance Components: 
## 
## outer factor: short_cite       (nlvls = 64)
## inner factor: factor(expt_num) (nlvls = 15)
## 
##             estim    sqrt  fixed
## tau^2      0.0706  0.2658     no
## rho        0.2384             no
## 
## Test for Heterogeneity: 
## Q(df = 285) = 1347.9098, p-val < .0001
## 
## Model Results:
## 
## estimate       se     zval     pval    ci.lb    ci.ub          
##   0.1534   0.0278   5.5163   <.0001   0.0989   0.2079      *** 
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

To summarize, all these models differ in their effect size estimates, but do not change the statistical outcome. The effect remains small but significantly above 0. Adding the level of experiment number did not dramatically change the result.

Moderator Model from the Paper

#Centering mean age
inworddb$ageC <- inworddb$mean_age-mean(inworddb$mean_age)

StandardMod <- rma(g_calc, g_var_calc, mod = ageC, data = inworddb)
summary(StandardMod)
## 
## Mixed-Effects Model (k = 286; tau^2 estimator: REML)
## 
##    logLik   deviance        AIC        BIC       AICc  
## -149.5873   299.1747   305.1747   316.1216   305.2604  
## 
## tau^2 (estimated amount of residual heterogeneity):     0.1155 (SE = 0.0126)
## tau (square root of estimated tau^2 value):             0.3398
## I^2 (residual heterogeneity / unaccounted variability): 81.43%
## H^2 (unaccounted variability / sampling variability):   5.39
## R^2 (amount of heterogeneity accounted for):            0.00%
## 
## Test for Residual Heterogeneity: 
## QE(df = 284) = 1347.0650, p-val < .0001
## 
## Test of Moderators (coefficient(s) 2): 
## QM(df = 1) = 0.0026, p-val = 0.9590
## 
## Model Results:
## 
##          estimate      se    zval    pval    ci.lb   ci.ub     
## intrcpt    0.1822  0.0230  7.9080  <.0001   0.1371  0.2274  ***
## mods       0.0000  0.0003  0.0514  0.9590  -0.0005  0.0006     
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Two Level Model: Paper

PerPaperMod <- rma.mv(g_calc, g_var_calc, mod = ageC, random = ~ 1 | short_cite, data = inworddb)
summary(PerPaperMod)
## 
## Multivariate Meta-Analysis Model (k = 286; method: REML)
## 
##    logLik   Deviance        AIC        BIC       AICc  
## -321.3824   642.7648   648.7648   659.7117   648.8505  
## 
## Variance Components: 
## 
##             estim    sqrt  nlvls  fixed      factor
## sigma^2    0.0487  0.2206     64     no  short_cite
## 
## Test for Residual Heterogeneity: 
## QE(df = 284) = 1347.0650, p-val < .0001
## 
## Test of Moderators (coefficient(s) 2): 
## QM(df = 1) = 8.0314, p-val = 0.0046
## 
## Model Results:
## 
##          estimate      se    zval    pval   ci.lb   ci.ub     
## intrcpt    0.1451  0.0302  4.8076  <.0001  0.0859  0.2042  ***
## mods       0.0006  0.0002  2.8340  0.0046  0.0002  0.0010   **
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Three Level Model: Paper and Experiment

PerExpPaperMod <- rma.mv(g_calc, g_var_calc, mod = ageC, random = ~ factor(expt_num) |  short_cite, data = inworddb)
summary(PerExpPaperMod)
## 
## Multivariate Meta-Analysis Model (k = 286; method: REML)
## 
##    logLik   Deviance        AIC        BIC       AICc  
## -241.7889   483.5779   491.5779   506.1738   491.7212  
## 
## Variance Components: 
## 
## outer factor: short_cite       (nlvls = 64)
## inner factor: factor(expt_num) (nlvls = 15)
## 
##             estim    sqrt  fixed
## tau^2      0.0760  0.2757     no
## rho        0.2985             no
## 
## Test for Residual Heterogeneity: 
## QE(df = 284) = 1347.0650, p-val < .0001
## 
## Test of Moderators (coefficient(s) 2): 
## QM(df = 1) = 5.6924, p-val = 0.0170
## 
## Model Results:
## 
##          estimate      se    zval    pval   ci.lb   ci.ub     
## intrcpt    0.1509  0.0296  5.1004  <.0001  0.0929  0.2089  ***
## mods       0.0006  0.0003  2.3859  0.0170  0.0001  0.0011    *
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

To summarize, introducing hierarchical structure changed the outcome for the moderator test. Age (centered) now has a small, but significant, effect on effect sizes. This is not the case when ignoring the nested structure of effect sizes. This result mirrors the reported analyses in Bergmann & Cristia (2015) that there is a small, positive effect of age when only considering papers which test at least two age groups in the same set-up.