Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.


The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2016 Jun;12(6):645-53.
doi: 10.1016/j.jalz.2016.02.006. Epub 2016 Apr 11.

Crowdsourced estimation of cognitive decline and resilience in Alzheimer's disease

Genevera I Allen  1 Nicola Amoroso  2 Catalina Anghel  3 Venkat Balagurusamy  4 Christopher J Bare  5 Derek Beaton  6 Roberto Bellotti  2 David A Bennett  7 Kevin L Boehme  8 Paul C Boutros  9 Laura Caberlotto  10 Cristian Caloian  3 Frederick Campbell  1 Elias Chaibub Neto  5 Yu-Chuan Chang  11 Beibei Chen  12 Chien-Yu Chen  13 Ting-Ying Chien  14 Tim Clark  15 Sudeshna Das  15 Christos Davatzikos  16 Jieyao Deng  17 Donna Dillenberger  4 Richard J B Dobson  18 Qilin Dong  17 Jimit Doshi  16 Denise Duma  19 Rosangela Errico  20 Guray Erus  16 Evan Everett  1 David W Fardo  21 Stephen H Friend  5 Holger Fröhlich  22 Jessica Gan  1 Peter St George-Hyslop  23 Satrajit S Ghosh  24 Enrico Glaab  25 Robert C Green  26 Yuanfang Guan  27 Ming-Yi Hong  13 Chao Huang  28 Jinseub Hwang  29 Joseph Ibrahim  28 Paolo Inglese  30 Anandhi Iyappan  31 Qijia Jiang  1 Yuriko Katsumata  32 John S K Kauwe  33 Arno Klein  34 Dehan Kong  28 Roland Krause  25 Emilie Lalonde  3 Mario Lauria  10 Eunjee Lee  28 Xihui Lin  3 Zhandong Liu  1 Julie Livingstone  3 Benjamin A Logsdon  5 Simon Lovestone  35 Tsung-Wei Ma  12 Ashutosh Malhotra  31 Lara M Mangravite  36 Taylor J Maxwell  37 Emily Merrill  38 John Nagorski  1 Aishwarya Namasivayam  25 Manjari Narayan  1 Mufassra Naz  31 Stephen J Newhouse  39 Thea C Norman  5 Ramil N Nurtdinov  40 Yen-Jen Oyang  11 Yudi Pawitan  41 Shengwen Peng  17 Mette A Peters  42 Stephen R Piccolo  8 Paurush Praveen  43 Corrado Priami  10 Veronica Y Sabelnykova  3 Philipp Senger  44 Xia Shen  45 Andrew Simmons  46 Aristeidis Sotiras  16 Gustavo Stolovitzky  47 Sabina Tangaro  48 Andrea Tateo  49 Yi-An Tung  50 Nicholas J Tustison  51 Erdem Varol  16 George Vradenburg  52 Michael W Weiner  53 Guanghua Xiao  12 Lei Xie  54 Yang Xie  12 Jia Xu  12 Hojin Yang  28 Xiaowei Zhan  12 Yunyun Zhou  12 Fan Zhu  55 Hongtu Zhu  28 Shanfeng Zhu  56 Alzheimer's Disease Neuroimaging Initiative
Free PMC article

Crowdsourced estimation of cognitive decline and resilience in Alzheimer's disease

Genevera I Allen et al. Alzheimers Dement. 2016 Jun.
Free PMC article


Identifying accurate biomarkers of cognitive decline is essential for advancing early diagnosis and prevention therapies in Alzheimer's disease. The Alzheimer's disease DREAM Challenge was designed as a computational crowdsourced project to benchmark the current state-of-the-art in predicting cognitive outcomes in Alzheimer's disease based on high dimensional, publicly available genetic and structural imaging data. This meta-analysis failed to identify a meaningful predictor developed from either data modality, suggesting that alternate approaches should be considered for prediction of cognitive performance.

Keywords: Azheimer's disease; Big data; Bioinformatics; Biomarkers; Cognitive decline; Crowdsource; Genetics; Imaging.


Fig. 1
Fig. 1. Challenge overview
The top schematic summarizes the three challenge questions on the left column, the training data in the middle, and the test data on the right, including numbers of subjects. The symbols represent sources of data (demographic, ROS/MAP genetic, and ADNI or ANM brain images and shape information). The bottom panel provides example brain image labels and shape information provided to the participants for question 3. Anatomical labels for left cortical regions are shown on the left and just a couple of the cortical surface shape measures are shown on the right (travel depth on top and mean curvature below), for both uninflated and inflated surfaces (top and bottom rows, respectively).
Figure 2
Figure 2. Performance evaluation results
Panels a, b, and c report the p-values (in negative log10 scale) for intersection union tests investigating which teams performed better than random for questions 1, 2, and 3, respectively. Explicitly, for question 1 (panel a) we tested the null hypothesis that at least one of the four correlation coefficients (namely, Pearson/clinical, Pearson/clinical + genetics, Spearman/clinical, Spearman/clinical + genetics) is equal to zero, against the alternative that all four correlation coefficients are larger than zero. Adopting a 0.05 significance level, 26 out of the 32 submissions were statistically better than random, after Bonferroni multiple testing correction for 32 tests (submissions crossing the black vertical line). For question 2 (panel b), we tested the null hypothesis that balanced accuracy = 0.5 or AUC = 0.5, against the alternative that balanced accuracy > 0.5 and AUC > 0.5. In this case, no model performed significantly better than random and, therefore, no best performer was declared. For question 3 (panel c), we tested the null hypothesis that Pearson’s correlation (COR) or Lin’s concordance correlation coefficient (CCC) are equal to zero, against the alternative that both COR and CCC are larger than zero. Adopting a 0.05 significance level, all 23 submissions were statistically better than random, after Bonferroni correction. For all three questions, the p-values were computed from an empirical null distribution based on 10,000 permutations. Panels d and e report the bootstrapped assessment of ranks for questions 1 and 3, respectively. Samples were resampled with replacement from the original data (true outcome and team’s predictions), and the ranks of the different teams were re-assessed in each of 100,000 re-samplings. Submissions were sorted according to the median of their bootstrapped average ranking distributions. The black horizontal line represents the posterior odds cutoff from the Bayesian analysis. Teams above the black line are statistically tied to the top ranked model, according to a posterior odds threshold of 3.

Similar articles

Cited by

Publication types

MeSH terms