Benchmarking atlas-level data integration in single-cell genomics

single-cell
rna-seq
integration
benchmarking
Authors

Malte D Luecken

Maren Büttner

Kridsadakorn Chaichoompu

Anna Danese

Marta Interlandi

Michaela F Mueller

Daniel C Strobl

Luke Zappia

Martin Dugas

Maria Colomé-Tatché

Fabian J Theis

Date

December 23, 2021

Links
Citation stats
Abstract

Single-cell atlases often include samples that span locations, laboratories and conditions, leading to complex, nested batch effects in data. Thus, joint analysis of atlas datasets requires reliable data integration. To guide integration method choice, we benchmarked 68 method and preprocessing combinations on 85 batches of gene expression, chromatin accessibility and simulation data from 23 publications, altogether representing >1.2 million cells distributed in 13 atlas-level integration tasks. We evaluated methods according to scalability, usability and their ability to remove batch effects while retaining biological variation using 14 evaluation metrics. We show that highly variable gene selection improves the performance of data integration methods, whereas scaling pushes methods to prioritize batch removal over conservation of biological variation. Overall, scANVI, Scanorama, scVI and scGen perform well, particularly on complex integration tasks, while single-cell ATAC-sequencing integration performance is strongly affected by choice of feature space. Our freely available Python module and benchmarking pipeline can identify optimal data integration methods for new data, benchmark new methods and improve method development.

Citation

BibTeX citation:
@article{dluecken2021,
  author = {Malte D Luecken and Maren Büttner and Kridsadakorn
    Chaichoompu and Anna Danese and Marta Interlandi and Michaela F
    Mueller and Daniel C Strobl and Luke Zappia and Martin Dugas and
    Maria Colomé-Tatché and Fabian J Theis},
  title = {Benchmarking Atlas-Level Data Integration in Single-Cell
    Genomics},
  journal = {Nature methods},
  date = {2021-12-23},
  url = {https://lazappi.id.au/publications/2021-luecken-scIB},
  doi = {10.1038/s41592-021-01336-8},
  issn = {1548-7091},
  langid = {en},
  abstract = {Single-cell atlases often include samples that span
    locations, laboratories and conditions, leading to complex, nested
    batch effects in data. Thus, joint analysis of atlas datasets
    requires reliable data integration. To guide integration method
    choice, we benchmarked 68 method and preprocessing combinations on
    85 batches of gene expression, chromatin accessibility and
    simulation data from 23 publications, altogether representing
    \textgreater1.2 million cells distributed in 13 atlas-level
    integration tasks. We evaluated methods according to scalability,
    usability and their ability to remove batch effects while retaining
    biological variation using 14 evaluation metrics. We show that
    highly variable gene selection improves the performance of data
    integration methods, whereas scaling pushes methods to prioritize
    batch removal over conservation of biological variation. Overall,
    scANVI, Scanorama, scVI and scGen perform well, particularly on
    complex integration tasks, while single-cell ATAC-sequencing
    integration performance is strongly affected by choice of feature
    space. Our freely available Python module and benchmarking pipeline
    can identify optimal data integration methods for new data,
    benchmark new methods and improve method development.}
}
For attribution, please cite this work as:
Malte D Luecken, Maren Büttner, Kridsadakorn Chaichoompu, Anna Danese, Marta Interlandi, Michaela F Mueller, Daniel C Strobl, et al. 2021. “Benchmarking Atlas-Level Data Integration in Single-Cell Genomics.” Nature Methods, December. https://doi.org/10.1038/s41592-021-01336-8.