My packages

New packages

• {BatChef} - benchmark batch correction methods for scRNA-seq data and help pick an appropriate one
• {Battlefield} - low-level utilities for working with spatial transcriptomics regions, interfaces and layers
• {betterChromVAR} - faster chromVAR-style inference of TF activity for bulk and single-cell ATAC-seq
• {CellMentor} - supervised dimensionality reduction that tries to preserve known cell-type structure
• {DenoIST} - removes neighbourhood contamination from image-based spatial transcriptomics data
• {dominatR} - visualises feature dominance using concepts from physics
• {GraphExperiment} - extends SingleCellExperiment with infrastructure for storing feature-level networks
• {hammers} - utilities package for scRNA-seq analysis using both Seurat and SingleCellExperiment
• {jvecfor} - faster nearest-neighbour search for large single-cell datasets with drop-in replacements for common Bioconductor workflows
• {RankMap} - fast reference-based cell type annotation for single-cell and spatial transcriptomics data
• {scConform} - cell type annotation with conformal prediction intervals and uncertainty quantification
• {scECODA} - workflow for analysing cell type proportions as compositional data
• {scLang} - developer-facing helpers for writing scRNA-seq packages that work with both Seurat and SingleCellExperiment
• {scPassport} - stores a persistent metadata passport inside Seurat and SingleCellExperiment objects
• {scTypeEval} - tools for evaluating cell type assignments with limited ground truth data
• {SpatialArtifacts} - quality control for identifying spatial artifacts in Visium and Visium HD data
• {SpNeigh} - neighbourhood-aware spatial transcriptomics analysis including boundary detection and spatial differential expression
• {tidyprint} - tidier print methods for SummarizedExperiment objects
• {VISTA} - wraps differential expression workflows and visualisation in a SummarizedExperiment-based container
• {ZarrArray} - DelayedArray-backed infrastructure for working with Zarr datasets in R

Updates

• {DropletUtils} - downsampleReads() now uses the faster downsampling algorithm from {scuttle}
• {edgeR} - new DGEListFromTximport() and DGEListFromTximeta() helpers plus a sampleWeights() function
• {limma} - voom() gains offset support and topTableF() is now finally removed
• {Rarr} - major updates including moving the DelayedArray backend into the new {ZarrArray} package and improved Zarr v3 support
• {Rhdf5lib} - build updates and an update to HDF5 1.14.6
• {rhdf5} - updated to HDF5 1.14.6 and various fixes and improvements
• {scran} - deprecates several functions in favour of {scrapper} and fixes overflow bugs
• {scrapper} - multiple updates to several functions, continues the migration of functionality out of {scran} and {scuttle}
• {scuttle} - faster downsampleMatrix() and summarizeAssayByGroup(), more deprecations in favour of scrapper
• {SingleCellExperiment} - improved warnings for named assay getters and setters
• {tximeta} - matching updates for {edgeR}’s new DGEListFromTximeta() workflow
• {tximport} - vignette updates around the new {edgeR} integration

Motivation

Like for many steps in scRNA-seq analysis, integration methods typically recommend selecting a subset of genes to combine datasets. In a previous benchmark² we found that using highly variable genes did in fact improve integration performance compared to using all features but there remained some open questions:

1. What feature selection method to use? In the scIB benchmark we used 2000 batch-aware highly variable genes but there are many other feature selection methods, some of which have been shown to perform better in benchmarks of other tasks.
2. What effect does feature selection on using an integrated reference? The scIB benchmark looked at how well datasets were integrated but didn’t consider mapping new datasets to that reference.

The second question was of particular interest to me. We had already shown that feature selection improves integration but what about using that integrated dataset as a reference? Whenever I had see an example of mapping data to a reference I add also wondered about what happens to any variation in the features not included in the model. For example, can we reliably identify new populations in a mapped query dataset if the variation that separates them is in genes that the model knows nothing about?

As always, the scope of the benchmark grew a bit beyond this but this was the starting point for the study.

Study design

We followed the standard benchmarking setup of selecting some test datasets, finding methods to compare and using a set of metrics to evaluate them. The datasets were split into query and reference sets. Features were then selected on the reference and used to integrate it, before mapping the query and classifying the query cells. We selected some cell populations to be only present in the query which we tried to identify after mapping.

There are few things about how we did the study that I think are interesting.

Metric selection

The first results section in the paper doesn’t involve feature selection methods at all, instead it focuses on comparing and selecting the metrics to use. As mentioned above, this was somewhat forced upon us by the paper format but I think it was a worthwhile exercise that should receive more attention. In most benchmarks this would probably be done informally (if at all) but because we had to include it in the project proposal we had to formalise the process.

In the development phase of the project we had collected as many metrics as possible that had been used in previous benchmarks or method comparisons.² We divided these into five categories: removal of batch effects during integration (Integration (Batch)), conservation of biological variance during integration (Integration (Bio)), mapping of a query to the reference (Mapping), label projection to cells in the query (Classification) and detection of new populations in the query (Unseen). There were some that we excluded straight away (because they didn’t have a useable implementation or didn’t fit into our framework) but we tried to implement as many as possible. In the metric selection step, we then wanted to decide which metrics to consider. There were a few criteria we considered:

• Does the metric have a useable dynamic range (i.e. does it actually measure something in our scenario)?
• Is the metric correlated with technical factors like the number of selected features?
• Is the metric redundant (i.e. is it overly correlated with other metrics in the same category)?
• Is the metric correctly categorised (i.e. is it correlated with metrics in the same category and anti-correlated with metrics in other categories)?

We did this by running the benchmark using sets of randomly selected features of different sizes which allowed us to investigate metric behaviour without biasing towards any particular method. We also used a highly variable gene method to check correlation with the number of selected features as random sets do not have any inherent ordering. This worked well in our benchmark but could be more challenging in other scenarios where it is not as easy to simulate the methods you are trying to evaluate.

The metric selection step allowed us to choose a set of metrics that we were confident reliably measured what were were interested in evaluating.³

Results

Now that I have explained how we did the study, let’s talk about the results! I’m going to mention some of the key points but you should really read the paper for the details.

Summary

Ok, so after all that, what did we learn?

• The registered report is an interesting format but it requires you to do a lot of work in advance and doesn’t necessarily make things quicker
• Setting up a proper workflow takes some effort but is invaluable when you need to re-run things, especially in a large, computationally-intense project
• Choosing the metrics to use is vital and something that more time should be spent on in benchmarking papers
• Think carefully about how to scale and combine metric scores, use baseline methods if you can
• Around 2000 features generally works well but you should tune this for you dataset and application
• Using batch-aware features doesn’t add much
• Integrating lineages separately results in worse detection of unseen populations
• scANVI generally improves performance over scVI regardless of the feature selection method but Symphony struggled with detecting new populations

Thanks for reading! You can find more about this study here:

• Publication
• GitHub repository
• figshare collection

Keynote sketchotes

Here are my sketchnotes summarising the keynote talks (click to expand):

Hang on, what’s scRNA-tools?

scRNA-tools is a database that catalogues software tools for analysing scRNA-seq data which I created during my PhD and have curated and maintained ever since. You can read more about it on the project page, in the original paper and in our analysis of the first 1000 tools in the database.

So, what’s happening?

In May 2024 I will be putting scRNA-tools into what I am calling “low-maintenance mode”. Basically, this means I will stop actively seeking out new tools to add to the database and updates for existing tools. I will still make changes based on user submissions and the existing automated checks but the rate of updates will significantly decrease. Upgrades to the database, website and other code are unlikely to happen unless things break.

That’s sad 😿, why now?

I’m moving onto the next step in my career and can no longer justify the 2-3 hours per week it takes review new papers, curate them and add tools to the database. Transiting to low-maintenance mode means less frequent updates and should reduce the commitment to something I can keep up with, similar to the software packages I maintain. I would love to keep scRNA-tools running as it has but the continuous, ongoing nature of the curation makes it difficult when you can no-longer justify it as “work”, and this will only increase as the field continuous to grow.

I also have to acknowledge that I haven’t developed scRNA-tools as I would have liked over the last few years. There are several new categories and features I wanted to add but, as the typical academic says, I never found the time.

Maybe someone else could take over?

I always wanted scRNA-tools to be a community project so I am definitely open to this but I would need to be convinced that whoever takes over would be able to commit to actively maintaining the project. In the past I have had people reach out to contribute, and even recruited a team to help with curation and expanding the database, but that always petered out after a few weeks or months. I totally don’t blame anyone for that, there is very little for a PhD student to gain from contributing to a project like this, but it does make me cautious about handing over the project to someone who may not end up putting more time into it than I can.

Last thoughts

scRNA-tools has helped record the first years of the single-cell genomics revolution and has been a big part of my work life for the last eight or so years. While I think it is a valuable resource it is also limited, especially as the field expands into other modalities. It will be sad it see it not updated as frequently but it’s time for us both to move on.

My packages

New packages

• {ADImpute} - dropout imputation using information from gene regulatory networks
• {aggregateBioVar} - provides tools to aggregating SummarizedExperiment objects at the subject level
• {BayesSpace} - tools for clustering tha enhancing spatial gene expression
• {BiocIO} - generics for importing and exporting biological data
• {biocthis} - automate setting up packages for Bioconductor
• {bluster} - wraps common clustering algorithms for Bioconductor
• {corral} - correspondence analysis for single-cell data
• {densvis} - implements the density-preserving modifications to t-SNE and UMAP
• {escape} - bridging package for scRNA-seq gene set enrichment
• {ExperimentSubset} - interface for accessing subsets of SummarizedExperiment objects
• {Herper} - interface for managing conda environments
• {ILoReg} - high-resolution scRNA-seq population identification
• {Informeasure} - implementation of information measures such as mutual information
• {Nebulosa} - visualisation of scRNA-seq using gene-weighted density estimation
• {NewWave} - dimensionality reduction and batch correction for scRNA-seq
• {pipeComp} - simple framework for comparing pipelines
• {recount3} - access data from recount3
• {scCB2} - extends the EmptyDrops method for identifying real cells by testing clusters
• {scDataviz} - functions for plotting single-cell data
• {scuttle} - basic functions for single-cell analysis (most previously in {scater})
• {snifter} - R wrapper for the Python openTSNE library
• {SpatialExperiment} - S4 class for storing spatial experiments
• {SPsimSeq} - Semi-parametric RNA-seq simulation
• {velociraptor} - R wrapper for the scVelo Python package

Updates

• {BASiCS} - many updates and improvements
• {basilisk} - support for setting conda channels and safer environment construction
• {DESeq2} - overhaul of dispersion estimation allowing use of the {glmGamPoi} package for single-cell data
• {DropletUtils} - functions for handling ambient counts
• {edgeR} - support for SummarizedExperiment objects and improvements to the limma voom-lmFit pipeline
• {limma} - improvements to some fitting functions
• {scran} - various new function arguments, some functions moved to other packages

Getting the data

triple j has a recently played page on their website that has a live stream of the songs they are playing. This page also has a search function that let’s you find which song was played at a particular time. I will note that there may be some errors in this list, there was at least one time I heard a song played but saw something else get added to the list. To get the raw data I simply searched for each day of the Requestival, clicked the “Show more” button until all the songs for that day were shown and then downloaded the HTML page. It’s probably possible to do this all programmatically but it was only a few pages so it didn’t seem worth the effort. Thankfully the HTML was fairly simple and neat so it was relatively easy to extract the information I wanted using {rvest}. What I ended up with was a tibble with times, song name, artist and release and links to Spotify, YouTube and the triple j Unearthed website. There were 27 songs that didn’t have release information on the website for some reason. See the scraping page of the analysis website for more details of this part.

There was a bit more tidying I wanted to before any analysis. This was things like converting the file names into days, converting the time strings to hms objects in the correct timezone and replacing Unicode characters. The Spotify and YouTube links included queries which it might be useful to have so I also extracted those into new columns and created a column to indicate if a song was from Unearthed. The final tidying step was to select songs played between 6 am and 9 pm as these were the official hours of the Requestival. See the details here.

After all this I ended up with a dataset with 1187 songs from 896 artists off 1049 releases.

Augmentation

There isn’t much analysis you can do on a simple list of songs so I wanted to augment the data in some way. I have seen some cool analysis using information from Spotify before so I thought that would be a good source to try. Despite the {spotifyr} package providing a nice interface to the Spotify API this wasn’t quite as easy as I had hoped. I initially wanted to use the queries I had extracted from the Spotify links but these weren’t in the format that {spotifyr} needs so I had to construct them manually. I also ran into the problem that the combination of song, artist and release was both too specific and not specific enough. For some songs I got no results when I searched for all this information while for other songs I got many results. These were often duplicates due to releases with slight differences in different countries etc. but sometimes they were unrelated songs that just happened to have enough key words in common.

In the end I searched first for song, artist and release and took the first result if there was one, if not I searched for just song and artist. The only other constraint I had was that the album release date had to have a precision of “day” (an exact date, not just a month/year). I found that was able rule out some bad matches. In the end I was able to find matches for 994 songs but these aren’t perfect. There are probably some incorrect matches and there are a few I have seen where the song was correct but the best version wasn’t selected, for example the match for The Mamas & The Papas “California Dreamin’” had an album date from 2020 even though the album “If You Can Believe Your Eyes And Ears” was first released in 1966. This is the reality of working with real world data I guess and the majority of matches should be close enough to still be useful. See the augmentation page if you are interested in all the details.

Once I had the track ideas for these songs it was relatively simple to extract more information about them from Spotify including the album release date, song duration, whether it is marked as explicit and a Spotify “popularity” score. Spotify also has more musical information about each track. There are some basic things like whether the track is in a major key, the loudness and tempo but also complex scores between zero and one for “valence”, “energy”, “danceability”, “speechiness”, “acousticness” and “liveness”. See the Spotify page for more details but basically these try to capture the general feel of each song.

Analysis

Outro

This is really only an exploratory analysis and there are lots more things you could do: clustering to group songs, looking at correlations, text analysis etc. I had hoped to find some of the really unusal things that were played (like the Pokemon theme song) but I don’t think they are musically different enough for these characteristics to pull them out. In the end I would call this project a partial success, I didn’t find anything super interesting but I need learn some things and it was fun to play around with. The dataset is available on GitHub and I would be interested to see if you can pull anything else out of it. Maybe I will return to it and do something more detailed in the future.

Getting package network

{BiocPkgTools} is now a fully-fledged Bioconductor package and has some new functionst that make it easy to get information about the packages connected to {SingleCellExperiment}.

# Get Bioconductor package information
bioc_pkgs <- biocPkgList()
# Get Bioconductor dependencies
all_bioc_deps <- buildPkgDependencyDataFrame()
# Select reverse dependencies for SingleCellExperiment
bioc_revdeps  <- filter(all_bioc_deps, dependency == "SingleCellExperiment")
# Select dependencies of the reverse dependencies
bioc_edges <- bioc_revdeps %>%
    bind_rows(
        filter(all_bioc_deps, Package %in% bioc_revdeps$Package)
    )

The information from {BiocPkgTools} includes CRAN packages that are dependencies of Bioconductor packages but doesn’t include any CRAN packages that depend on {SingleCellExperiment}. To get those we will use a tidier version of the function in the previous blog post. This function just uses {purr} to loop over the different types of dependencies and calls the tools::package_dependencies() function.

get_cran_deps <- function(pkgs, db, reverse = FALSE,
                          types = c("Depends", "Imports", "Suggests")) {

    deps <- purrr::map_dfr(types, function(.type) {
        deps_list <- tools::package_dependencies(
            pkgs, db,
            which   = .type,
            reverse = reverse
        )
        tibble(Package = pkgs, dependency = deps_list, edgetype = .type)
    }) %>%
        unnest(dependency)

    if (reverse) {
        deps <- dplyr::select(
                deps,
                Package = dependency,
                dependency = Package,
                edgetype
            )
    }

    return(deps)
}

db <- available.packages(repos = "http://cran.r-project.org")
cran_revdeps <- get_cran_deps("SingleCellExperiment", db, reverse = TRUE)
cran_edges <- cran_revdeps %>%
    bind_rows(
        get_cran_deps(cran_revdeps$Package, db)
    )

Now we can combine the networks from Bioconductor and CRAN and create a graph structure.

revdeps <- bioc_revdeps %>%
    bind_rows(cran_revdeps) %>%
    distinct() %>%
    mutate(
        Repo = if_else(Package %in% bioc_pkgs$Package, "Bioconductor", "CRAN")
    )

nodes <- revdeps %>%
    select(Package, Repo) %>%
    add_row(Package = "SingleCellExperiment", Repo = "Bioconductor")

edges <- bioc_edges %>%
    bind_rows(cran_edges) %>%
    filter(dependency %in% c("SingleCellExperiment", revdeps$Package)) %>%
    distinct()

graph <- tbl_graph(nodes = nodes, edges = edges)

We now have a graph that contains {SingleCellExperiment} and the 77 packages that depend on it. We also have edges between the reverse dependencies but have removed edges to other packages.

What uses {SingleCellExperiment}?

ggplot(revdeps, aes(x = Repo, fill = Repo)) +
    geom_bar() +
    scale_fill_brewer(palette = "Dark2") +
    theme_minimal()

edges %>%
    filter(dependency == "SingleCellExperiment") %>%
    ggplot(aes(x = edgetype, fill = edgetype)) +
    geom_bar() +
    scale_fill_brewer(palette = "Dark2") +
    theme_minimal()

bioc_views <- bioc_pkgs %>%
    filter(Package %in% revdeps$Package) %>%
    select(biocViews) %>%
    unnest(biocViews) %>%
    mutate(biocViews = fct_lump_n(fct_infreq(biocViews), 30))

ggplot(bioc_views, aes(x = fct_rev(biocViews))) +
    geom_bar() +
    coord_flip() +
    ggtitle("Most common biocViews") +
    ylab("Number of packages") +
    theme_minimal() +
    theme(axis.title.y = element_blank())

Relationships between packages

Now that we have a bit of an overview of what packages there are let’s see how they relate to each other. First let’s plot all the packages in our graph. Remember that everything here depends on {SingleCellExperiment} and we have links between them but not dependencies to other packages.

ggraph(graph, layout = "fr") +
    geom_edge_fan(aes(colour = edgetype),
                  arrow = arrow(length = unit(4, 'mm')),
                  end_cap = circle(3, 'mm')) +
    geom_node_point(aes(colour = Repo)) +
    geom_node_text(aes(label = Package, colour = Repo), repel = TRUE) +
    scale_color_brewer(palette = "Set1") +
    scale_edge_color_brewer(palette = "Dark2") +
    theme_graph()

A lot of these packages have no other dependencies in this community except for {SingleCellExperiment}. While I’m sure that they are very useful they aren’t very interesting for this analysis so let’s remove them by excluding nodes with a single edge.

graph_deg2 <- graph %>%
    activate(nodes) %>%
    mutate(Degree = igraph::degree(graph)) %>%
    filter(Degree >= 2)

ggraph(graph_deg2, layout = "fr") +
    geom_edge_fan(aes(colour = edgetype),
                  arrow = arrow(length = unit(4, 'mm')),
                  end_cap = circle(3, 'mm')) +
    geom_node_point(aes(colour = Repo)) +
    geom_node_text(aes(label = Package, colour = Repo), repel = TRUE) +
    scale_color_brewer(palette = "Set1") +
    scale_edge_color_brewer(palette = "Dark2") +
    theme_graph()

That’s better but it’s still pretty crowded. Let’s see if we can pick out the most important nodes using a centrality measure.

graph_central <- graph_deg2 %>%
    activate(nodes) %>%
    mutate(Centrality = centrality_authority()) %>%
    filter(Centrality > 0.1)

ggraph(graph_central, layout = "fr") +
    geom_edge_fan(aes(colour = edgetype),
                  arrow = arrow(length = unit(4, 'mm')),
                  end_cap = circle(3, 'mm')) +
    geom_node_point(aes(colour = Repo)) +
    geom_node_text(aes(label = Package, colour = Repo), repel = TRUE) +
    scale_color_brewer(palette = "Set1") +
    scale_edge_color_brewer(palette = "Dark2") +
    theme_graph()

No surprises there! This has picked out what are probably the most used and influential R packages for scRNA-seq analysis. The central Bioconductor packages of {SingleCellExperiement} (object), {scater} (quality control and visualisation) and {scran} (normalisation and downstream analysis). They are joined by {Seurat} which is perhaps the most complete R scRNA-seq analysis package but is hosted on CRAN and uses it’s own object. It’s centrality to this graph suggests that many Bioconductor packages provide some kind of support for {Seurat} objects.

Let’s zoom out a little bit by relaxing the centrality threshold and see what else we find.

graph_central <- graph_deg2 %>%
    activate(nodes) %>%
    mutate(Centrality = centrality_authority()) %>%
    filter(Centrality > 0.05)

ggraph(graph_central, layout = "fr") +
    geom_edge_fan(aes(colour = edgetype),
                  arrow = arrow(length = unit(4, "mm")),
                  end_cap = circle(3, "mm")) +
    geom_node_point(aes(colour = Repo)) +
    geom_node_text(aes(label = Package, colour = Repo), repel = TRUE) +
    scale_color_brewer(palette = "Set1") +
    scale_edge_color_brewer(palette = "Dark2") +
    theme_graph()

This has added a few more central packages: {iSEE} which is a Shiny app for interacting with {SingleCellExperiment} objects, {MAST} for differential expression testing, {zinbwave} for dimensionality reduction and integration and {destiny} for creating diffusion maps.

Conclusion

It is great to see the expansion of the SCE-verse! The object has been well taken up by the community and I think there is now a better understanding of the value of using standard objects, both in increased interoperabilty between packages but all time and effort saved during development. I had hoped to see more connections between packages though. Apart from a few central packages there aren’t that many dependencies with the SCE-verse which suggests there could be a lot of duplicated functionality within packages. Now that we have a common object to work with perhaps the next step forward in the development of the ecosystem is to centralise and reuse common functions so that developers can focus on innovative new methods? I have been fairly selective in this analysis though so it is possible I have missed some of this which is happening already.

It would (still) be good with repeat this for other major object, perhaps I will get to that in a future post.

New packages

• {basilisk} - installs a self-contained Python instance that is maintained by the R installation
• {BiocDockerManager} - management of Bioconductor Docker images
• {CiteFuse} - suite of methods for working with CITE-seq data
• {clustifyr} - classify cells in scRNA-seq data using external references
• {cmapR} - interface for the Broad Institute Connectivity Map resource of gene perturbation expression profiles
• {ctgGEM} - streamlines building of cell-state hierarchies from single-cell gene expression data
• {distinct} - differential testing of distributions
• {dittoSeq} - user friendly visualisation of bulk and single-cell RNA-seq data including consideration of colour blindness
• {Dune} - merges pairs of clusters to increase ARI and improve reproducibility
• {frenchFISH} - Poisson models for DNA copy number from FISH data (bonus points for the excellent name)
• {GeneTonic} - Shiny app for looking at enrichment results from expression data (another great name)
• {HIPPO} - scRNA-seq feature selection and clustering
• {ISEEu} - extensions for the {iSEE} shiny app
• {mitch} - multi-contrast enrichment analysis
• {peco} - predicting cell-cycle progression using scRNA-seq data
• {scClassify} - multi-scale classification of scRNA-seq data using cell hierarchies
• {scHOT} - testing changes in higher-order structure of gene expression such as across (pseudo) time or space
• {scry} - count-based feature selection and dimensionality reduction for small count data
• {scTHI} - identify active ligand-receptor pairs in scRNA-seq data
• {SingleCellSignalR} - clustering of scRNA-seq data and inference of cell-cell interactions
• {sparseMatrixStats} - high-performance functions for row and column operations on sparse matrices, inspired by {matrixStats}
• {tidybulk} - tidy wrappers for bulk RNA-seq analysis

Updates

• {BASiCS} - many updates and release of version 2
• {batchelor} - support of arbitrary design matrices and extension of MNN integration to cluster centroids
• {DropletUtils} - down sampling of batches, writing to 10x format, removal of chimeric reads, demultiplexing of cell hashing experiments
• {edgeR} - integration of the limma voom-lmFit pipeline, support for {SummarizedExperiment} objects
• {iSEE} - support of extensions such as those in {ISEEu}
• {limma} - changeLog() function can be used with any package, improved treatment of NA values
• {missMethyl} - bug fixes and fract.counts argument
• {PCAtools} - functions for choosing the ideal number of components to retain
• {scater} - multi-feature set UMAP and various new arguments
• {scran} - new functions for sub-clustering and cluster bootstrapping, wrappers for graph-based clustering, updates to marker identification functions
• {SummarizedExperiment} - support for assays with more than four dimensions, changes to assay getters and setters to check dimnames
• {tximeta} - functions for splitting {SummarizedExperiment} objects and conversion to DGEList
• {tximport} - improved support for importing Alevin output

Workflow packages

• {fluentGenomics} - extended workflow using the {plyranges} and {tximeta} packages for fluent genomic analysis

blogdown file formats

I mentioned that blogdown works with R Markdown files but it actually handles three different file types which are treated in different ways (see here for more details):

1. .Rmd - R Markdown files that are rendered directly to .html by blogdown and friends (including Pandoc)
2. .rmarkdown - R Markdown files that are knitted to .markdown files by blogdown and then rendered to .html by Hugo.
3. .md - Standard Markdown files which are ignored by blogdown and rendered by Hugo.

The .Rmd workflow is usually recommended and because it makes use of Pandoc it enables several features including citations which are useful for an academic blog. However it also comes with the (potential) problem with re-running code mentioned above. What I would like to have is something like the .rmarkdown workflow but where the intermediate Markdown file is still rendered to .html using Pandoc instead of Hugo.

The blogdown build process

Before we try and modify it let’s have a look at how the standard blogdown build process works. To build the website we use the blogdown::build_site() function. This takes a local argument which sets whether the site is being viewed locally or not as well as a method argument (which we will get to later). This is (briefly) what happens when you call build_site():

1. Checks arguments and gets a list of files to build
2. Calls the blogdown:::build_rmds() function
   1. This function copies by-product files (such as plot output) from where they have been stored to the build directory
   2. Each file is passed to the blogdown:::render_page() function
      1. This function is a wrapper which calls the render_page.R script
      2. The script creates a new local environment (I assume there is a good reason to do this)
      3. The file is rendered in the new environment (with some post-processing if the output is Markdown)
   3. After rendering (if the output is Markdown) the YAML frontmatter is copied to the output file
3. By-products are moved back to their storage locations
4. Hugo is called to build the website

My modifications

I mentioned earlier that blogdown::build_site() has a method argument. This can take values of "html" which is the default process I have just described or "custom" which replaces this process by running a R/build.R script which can do whatever you like.² I have created a custom build script which is very similar to the blogdown functions with a few modifications. It is also inspired by this post but this method caches .md files which are rendered by Hugo rather than Pandoc. I’ll include some snippets below but the full script is here if you are interested.

rmd_files <- blogdown:::list_rmds("content", TRUE)
message("Found ", length(rmd_files), " R Markdown files")
md_files <- sub("\\.Rmd$", ".md.cached", rmd_files)

# Only knit Rmd files if...
to_render <- !file.exists(md_files) |             # md file does not exist OR
    utils::file_test("-ot", md_files, rmd_files)  # it is older than the Rmd

message("Rendering ", sum(!to_render), " cached Markdown files...")
for (md in md_files[!to_render]) {
    message("Rendering ", md, "...")
    render_md(md, base)
}

message("Rendering ", sum(to_render), " R Markdown files...")
for (rmd in rmd_files[to_render]) {
    message("Rendering ", rmd, "...")
    render_rmd(rmd, base)
}

blogdown:::prepend_yaml(md, out, x, callback = function(s) {
    if (!getOption("blogdown.draft.output", FALSE)) {
        return(s)
    }
    if (length(s) < 2 || length(grep("^draft: ", s)) > 0) {
        return(s)
    }
    append(s, "draft: yes", 1)
})

output_format <- rmarkdown::resolve_output_format(input)
output_format$keep_md <- TRUE

Wrapping up

These scripts are now being used to build this blog. They seem to work 🤞 but I haven’t tested them extensively and I expect there will be some issues if I try some posts with more complex code in them (for example I’m not sure what will happen if I try and include a HTML widget). I’m still not certain this is the best approach but I have learnt a lot about how blogdown work (although there is still a lot I don’t understand 😸).

What is a SingleCellExperiment?

Before we have a look at what packages use the SingleCellExperiment it’s probably useful to briefly discuss what it is. The SingleCellExperiment object is an extension of the older SummarizedExperiment object. This is an S4 class developed for use in Bioconductor packages with the main parts being a central set of matrix “assays” along with tables providing extra information about the rows and columns. There is also a metadata slot which is a list containing any other information related to the experiment.

One of the key benefits of using a structure like the SummarizedExperiment is that all the data related to an analysis is held in one spot. This makes it easier to pass things between functions or output results as well as reducing the possibilities for mismatches. The SingleCellExperiment adds some extra features that are useful for scRNA-seq analysis including:

• Slots for holding:
  • Dimenstionality reductions
  • Spike-in information
  • Size factors for normalisation
• Convenient access for named assays - counts, normcounts, cpm etc.

The idea is then that the SingleCellExperiment can be used by a range of packages to store data during analysis and extended when required. Let’s have a look at what those packages are currently by seeing what depends on SingleCellExperiment.

Getting package information

First let’s load the packages we need for this analysis. BiocPkgTools will need to be installed from GitHub but the rest are on CRAN.

library("BiocPkgTools") # https://github.com/seandavi/BiocPkgTools
library("tidygraph")
library("ggraph")
library("tidyverse")

We can use Sean Davis’ BiocPkgTools package to get information about Bioconductor packages, but then we need to do some filtering to get the information we want. It turns out I needed to do this multiple times so here is a function I wrote to make it a bit easier. It takes the database of information about all Bioconductor packages, the name of a package we are interested in and a flag indicating if we want normal or reverse dependencies.

get_bioc_deps <- function(bpi, pkg, reverse) {
    deps <- bpi %>%
        filter(Package == pkg)

    if (reverse) {
        deps <- deps %>%
            select(depends = dependsOnMe, imports = importsMe,
                   suggests = suggestsMe)
    } else {
        deps <- deps %>%
            select(depends = Depends, imports = Imports,
                   suggests = Suggests)
    }

    deps <- deps %>%
        gather(key = "type", value = "package") %>%
        separate_rows() %>%
        filter(!is.na(package))

    if (reverse) {
        deps <- deps %>%
            mutate(package2 = pkg) %>%
            rename(package1 = package)
    } else {
        deps <- deps %>%
            mutate(package1 = pkg) %>%
            rename(package2 = package)
    }

    deps <- deps %>% select(package1, uses = type, package2)
}

If we use it to search for reverse dependencies of SingleCellExperiment we can see it returns a data frame with which Bioconductor packages use SingleCellExperiment and the relationship between them.

bpi <- getBiocPkgList()
bioc_revdeps <- get_bioc_deps(bpi, "SingleCellExperiment", reverse = TRUE)
head(bioc_revdeps)

We can do a similar thing for CRAN packages with use of the tools::package_dependencies function.

get_cran_deps <- function(pkg, db, reverse) {

    types <- c("Depends", "Imports", "Suggests")

    deps <- sapply(types, function(type) {
        deps <- tools::package_dependencies(pkg, db, which = type,
                                            reverse = reverse)
        c(type = type, package = paste(deps[[1]], collapse = ", "))
    })

    deps <- deps %>%
        t() %>%
        as_data_frame() %>%
        mutate(type = tolower(type)) %>%
        filter(package != "") %>%
        separate_rows(package)

    if (nrow(deps) == 0) {
        return(tibble(package1 = character(), uses = character(),
                      package2 = character()))
    }

    if (reverse) {
        deps <- deps %>%
            mutate(package2 = pkg) %>%
            rename(package1 = package)
    } else {
        deps <- deps %>%
            mutate(package1 = pkg) %>%
            rename(package2 = package)
    }

    deps <- deps %>% select(package1, uses = type, package2)
}
db <- available.packages(repos = "http://cran.r-project.org")
cran_revdeps <- get_cran_deps("SingleCellExperiment", db, reverse = TRUE)
head(cran_revdeps)

What uses SingleCellExperiment?

We now have two tables showing us which Bioconductor and CRAN packages make use of SingleCellExperiment. Tables can be fairly boring to look at though so let’s use the relationships to construct a graph using tidygraph.

nodes <- bioc_revdeps %>%
    bind_rows(cran_revdeps) %>%
    select(-uses) %>%
    gather(key = id, value = package) %>%
    select(-id) %>%
    distinct() %>%
    mutate(repo = if_else(package %in% bpi$Package, "Bioconductor", "CRAN"))
edges <- bioc_revdeps %>%
    bind_rows(cran_revdeps) %>%
    rename(from = package1, to = package2)
graph <- tbl_graph(nodes = nodes, edges = edges)

We can now visualise the relationships using ggraph:

ggraph(graph, layout = "fr") +
    geom_edge_fan(aes(colour = uses),
                  arrow = arrow(length = unit(4, 'mm')),
                  end_cap = circle(3, 'mm')) +
    geom_node_point(aes(colour = repo)) +
    geom_node_text(aes(label = package, colour = repo), repel = TRUE) +
    scale_color_brewer(palette = "Set1") +
    scale_edge_color_brewer(palette = "Dark2") +
    theme_graph()

This doesn’t tell us a lot we didn’t already know but it does allow us to see everything in one place. We can see that there are only a couple of CRAN packages, which is unsurprising given that SingleCellExperiment is part of Bioconductor, and that most packages either “import” or “depend” on SingleCellExperiment.

What about the relationships between the packages that depend on SingleCellExperiment? Are there communitites of related scRNA-seq analysis tools?

Adding an extra hop

We can reuse the functions we wrote earlier to get the dependencies (and reverse dependencies) of our list of scRNA-seq packages. We will also do a little bit of extra processing to tidy up some of the results.

more_deps <- map2(nodes$package, nodes$repo, function(x, y) {
    if (y == "Bioconductor") {
        get_bioc_deps(bpi, x, FALSE)
    } else {
        get_cran_deps(x, db, FALSE)
    }
}) %>%
    bind_rows() %>%
    mutate(package2 = str_remove(package2, " ?\\(\\D+[0-9\\.]+\\)")) %>%
    filter(package2 != "R")
more_revdeps <- map2(nodes$package, nodes$repo, function(x, y) {
    if (y == "Bioconductor") {
        get_bioc_deps(bpi, x, TRUE)
    } else {
        get_cran_deps(x, db, TRUE)
    }
}) %>% bind_rows()

Let’s build another graph and plot what we get. As you can see it’s a bit crowded so I have left off the package labels.

nodes <- more_deps %>%
    bind_rows(more_revdeps) %>%
    select(-uses) %>%
    gather(key = id, value = package) %>%
    select(-id) %>%
    distinct() %>%
    mutate(repo = if_else(package %in% bpi$Package, "Bioconductor", "CRAN"))
edges <- more_deps %>%
    bind_rows(more_revdeps) %>%
    rename(from = package1, to = package2) %>%
    distinct()
graph <- tbl_graph(nodes = nodes, edges = edges)
ggraph(graph, layout = "fr") +
    geom_edge_fan(aes(colour = uses),
                  arrow = arrow(length = unit(4, 'mm')),
                  end_cap = circle(3, 'mm')) +
    geom_node_point(aes(colour = repo)) +
    #geom_node_text(aes(label = package, colour = repo), repel = TRUE) +
    scale_color_brewer(palette = "Set1") +
    scale_edge_color_brewer(palette = "Dark2") +
    theme_graph()

Our graph has a lot more information now, but is probably too complext to tell us anything useful. In particularly we can see there are a lot of nodes around the edges that have one package depending on them. Let’s get rid of those by removing sink node.

graph <- tbl_graph(nodes = nodes, edges = edges) %>%
    activate(nodes) %>%
    filter(!node_is_sink())
ggraph(graph, layout = "fr") +
    geom_edge_fan(aes(colour = uses),
                   arrow = arrow(length = unit(4, 'mm')),
                   end_cap = circle(3, 'mm')) +
    geom_node_point(aes(colour = repo)) +
    geom_node_text(aes(label = package, colour = repo), size = 2,
                   repel = TRUE) +
    scale_color_brewer(palette = "Set1") +
    scale_edge_color_brewer(palette = "Dark2") +
    theme_graph()

That’s much better! We can now see some of the structure between our packages. There are quite few packages that rely on SingleCellExperiment but nothing else. Removing source nodes as well will tidy this up a bit more.

graph <- tbl_graph(nodes = nodes, edges = edges) %>%
    activate(nodes) %>%
    filter(!node_is_sink()) %>%
    filter(!node_is_source())
ggraph(graph, layout = "fr") +
    geom_edge_fan(aes(colour = uses),
                   arrow = arrow(length = unit(4, 'mm')),
                   end_cap = circle(3, 'mm')) +
    geom_node_point(aes(colour = repo)) +
    geom_node_text(aes(label = package, colour = repo),
                   repel = TRUE) +
    scale_color_brewer(palette = "Set1") +
    scale_edge_color_brewer(palette = "Dark2") +
    theme_graph()

Now we can see the core SingleCellExperiment package network. Apart from SingleCellExperiment itself there are three main packages: scater, scran and splatter. Scater and scran are two low-level scRNA-seq analysis package with scater providing functions for tasks such as visualisation and filtering and scran focusing more on normalisation and removal of batch effects. It is unsurprising that these packages show up as Aaron Lun is heavily involved in the development of both of them as well as SingleCellExperiment. Splatter is a bit of a different case as it suggests many packages that provide the core functions for it’s simulations but isn’t used by any other analysis packages. There are a few other influential packages on the periphery of this network, particularly monocle and Seurat.

What do these packages do?

We have had a look at how packages that use SingleCellExperiment are related but what do they actually do? Bioconductor categorises packages using “biocViews”, tags that describe software in various ways. Let’s summarise those for the packages that use SingleCellExperiment.

plot_data <- bpi %>%
    filter(Package %in% bioc_revdeps$package1) %>%
    select(Package, biocViews) %>%
    separate_rows(biocViews) %>%
    group_by(biocViews) %>%
    summarise(count = n()) %>%
    arrange(-count) %>%
    mutate(prop = count / n()) %>%
    mutate(biocViews = factor(biocViews, level = biocViews))
ggplot(plot_data, aes(x = biocViews, y = prop)) +
    geom_col() +
    scale_y_continuous(labels = scales::percent) +
    ggtitle("biocViews for packages that use SingleCellExperiment") +
    theme_minimal() +
    theme(axis.text.x = element_text(angle = 90, hjust = 1, vjust = 0.5),
          axis.title = element_blank(),
          panel.grid = element_blank())

Unsurprisingly the top few categories (“SingleCell”, “GeneExpression”, “RNASeq”, “Transcriptomics”, “Sequencing”) are related to scRNA-seq data in general, along with the “Software” tag. I actually find it a bit surprising that these aren’t more common with only 66 percent of packages labelled as “SingleCell”. After these we have some of the most common scRNA-seq analysis tasks, “Clustering”, “Visualization”, “DifferentialExpression” and “DimensionReduction”. This is similar to what we see in the scRNA-tools database, but obviously from a much smaller sample.

We can’t do the same thing for the CRAN packages, but as there are only a couple of these we can just describe them. Seurat is perhaps the most complete R scRNA-seq analysis covering most steps in a standard workflow. It’s connection to SingleCellExperiment is through functions that have recently been added to convert to/from it’s own object. Clustree is a package for visualising clustering results in general and suggests SingleCellExperiment to provide a convenience function for people working with scRNA-seq data.

Where to from here?

It’s only been about six months since SingleCellExperiment joined Bioconductor release but we are already seeing a community of packages growing up around it. Hopefully we see this continue and there is a new batch of packages using it in the next Bioconductor release. For anyone who is working on a scRNA-seq package I strongly encourage you to consider basing it around SingleCellExperiment. It can take some time to get your head around how it works but the infrastructure it provides will save you a lot of time in the long run. It also makes things a lot easier for your uses who won’t have to learn a new data structure to use your package and can make use of a range of packages without having to convert between objects. If you don’t want to be locked into the Bioconductor ecosystem think about using the Seurat object instead or if you work in Python consider the anndata object. There is also the loom format which has both R and Python interfaces. Whatever standard works for you everyone will be better off if the community can make use of a small number of data structure rather than each package using their own.

Single-cell RNA-seq

My interest is in single-cell RNA-seq analysis, so I am going to start off with packages related to this.

Other areas

Round 5

Introduction

An analysis of tweets from the Joining the Dots symposium. 1237 tweets were collected using the rtweet R package:

jtdwehi <- search_tweets("#jtdwehi", 10000)
saveRDS(jtdwehi, "data/jtdwehi.Rds")

Timeline

Users

Networks

Retweets

Favourites

Quotes

Media

Tweet text

The 100 words used 3 or more times.

Setup

First we need to load the libraries we are going to use:

# Simulation
library(splatter)

# Clustering
library(Seurat) # Installed from https://github.com/satijalab/seurat

# Graphs
library(igraph)

# Plotting
library(ggraph)
library(viridis)

# Data manipulation
library(tidyverse)

For this example I am going to simulate some scRNA-seq data with eight different groups with different numbers of cells using Splatter.

sim <- splatSimulateGroups(groupCells = c(100, 80, 60, 50, 30, 20, 20, 15),
                            seed = 10, verbose = FALSE)

Let’s take a quick look at this to see if it is anything like we would expect:

plotTSNE(sim, colour_by = "Group")

Here we can see the different groups, so there should be something for our clustering analysis to find.

The clustering pacakge we are going to use is Seurat which uses it’s own object to store the data. Here is a small function I have written to convert from the SCESet object produced by Splatter to the Seurat object required by Seurat.

SCESetToSeurat <- function(sce) {
    if (!is(sce,'SCESet')) {
        stop("sce must be an SCESet object")
    }

    counts <- scater::counts(sce)

    if (is.null(counts)) {
        stop("sce must contain counts to convert to Seurat")
    }

    seurat <- new("seurat",
                    raw.data = counts,
                    is.expr = sce@lowerDetectionLimit,
                    data.info = Biobase::pData(sce),
                    cell.names = Biobase::sampleNames(sce))

    return(seurat)
}

Now to convert the dataset.

seurat <- SCESetToSeurat(sim)

Clustering

We now have a dataset in the format required by Seurat. Before we do any clustering we need to run through some setup steps. I’m not going to explain what they are doing here, if you want to now the details refer to the Seurat tutorials.

seurat <- Setup(seurat, project = "Example", meta.data = seurat@data.info)
seurat <- MeanVarPlot(seurat, fxn.x = expMean, fxn.y = logVarDivMean,
                        x.low.cutoff = 0.1, x.high.cutoff = 3,
                        y.cutoff = 0.5, do.contour = FALSE)

seurat <- PCA(seurat, pc.genes = seurat@var.genes, do.print = FALSE)

Now we can do the clustering. The parameter we are interested in is the resolution parameter which controls how many clusters Seurat returns. I start by setting resolution = 0. This will create a cluster containing all cells that will serve as the root of our tree. We also ask Seurat to store some of the intermediate calculations so we don’t have to do them again when we cluster with different resolutions:

seurat <- FindClusters(seurat, pc.use = 1:20, resolution = 0, algorithm = 3,
                        print.output = FALSE, save.SNN = TRUE)

We can now loop over a range of resolutions that we are interested in. I have only tried a few values here but if this was a real dataset you might want to try some more.

for (res in c(0.3, 0.6, 0.9, 1.2)) {
    seurat <- FindClusters(seurat, resolution = res, algorithm = 3,
                            print.output = FALSE)
}

Get results

Seurat stores the cluster labels in the data.info slot in columns starting with res.. This is the only part we are interested in so let’s extract just those columns.

clusterings <- seurat@data.info %>% select(contains("res."))

head(clusterings)

##       res.0 res.0.3 res.0.6 res.0.9 res.1.2
## Cell1     0       1       0       0       0
## Cell2     0       1       0       0       0
## Cell3     0       1       0       0       0
## Cell4     0       1       0       0       0
## Cell5     0       1       0       0       0
## Cell6     0       1       0       0       0

We now know which cluster each cell was assigned to at each resolution but to build the tree we need some more information. This next function looks at two neighbouring resolutions and works out how many cells moved from a cluster in the lower resolution to each cluster in the higher resolution. These transitions are going to form the edges of our tree.

getEdges <- function(clusterings) {

    # Loop over the different resolutions
    transitions <- lapply(1:(ncol(clusterings) - 1), function(i) {

        # Extract two neighbouring clusterings
        from.res <- sort(colnames(clusterings))[i]
        to.res <- sort(colnames(clusterings))[i + 1]

        # Get the cluster names
        from.clusters <- sort(unique(clusterings[, from.res]))
        to.clusters <- sort(unique(clusterings[, to.res]))

        # Get all possible combinations
        trans.df <- expand.grid(FromClust = from.clusters,
                                ToClust = to.clusters)

        # Loop over the possible transitions
        trans <- apply(trans.df, 1, function(x) {
            from.clust <- x[1]
            to.clust <- x[2]

            # Find the cells from those clusters
            is.from <- clusterings[, from.res] == from.clust
            is.to <- clusterings[, to.res] == to.clust

            # Count them up
            trans.count <- sum(is.from & is.to)

            # Get the sizes of the two clusters
            from.size <- sum(is.from)
            to.size <- sum(is.to)

            # Get the proportions of cells moving along this edge
            trans.prop.from <- trans.count / from.size
            trans.prop.to <- trans.count / to.size

            return(c(trans.count, trans.prop.from, trans.prop.to))
        })

        # Tidy up the results
        trans.df$FromRes <- as.numeric(gsub("res.", "", from.res))
        trans.df$ToRes <- as.numeric(gsub("res.", "", to.res))
        trans.df$TransCount <- trans[1, ]
        trans.df$TransPropFrom <- trans[2, ]
        trans.df$TransPropTo <- trans[3, ]

        return(trans.df)
    })

    # Bind the results from the different resolutions together
    transitions <- do.call("rbind", transitions)

    # Tidy everything up
    levs <- sort(as.numeric(levels(transitions$ToClust)))
    transitions <- transitions %>%
        mutate(FromClust = factor(FromClust,
                                    levels = levs))  %>%
        mutate(ToClust = factor(ToClust, levels = levs))

    return(transitions)
}

edges <- getEdges(clusterings)
head(edges)

##   FromClust ToClust FromRes ToRes TransCount TransPropFrom TransPropTo
## 1         0       0     0.0   0.3        135     0.3600000           1
## 2         0       1     0.0   0.3        100     0.2666667           1
## 3         0       2     0.0   0.3         60     0.1600000           1
## 4         0       3     0.0   0.3         50     0.1333333           1
## 5         0       4     0.0   0.3         30     0.0800000           1
## 6         0       0     0.3   0.6          0     0.0000000           0

Some of these columns are pretty obvious but the last three could do with an explanation. TransCount is the number of cells that move along this edge. TransPropFrom is the proportion of the cells in the lower resolution cluster that have made this transition and TransPropTo is the proportion of cells in the higher resolution cluster that came from this edge.

Getting the information about the nodes of the tree is easier as these just represent the clusters. This function summarises the cluster information and converts it to long format.

getNodes <- function(clusterings) {
    nodes <- clusterings %>%
        gather(key = Res, value = Cluster) %>%
        group_by(Res, Cluster) %>%
        summarise(Size = n()) %>%
        ungroup() %>%
        mutate(Res = stringr::str_replace(Res, "res.", "")) %>%
        mutate(Res = as.numeric(Res), Cluster = as.numeric(Cluster)) %>%
        mutate(Node = paste0("R", Res, "C", Cluster)) %>%
        select(Node, everything())
}

nodes <- getNodes(clusterings)
head(nodes)

## # A tibble: 6 x 4
##     Node   Res Cluster  Size
##    <chr> <dbl>   <dbl> <int>
## 1   R0C0   0.0       0   375
## 2 R0.3C0   0.3       0   135
## 3 R0.3C1   0.3       1   100
## 4 R0.3C2   0.3       2    60
## 5 R0.3C3   0.3       3    50
## 6 R0.3C4   0.3       4    30

Each node needs a unique ID which I have made by combining the resolution and cluster number. We also record the number of cells in each cluster.

Now we can build the graph we will use as the starting point for our plot. Some of the possible edges between clusters will have no cells travelling along them so we filter them out. We also remove edges that correspond to a small proportion (< 2%) of cells in the higher resolution cluster.

graph <- edges %>%
    # Remove edges without any cell...
    filter(TransCount > 0) %>%
    # ...or making up only a small proportion of the new cluster
    filter(TransPropTo > 0.02) %>%
    # Rename the nodes
    mutate(FromNode = paste0("R", FromRes, "C", FromClust)) %>%
    mutate(ToNode = paste0("R", ToRes, "C", ToClust)) %>%
    # Reorder columns
    select(FromNode, ToNode, everything()) %>%
    # Build a graph using igraph
    graph_from_data_frame(vertices = nodes)

print(graph)

## IGRAPH b1b93c3 DN-- 23 23 --
## + attr: name (v/c), Res (v/n), Cluster (v/n), Size (v/n),
## | FromClust (e/c), ToClust (e/c), FromRes (e/n), ToRes (e/n),
## | TransCount (e/n), TransPropFrom (e/n), TransPropTo (e/n)
## + edges from b1b93c3 (vertex names):
##  [1] R0C0  ->R0.3C0 R0C0  ->R0.3C1 R0C0  ->R0.3C2 R0C0  ->R0.3C3
##  [5] R0C0  ->R0.3C4 R0.3C1->R0.6C0 R0.3C0->R0.6C1 R0.3C4->R0.6C1
##  [9] R0.3C0->R0.6C2 R0.3C2->R0.6C3 R0.3C3->R0.6C4 R0.6C0->R0.9C0
## [13] R0.6C2->R0.9C1 R0.6C3->R0.9C2 R0.6C1->R0.9C3 R0.6C4->R0.9C4
## [17] R0.6C1->R0.9C5 R0.9C0->R1.2C0 R0.9C1->R1.2C1 R0.9C2->R1.2C2
## [21] R0.9C3->R1.2C3 R0.9C4->R1.2C4 R0.9C5->R1.2C5

Plot the tree

The last step is to pass our graph to the ggraph library for plotting.

# Plot our graph using the `tree` layout
ggraph(graph, layout = "tree") +
    # Plot the edges, colour is the number of cells and transparency is the
    # proportion contribution to the new cluster
    geom_edge_link(arrow = arrow(length = unit(1, 'mm')),
                    end_cap = circle(3.5, "mm"), edge_width = 1,
                    aes(colour = log(TransCount), alpha = TransPropTo)) +
    # Plot the nodes, size is the number of cells
    geom_node_point(aes(colour = factor(Res),
                        size = Size)) +
    geom_node_text(aes(label = Cluster), size = 3) +
    # Adjust the scales
    scale_size(range = c(4, 15)) +
    scale_edge_colour_gradientn(colours = viridis(100)) +
    # Add legend labels
    guides(size = guide_legend(title = "Cluster Size", title.position = "top"),
            colour = guide_legend(title = "Clustering Resolution",
                                    title.position = "top"),
            edge_colour = guide_edge_colorbar(title = "Cell Count (log)",
                                                title.position = "top"),
            edge_alpha = guide_legend(title = "Cluster Prop",
                                        title.position = "top", nrow = 2)) +
    # Remove the axes as they don't really mean anything
    theme_void() +
    theme(legend.position = "bottom")

And here is the result! We can see that see that Seurat finds three of the clusters easily and that these don’t change as the resolution increases. A fourth group contains most of the cells and is sub-divided as we increase resolution. Interestingly at the lowest resolution there is a small cluster which is then absorbed into one of the other branches.

This tree is cleaner and has less branches than what we would be likely to see with a real dataset but the process to create it would be the same. I have used Seurat as the clustering method in this example but it should be easy to adapt the process to any other method that allows you to adjust the number of clusters. I have found this visualisation useful in my analysis particularly for looking at which clusters are very distinct and the relationships between different clusters and clusterings.

Good luck creating your own clustering trees!

devtools::session_info()

## Session info -------------------------------------------------------------

##  setting  value
##  version  R version 3.4.1 (2017-06-30)
##  system   x86_64, darwin15.6.0
##  ui       RStudio (1.0.143)
##  language (EN)
##  collate  en_AU.UTF-8
##  tz       Australia/Melbourne
##  date     2017-07-19

## Packages -----------------------------------------------------------------

##  package        * version  date       source
##  AnnotationDbi    1.38.1   2017-06-01 Bioconductor
##  ape              4.1      2017-02-14 cran (@4.1)
##  assertthat       0.2.0    2017-04-11 CRAN (R 3.4.0)
##  backports        1.1.0    2017-05-22 CRAN (R 3.4.0)
##  base           * 3.4.1    2017-07-07 local
##  beeswarm         0.2.3    2016-04-25 CRAN (R 3.4.0)
##  bindr            0.1      2016-11-13 CRAN (R 3.4.0)
##  bindrcpp       * 0.2      2017-06-17 CRAN (R 3.4.0)
##  Biobase        * 2.36.2   2017-05-04 Bioconductor
##  BiocGenerics   * 0.22.0   2017-04-25 Bioconductor
##  BiocParallel     1.10.1   2017-05-03 Bioconductor
##  biomaRt          2.32.1   2017-06-09 Bioconductor
##  bit              1.1-12   2014-04-09 CRAN (R 3.4.0)
##  bit64            0.9-7    2017-05-08 CRAN (R 3.4.0)
##  bitops           1.0-6    2013-08-17 CRAN (R 3.4.0)
##  blob             1.1.0    2017-06-17 CRAN (R 3.4.0)
##  broom            0.4.2    2017-02-13 CRAN (R 3.4.0)
##  car              2.1-5    2017-07-04 cran (@2.1-5)
##  caret            6.0-76   2017-04-18 cran (@6.0-76)
##  caTools          1.17.1   2014-09-10 cran (@1.17.1)
##  cellranger       1.1.0    2016-07-27 CRAN (R 3.4.0)
##  checkmate        1.8.3    2017-07-03 CRAN (R 3.4.1)
##  class            7.3-14   2015-08-30 CRAN (R 3.4.1)
##  cluster          2.0.6    2017-03-10 CRAN (R 3.4.1)
##  codetools        0.2-15   2016-10-05 CRAN (R 3.4.1)
##  colorspace       1.3-2    2016-12-14 CRAN (R 3.4.0)
##  compiler         3.4.1    2017-07-07 local
##  cowplot        * 0.7.0    2016-10-28 cran (@0.7.0)
##  data.table       1.10.4   2017-02-01 CRAN (R 3.4.0)
##  datasets       * 3.4.1    2017-07-07 local
##  DBI              0.7      2017-06-18 CRAN (R 3.4.0)
##  DEoptimR         1.0-8    2016-11-19 cran (@1.0-8)
##  devtools         1.13.2   2017-06-02 CRAN (R 3.4.0)
##  digest           0.6.12   2017-01-27 CRAN (R 3.4.0)
##  diptest          0.75-7   2016-12-05 cran (@0.75-7)
##  dplyr          * 0.7.1    2017-06-22 CRAN (R 3.4.1)
##  edgeR            3.18.1   2017-05-06 Bioconductor
##  evaluate         0.10.1   2017-06-24 CRAN (R 3.4.1)
##  fastICA          1.2-1    2017-06-12 cran (@1.2-1)
##  flexmix          2.3-14   2017-04-28 cran (@2.3-14)
##  FNN              1.1      2013-07-31 cran (@1.1)
##  forcats          0.2.0    2017-01-23 CRAN (R 3.4.0)
##  foreach          1.4.3    2015-10-13 cran (@1.4.3)
##  foreign          0.8-69   2017-06-22 CRAN (R 3.4.1)
##  fpc              2.1-10   2015-08-14 cran (@2.1-10)
##  gdata            2.18.0   2017-06-06 cran (@2.18.0)
##  ggbeeswarm       0.5.3    2016-12-01 CRAN (R 3.4.0)
##  ggforce          0.1.1    2016-11-28 CRAN (R 3.4.0)
##  ggplot2        * 2.2.1    2016-12-30 CRAN (R 3.4.0)
##  ggraph         * 1.0.0    2017-02-24 CRAN (R 3.4.0)
##  ggrepel          0.6.5    2016-11-24 CRAN (R 3.4.0)
##  glue             1.1.1    2017-06-21 CRAN (R 3.4.1)
##  gplots           3.0.1    2016-03-30 cran (@3.0.1)
##  graphics       * 3.4.1    2017-07-07 local
##  grDevices      * 3.4.1    2017-07-07 local
##  grid             3.4.1    2017-07-07 local
##  gridExtra        2.2.1    2016-02-29 CRAN (R 3.4.0)
##  gtable           0.2.0    2016-02-26 CRAN (R 3.4.0)
##  gtools           3.5.0    2015-05-29 cran (@3.5.0)
##  haven            1.1.0    2017-07-09 CRAN (R 3.4.1)
##  hms              0.3      2016-11-22 CRAN (R 3.4.0)
##  htmltools        0.3.6    2017-04-28 CRAN (R 3.4.0)
##  httpuv           1.3.5    2017-07-04 CRAN (R 3.4.1)
##  httr             1.2.1    2016-07-03 CRAN (R 3.4.0)
##  igraph         * 1.1.1    2017-07-16 CRAN (R 3.4.1)
##  IRanges          2.10.2   2017-05-25 Bioconductor
##  irlba            2.2.1    2017-05-17 cran (@2.2.1)
##  iterators        1.0.8    2015-10-13 cran (@1.0.8)
##  jsonlite         1.5      2017-06-01 CRAN (R 3.4.0)
##  kernlab          0.9-25   2016-10-03 cran (@0.9-25)
##  KernSmooth       2.23-15  2015-06-29 CRAN (R 3.4.1)
##  knitr            1.16     2017-05-18 CRAN (R 3.4.1)
##  labeling         0.3      2014-08-23 CRAN (R 3.4.0)
##  lars             1.2      2013-04-24 cran (@1.2)
##  lattice          0.20-35  2017-03-25 CRAN (R 3.4.1)
##  lazyeval         0.2.0    2016-06-12 CRAN (R 3.4.0)
##  limma            3.32.3   2017-07-16 Bioconductor
##  lme4             1.1-13   2017-04-19 cran (@1.1-13)
##  locfit           1.5-9.1  2013-04-20 CRAN (R 3.4.0)
##  lubridate        1.6.0    2016-09-13 CRAN (R 3.4.0)
##  magrittr         1.5      2014-11-22 CRAN (R 3.4.0)
##  MASS             7.3-47   2017-02-26 CRAN (R 3.4.1)
##  Matrix           1.2-10   2017-05-03 CRAN (R 3.4.1)
##  MatrixModels     0.4-1    2015-08-22 cran (@0.4-1)
##  matrixStats      0.52.2   2017-04-14 CRAN (R 3.4.0)
##  mclust           5.3      2017-05-21 cran (@5.3)
##  memoise          1.1.0    2017-04-21 CRAN (R 3.4.0)
##  methods        * 3.4.1    2017-07-07 local
##  mgcv             1.8-17   2017-02-08 CRAN (R 3.4.1)
##  mime             0.5      2016-07-07 CRAN (R 3.4.0)
##  minqa            1.2.4    2014-10-09 cran (@1.2.4)
##  mixtools         1.1.0    2017-03-10 cran (@1.1.0)
##  mnormt           1.5-5    2016-10-15 CRAN (R 3.4.0)
##  ModelMetrics     1.1.0    2016-08-26 cran (@1.1.0)
##  modelr           0.1.0    2016-08-31 CRAN (R 3.4.0)
##  modeltools       0.2-21   2013-09-02 cran (@0.2-21)
##  munsell          0.4.3    2016-02-13 CRAN (R 3.4.0)
##  mvtnorm          1.0-6    2017-03-02 cran (@1.0-6)
##  nlme             3.1-131  2017-02-06 CRAN (R 3.4.1)
##  nloptr           1.0.4    2014-08-04 cran (@1.0.4)
##  nnet             7.3-12   2016-02-02 CRAN (R 3.4.1)
##  numDeriv         2016.8-1 2016-08-27 cran (@2016.8-)
##  parallel       * 3.4.1    2017-07-07 local
##  pbapply          1.3-3    2017-07-04 cran (@1.3-3)
##  pbkrtest         0.4-7    2017-03-15 cran (@0.4-7)
##  pkgconfig        2.0.1    2017-03-21 CRAN (R 3.4.0)
##  plyr             1.8.4    2016-06-08 CRAN (R 3.4.0)
##  prabclus         2.2-6    2015-01-14 cran (@2.2-6)
##  psych            1.7.5    2017-05-03 CRAN (R 3.4.1)
##  purrr          * 0.2.2.2  2017-05-11 CRAN (R 3.4.0)
##  quantreg         5.33     2017-04-18 cran (@5.33)
##  R6               2.2.2    2017-06-17 CRAN (R 3.4.0)
##  ranger           0.8.0    2017-06-20 cran (@0.8.0)
##  RColorBrewer     1.1-2    2014-12-07 CRAN (R 3.4.0)
##  Rcpp             0.12.12  2017-07-15 CRAN (R 3.4.1)
##  RCurl            1.95-4.8 2016-03-01 CRAN (R 3.4.0)
##  readr          * 1.1.1    2017-05-16 CRAN (R 3.4.0)
##  readxl           1.0.0    2017-04-18 CRAN (R 3.4.0)
##  reshape2         1.4.2    2016-10-22 CRAN (R 3.4.0)
##  rhdf5            2.20.0   2017-04-25 Bioconductor
##  rjson            0.2.15   2014-11-03 CRAN (R 3.4.0)
##  rlang            0.1.1    2017-05-18 CRAN (R 3.4.0)
##  rmarkdown        1.6      2017-06-15 CRAN (R 3.4.1)
##  robustbase       0.92-7   2016-12-09 cran (@0.92-7)
##  ROCR             1.0-7    2015-03-26 cran (@1.0-7)
##  rprojroot        1.2      2017-01-16 CRAN (R 3.4.0)
##  RSQLite          2.0      2017-06-19 CRAN (R 3.4.1)
##  Rtsne            0.13     2017-04-14 cran (@0.13)
##  rvest            0.3.2    2016-06-17 CRAN (R 3.4.0)
##  S4Vectors        0.14.3   2017-06-03 Bioconductor
##  scales           0.4.1    2016-11-09 CRAN (R 3.4.0)
##  scater         * 1.4.0    2017-04-25 Bioconductor
##  segmented        0.5-2.1  2017-06-14 cran (@0.5-2.1)
##  Seurat         * 1.4.0.16 2017-07-19 Github (satijalab/seurat@3bd092a)
##  shiny            1.0.3    2017-04-26 CRAN (R 3.4.0)
##  shinydashboard   0.6.1    2017-06-14 CRAN (R 3.4.0)
##  sn               1.5-0    2017-02-10 cran (@1.5-0)
##  SparseM          1.77     2017-04-23 cran (@1.77)
##  splatter       * 1.0.3    2017-05-27 Bioconductor
##  splines          3.4.1    2017-07-07 local
##  stats          * 3.4.1    2017-07-07 local
##  stats4           3.4.1    2017-07-07 local
##  stringi          1.1.5    2017-04-07 CRAN (R 3.4.0)
##  stringr          1.2.0    2017-02-18 CRAN (R 3.4.0)
##  survival         2.41-3   2017-04-04 CRAN (R 3.4.1)
##  tclust           1.2-7    2017-06-30 cran (@1.2-7)
##  tibble         * 1.3.3    2017-05-28 CRAN (R 3.4.0)
##  tidyr          * 0.6.3    2017-05-15 CRAN (R 3.4.0)
##  tidyverse      * 1.1.1    2017-01-27 CRAN (R 3.4.0)
##  tools            3.4.1    2017-07-07 local
##  trimcluster      0.1-2    2012-10-29 cran (@0.1-2)
##  tsne             0.1-3    2016-07-15 cran (@0.1-3)
##  tweenr           0.1.5    2016-10-10 CRAN (R 3.4.0)
##  tximport         1.4.0    2017-04-25 Bioconductor
##  udunits2         0.13     2016-11-17 CRAN (R 3.4.0)
##  units            0.4-5    2017-06-15 CRAN (R 3.4.0)
##  utils          * 3.4.1    2017-07-07 local
##  VGAM             1.0-3    2017-01-11 cran (@1.0-3)
##  vipor            0.4.5    2017-03-22 CRAN (R 3.4.0)
##  viridis        * 0.4.0    2017-03-27 CRAN (R 3.4.0)
##  viridisLite    * 0.2.0    2017-03-24 CRAN (R 3.4.0)
##  withr            1.0.2    2016-06-20 CRAN (R 3.4.0)
##  XML              3.98-1.9 2017-06-19 CRAN (R 3.4.1)
##  xml2             1.1.1    2017-01-24 CRAN (R 3.4.0)
##  xtable           1.8-2    2016-02-05 CRAN (R 3.4.0)
##  yaml             2.1.14   2016-11-12 CRAN (R 3.4.0)
##  zlibbioc         1.22.0   2017-04-25 Bioconductor

Single-cell

These packages are specific to single-cell RNA-seq analysis. A couple of them I am already familiar with, particularly scater.

• cellity - identifying low-quality cells
• celTree - model the relationship between individual cells over time or space.
• scater - tools for analysis of single-cell RNA-seq data (particularly QC)
• scde - single-cell differential expression
• scran - normalisation, cell-cycle assignment, gene detection