Correlation Primer with Aster and R

Calculating correlations is often starting point before more advanced analytical steps take place. Big data (long data) always presents computational challenges of both scale and distributed nature. In turn they may get aggravated by the presence of large number of features (wide data). But challenges do not stop here as complex relationships induce analysis of correlations across subsets and groups.

Such mix of long and wide becomes more common in the age of internet-of-things, sensor and machine data with non-human data sources dominating analytical use cases.  
Thus, when computing correlations on big data the following capabilities matter:

• scale on large distributed data sets (long data)
• scale on wide distributed data sets (wide data / large number of features)
• flexibility on wide data sets (ability to permutate features such as Cartesian combinations, one-to-many, etc.)
• correlations on subsets and groups.

Correlations in R comes standard with stats function cor but it doesn't meet most of the capabilities above. As always Teradata Aster big data analytical platform offers both scalability and functionality far exceeding capabilities above. And thanks to Aster R (TeradataAsterR) package it is available without leaving R environment.

With Aster and R integration there are multiple ways of correlating on datasets. Before sending you to the link for detailed discussion I summarized approaches discussed there by the capabilities:

Method / Solution features	Variable (columns) Permutations	Calculating for Groups	SQL-MR	In-database R
Aster R ta.cor	N	N	Y	N
Aster R in-database ta.tapply	N	Y	N	Y
toaster computeCorrelations	Y	Y	Y	N

Please visit my latest RPubs post for detailed discussion and comparison of these methods.

R Graph Objects: igraph vs. network

While working on new graph functions for my package toaster I had to pick from the R packages that represent graphs. The choice was between network and graph objects from the network and igraph correspondingly - the two most prominent packages for creating and manipulating graphs and networks in R.

Interchangeability of network and graph objects

One can always use them interchangeably with little effort using package intergraph. Its sole purpose is providing "coercion routines for network data objects". Simply use its asNetwork and asIgraph functions to convert from one network representation to another:

library(igraph)
library(network)
library(intergraph)
 
# igraph 
pkg.igraph = graph_from_edgelist(edges.mat, directed = TRUE)
pkg.network.from.igraph = asNetwork(pkg.igraph)
all.equal(length(get.edgelist(pkg.igraph)), length(as.matrix(pkg.network.from.igraph, "edgelist")))
 
# network
pkg.network = network(edges.mat)
pkg.igraph.from.network = asIgraph(pkg.network)
all.equal(length(as.matrix(pkg.network, "edgelist")), length(get.edgelist(pkg.igraph.from.network)))

Created by Pretty R at inside-R.org

For more on using intergraph functions see tutorial.

Package dependencies with miniCRAN

To assess relative importance of packages network and igraph we will use package miniCRAN. Its access to CRAN packages' metadata including dependencies via "Depends", "Imports", "Suggests" provides necessary information about package relationships. Built-in makeDepGraph function recursively retrieves these dependencies and builds corresponding graph:

library(miniCRAN)
 
cranInfo = pkgAvail()
 
plot(makeDepGraph(c("network"), availPkgs = cranInfo))
plot(makeDepGraph(c("igraph"), availPkgs = cranInfo))

Created by Pretty R at inside-R.org


Unfortunately, these dependency graphs show how network and igraph depend on other CRAN packages while the goal is to evaluate relationships the other way around: how much other CRAN packages depend on the two.

This will require some assembly as we construct a network of packages manually with edges being directed relationships (one of "Depends", "Imports", or "Suggests") as defined in DESCRIPTION for all packages. The following code builds this igraph object (we chose igraph for its functions utilized later):

cranInfoDF = as.data.frame(cranInfo, stringsAsFactors = FALSE)
 
edges = ddply(cranInfoDF, .(Package), function(x) {
  # split all implied (depends, imports, and suggests) packages and then concat into single array
  l = unlist(sapply(x[c('Depends','Imports','Suggests')], strsplit, split="(,|, |,\n|\n,| ,| , )"))
 
  # remove version info and empty fields that became NA
  l = gsub("^([^ \n(]+).*$", "\\1", l[!is.na(l)])
 
  # take care of empty arrays
  if (is.null(l) || length(l) == 0) 
    NULL
  else
    data.frame(Package = x['Package'], Implies = l, stringsAsFactors = FALSE)
} )
 
edges.mat = as.matrix(edges, ncol=2, dimnames=c('from','to'))
pkg.graph = graph_from_edgelist(edges.mat, directed = TRUE)

Created by Pretty R at inside-R.org

The resulting network pkg.graph contains all CRAN packages and their relationships. Let's extract and compare the neighborhoods for the two packages we are interested in:

# build subgraphs for each package
subgraphs = make_ego_graph(pkg.graph, order=1, nodes=c("igraph","network"), mode = "in")
g.igraph = subgraphs[[1]]
g.network = subgraphs[[2]]
 
# plotting subgraphs
V(g.igraph)$color = ifelse(V(g.igraph)$name == "igraph", "orange", "lightblue")
plot(g.igraph, main="Packages pointing to igraph")
V(g.network)$color = ifelse(V(g.network)$name == "network", "orange", "lightblue")
plot(g.network, main="Packages pointing to network")

Created by Pretty R at inside-R.org

The igraph neighborhood is much denser populated subgraph than the network neighborhood and hence its importance and acceptance must be higher.

Package Centrality Scores

Package igraph can produce various centrality measures on the nodes of a graph. In particular, pagerank centrality and eigenvector centrality scores are principal indicators of the importance of a node in given graph. We finish this exercise with validation using centrality scores for our initial conclusion that igraph package is more accepted and utilized across CRAN ecosystem than network package:

# PageRank
pkg.pagerank = page.rank(pkg.graph, directed = TRUE)
 
# Eigenvector Centrality
pkg.ev = evcent(pkg.graph, directed = TRUE)
 
toplot = rbind(data.frame(centrality="pagerank", type = c('igraph','network'), 
                          value = pkg.pagerank$vector[c('igraph','network')]),
               data.frame(centrality="eigenvector", type = c('igraph','network'), 
                          value = pkg.ev$vector[c('igraph','network')]))
 
library(ggplot2)
library(ggthemes)
ggplot(toplot) +
  geom_bar(aes(type, value, fill=type), stat="identity") +
  facet_wrap(~centrality, ncol = 2)

Created by Pretty R at inside-R.org

Conclusion

Both packages igraph and network are widely used across CRAN ecosystem. Due to its versatility and rich set of functions igraph leads in acceptance and importance. But as far as graph objects concern it is still a matter of the requirements to prefer one's or another's objects in R.