Correlation Primer with Aster and R

Calculating correlations is often starting point before more advanced analytical steps take place. Big data (long data) always presents computational challenges of both scale and distributed nature. In turn they may get aggravated by the presence of large number of features (wide data). But challenges do not stop here as complex relationships induce analysis of correlations across subsets and groups.

Such mix of long and wide becomes more common in the age of internet-of-things, sensor and machine data with non-human data sources dominating analytical use cases.  
Thus, when computing correlations on big data the following capabilities matter:

• scale on large distributed data sets (long data)
• scale on wide distributed data sets (wide data / large number of features)
• flexibility on wide data sets (ability to permutate features such as Cartesian combinations, one-to-many, etc.)
• correlations on subsets and groups.

Correlations in R comes standard with stats function cor but it doesn't meet most of the capabilities above. As always Teradata Aster big data analytical platform offers both scalability and functionality far exceeding capabilities above. And thanks to Aster R (TeradataAsterR) package it is available without leaving R environment.

With Aster and R integration there are multiple ways of correlating on datasets. Before sending you to the link for detailed discussion I summarized approaches discussed there by the capabilities:

Method / Solution features	Variable (columns) Permutations	Calculating for Groups	SQL-MR	In-database R
Aster R ta.cor	N	N	Y	N
Aster R in-database ta.tapply	N	Y	N	Y
toaster computeCorrelations	Y	Y	Y	N

Please visit my latest RPubs post for detailed discussion and comparison of these methods.