Such mix of long and wide becomes more common in the age of internet-of-things, sensor and machine data with non-human data sources dominating analytical use cases.
Thus, when computing correlations on big data the following capabilities matter:
- scale on large distributed data sets (long data)
- scale on wide distributed data sets (wide data / large number of features)
- flexibility on wide data sets (ability to permutate features such as Cartesian combinations, one-to-many, etc.)
- correlations on subsets and groups.
Correlations in R comes standard with stats function cor but it doesn't meet most of the capabilities above. As always Teradata Aster big data analytical platform offers both scalability and functionality far exceeding capabilities above. And thanks to Aster R (TeradataAsterR) package it is available without leaving R environment.
With Aster and R integration there are multiple ways of correlating on datasets. Before sending you to the link for detailed discussion I summarized approaches discussed there by the capabilities:
Method / Solution features | Variable (columns) Permutations | Calculating for Groups | SQL-MR | In-database R |
---|---|---|---|---|
Aster R ta.cor |
N
|
N
|
Y
|
N
|
Aster R in-database ta.tapply |
N
|
Y
|
N
|
Y
|
toaster computeCorrelations |
Y
|
Y
|
Y
|
N
|
No comments:
Post a Comment