Excellent discussion - and the application area (cross-correlation of sensor network data) is actually quite a bit more popular than one might think! Avalanche/landslide detection, earthquakes, bridge monitoring, industrial settings...
On a tangent, if those cross-correlations are big-ish a frequency domain (FFT) correlation could be more efficient, assuming that hasn't already been tried. This would probably save storage memory as well, though I am not 100% sure. If you could get the sensors to send the FFTs directly that could be even better!
This also has echoes of a previous HN discussion on space filling curves [0], which led to the purchase of this book [1]. I still haven't dug into it yet, but maybe these are useful references for others who are interested.
Original author here. Thanks for the spacial data structures link, I'll look into it.
The correlations are done in the frequency domain. And, although I didn't look into it, I suspect they have pretty good algorithms to compute the correlations themselves. At least they have very smart people working on that.
The pseudo-code for cross-correlation implements the naive method, which is O(M*N), M and N being the lengths of the arrays being correlated. I think it might just have been used to illustrate, but just in case, the proper way of computing the cross-correlation is with the FFT, as that is "linearithmic" in time. It's commonly called "multiplication in the frequency domain" or just frequency domain convolution. Basically, take the FFT of both input sequences, multiply them, and take the inverse FFT of the product.
Original author here. They do this already, the pseudo function 'correlate' operates in the frequency domain. It doesn't negate the fact that you still correlate the signals of different stations pairwise.
Ah crap... I will not troll anymore on HN. ::slap::
I will not troll anymore on HN. ::slap::
I will not troll anymore on HN. ::slap::
I will not troll anymore on HN. ::slap::
On a tangent, if those cross-correlations are big-ish a frequency domain (FFT) correlation could be more efficient, assuming that hasn't already been tried. This would probably save storage memory as well, though I am not 100% sure. If you could get the sensors to send the FFTs directly that could be even better!
This also has echoes of a previous HN discussion on space filling curves [0], which led to the purchase of this book [1]. I still haven't dug into it yet, but maybe these are useful references for others who are interested.
[0] https://news.ycombinator.com/item?id=7480857
[1] http://www.amazon.com/Space-Filling-Curves-Introduction-Appl...