Most of the time, general statistics calculations are easy. Take all data points you have, find the average, the standard deviation, and you have 90% of stuff you need to present a result in a reasonable and familiar manner. However, what if you have streaming data?
Well, then you have a Welford's method. This magical algorithm enables you to calculate both average and standard deviation as data arrives without wasting a lot of memory accumulating it all.
So, of course, I wrote a C# code for it. To use it, just add something like this:
var stats = new WelfordVariance();
while(true) {
stats.Add(something.getValue());
output.SetMean(stats.Mean);
output.SetStandardDeviation(stats.StandardDeviation);
}Algorithm is not perfect and will sometime slightly differ from classically calculated standard deviation but it's generally within a spitting distance and uses minimum of memory. It's hard to find better bang-for-buck when it comes to large datasets.