class static

analytics.BufferedTDigest

TDigest is a methods that approximates the CDF function of streaming measurements. Data structure useful for percentile and quantile estimation for online data streams. It can be added to any anomaly detector to set the number of alarms triggered as a percentage of the total samples. Adding new samples to the distribution is achieved through insert and querying the model (computing quantiles) is implemented by quantile function.
This is based on the Data Lib Sketch Implementation: t-digest.js
Paper: Ted Dunning, Otmar Ertl - Computing Extremely Accurate Quantiles Using t-Digests.

Properties

memory

size

init

Methods

flush()
getParams()

insert(x)
quantile(x)

save(fout)

new BufferedTDigest([arg])

TDigest quantile estimation on streams

Example

// import modules
var qm = require('qminer');
var fs = qm.fs;
var analytics = qm.analytics;
// create the default BufferedTDigest object
var tdigest = new analytics.quantiles.BufferedTDigest();
// create the data used for calculating quantiles
var inputs = [10, 1, 2, 8, 9, 5, 6, 4, 7, 3];
// fit the BufferedTDigest model
for (var i = 0; i < inputs.length; i++) {
    tdigest.insert(inputs[i]);
}
// make the estimation for the 0.1 quantile
var quant = tdigest.quantile(0.1);
// save the model
tdigest.save(fs.openWrite('tdigest.bin')).close();
// open the tdigest model under a new variable
var tdigest2 = new analytics.quantiles.BufferedTDigest(fs.openRead('tdigest.bin'));

Parameter

Name	Type	Optional	Description
arg	(module:analytics.quantiles~BufferedTDigestParam or module:fs.FIn)	Yes	Construction arguments. There are two ways of constructing: 1. Using the module:analytics.quantiles~BufferedTDigestParam object, 2. using the file input stream module:fs.FIn.

Properties

static

memory

Returns the models current memory consumption.

static

size

Returns the current size of the algorithms summary in number of tuples.

init

Returns true when the model has enough data to initialize. Type boolean.

Example

// import modules
var qm = require('qminer');
var analytics = qm.analytics;
var fs = qm.fs;
// create the default BufferedTDigest object
var tdigest = new analytics.quantiles.BufferedTDigest();
// check if the model has enough data to initialize
if (tdigest.init) { console.log("Ready to initialize"); }

Methods

flush() → module:analytics.quantiles.BufferedTDigest

Flushed the input buffer.

Example

// import modules
var qm = require('qminer');
var analytics = qm.analytics;
// create the default BufferedTDigest object
var tdigest = new analytics.quantiles.BufferedTDigest();
// create the data used for calculating quantiles
var inputs = [10, 1, 2, 8, 9, 5, 6, 4, 7, 3];
// fit the BufferedTDigest model
for (var i = 0; i < inputs.length; i++) {
    tdigest.insert(inputs[i]);
}

tdigest.flush()
// make the estimation for the 0.1 quantile
var quant = tdigest.quantile(0.1);

Returns: module:analytics.quantiles.BufferedTDigestB Self. The model has been updated.

getParams() → module:analytics.quantiles~BufferedTDigestParam

Returns the parameters.

Example

// import modules
var qm = require('qminer');
var analytics = qm.analytics;
// create the default BufferedTDigest object
var tdigest = new analytics.quantiles.BufferedTDigest();
// get the parameters of the object
var params = tdigest.getParams();

Returns: module:analytics.quantiles~BufferedTDigestParamB The construction parameters.

insert(x) → module:analytics.quantiles.BufferedTDigest

Adds a new measurement to the model and updates the approximation of the data distribution.

Example

// import modules
var qm = require('qminer');
var analytics = qm.analytics;
// create the default BufferedTDigest object
var tdigest = new analytics.quantiles.BufferedTDigest();
// create the data used for calculating quantiles
var inputs = [10, 1, 2, 8, 9, 5, 6, 4, 7, 3];
// fit the BufferedTDigest model with all input values
for (var i = 0; i < inputs.length; i++) {
    tdigest.insert(inputs[i]);
}

Parameter

Name	Type	Optional	Description
x	number		Input number.

Returns: module:analytics.quantiles.BufferedTDigestB Self. The model has been updated.

quantile(x) → number

Returns a quantile given input number, that is the approximate fraction of samples smaller than the input (0.05 means that 5% of data is smaller than the input value).

Example

// import modules
var qm = require('qminer');
var analytics = qm.analytics;
// create the default BufferedTDigest object
var tdigest = new analytics.quantiles.BufferedTDigest();
// create the data used for calculating quantiles
var inputs = [10, 1, 2, 8, 9, 5, 6, 4, 7, 3];
// fit the BufferedTDigest model
for (var i = 0; i < inputs.length; i++) {
    tdigest.insert(inputs[i]);
}

tdigest.flush()
// make the estimation for the 0.1 quantile
var quant = tdigest.quantile(0.1);

Parameter

Name	Type	Optional	Description
x	number		Input number.

Returns: numberB Quantile (between 0 and 1).

save(fout) → module:fs.FOut

Saves BufferedTDigest internal state into (binary) file.

Example

// import modules
var qm = require('qminer');
var analytics = qm.analytics;
var fs = qm.fs;
// create the default BufferedTDigest object
var tdigest = new analytics.quantiles.BufferedTDigest();
// create the data used for calculating quantiles
var inputs = [10, 1, 2, 8, 9, 5, 6, 4, 7, 3];
// fit the BufferedTDigest model
for (var i = 0; i < inputs.length; i++) {
    tdigest.insert(inputs[i]);
}
// save the model
tdigest.save(fs.openWrite('tdigest.bin')).close();
// open the tdigest model under a new variable
var tdigest2 = new analytics.quantiles.BufferedTDigest(fs.openRead('tdigest.bin'));

Parameter

Name	Type	Optional	Description
fout	module:fs.FOut		The output stream.

Returns: module:fs.FOutB The output stream fout.