Properties

Classes

Properties

static
flags

Returns an object with all compile flags. Type module:qm~QMinerFlags.

Example

// import qm module
var qm = require('qminer');
// get the compile flags
var flags = qm.flags;

static
version

Returns the module version.

Example

// import qm module
var qm = require('qminer');
// get the module version
var version = qm.version;
Returns

string The module version.

Methods

static
config([configPath][, overwrite][, portN][, cacheSize])

Creates a directory structure.

Parameters

Name Type Optional Description

configPath

string

Yes

The path to configuration file.

Defaults to 'qm.conf'.

overwrite

boolean

Yes

If you want to overwrite the configuration file.

Defaults to false.

portN

number

Yes

The number of the port. Currently not used.

Defaults to 8080.

cacheSize

number

Yes

Sets available memory for indexing (in MB).

Defaults to 1024.

static
create([configPath][, schemaPath][, clear]) → module:qm.Base

Creates an empty base.

Parameters

Name Type Optional Description

configPath

string

Yes

Configuration file path.

Defaults to 'qm.conf'.

schemaPath

string

Yes

Schema file path.

Defaults to ''.

clear

boolean

Yes

Clear the existing db folder.

Defaults to false.

Returns

module:qm.Base The newly created base.

static
open([configPath][, readOnly]) → module:qm.Base

Opens a base.

Parameters

Name Type Optional Description

configPath

string

Yes

The configuration file path.

Defaults to 'qm.conf'.

readOnly

boolean

Yes

Open in read-only mode.

Defaults to false.

Returns

module:qm.Base The loaded base.

static
stats()

Returns an JSON with two properties: "byClass" and "total". The "byClass" value is a JSON where each key is a class ID and each value is of the form { newFromCpp: number, newFromJs: number, destructorCalls: number} and the value of "total" is of the same form (aggregated over "byClass")

static
verbosity([level])

Set verbosity of QMiner internals.

Parameter

Name Type Optional Description

level

number

Yes

verbosity level. Possible options:
1. 0 - No output,
2. 1 - Log output,
3. 2 - Log and debug output.

Defaults to 0.

Abstract types

inner
BaseConstructorParam  Object

Base constructor parameter used for module:qm.Base.

Properties

Name Type Optional Description

mode

string

Yes

Base access mode. Can be one of the following:
1. 'create' - Sets up the db folder,
2. 'createClean' - Cleans db folder and then sets it up,
3. 'open' - Opens the db with read/write permissions,
4. 'openReadOnly' - Opens the db in read only mode.

Defaults to 'openReadOnly'.

indexCache

number

Yes

The ammount of memory reserved for indexing (in MB).

Defaults to 1024.

storeCache

number

Yes

The ammount of memory reserved for store cache (in MB).

Defaults to 1024.

schemaPath

string

Yes

The path to schema definition file.

Defaults to ''.

schema

Array of module:qm~SchemaDef

Yes

Schema definition object array.

Defaults to [].

dbPath

string

Yes

The path to db directory.

Defaults to './db/'.

inner
BaseLoadCSVParam  object

The parameter given to module:qm.Base#loadCSV.

Properties

Name Type Optional Description

file

string

 

The name of the input file.

store

string

 

Name of the store which will be created.

base

module:qm.Base

 

QMiner base object that creates the store.

delimiter

string

Yes

Optional delimiter.

Defaults to ','.

quote

string

Yes

Optional character to escape values that contain a delimiter.

Defaults to '"'.

inner
DetailKeyObject  object

The details about the key object used in module:qm.Store#key and module:qm.Store#keys.

Properties

Name Type Optional Description

fq

module:la.IntVector

 

The frequency.

vocabulary

module:la.StrVector

 

The vocabulary.

name

string

 

The key name.

store

module:qm.Store

 

The store.

inner
FeatureExtractor  Object

Feature extractor types. Used for constructing module:qm.FeatureSpace objects.

Properties

Name Type Optional Description

constant

module:qm~FeatureExtractorConstant

 

The constant type. Adds a constant value as a feature.

random

module:qm~FeatureExtractorRandom

 

The random type. Adds a random value as a feature.

numeric

module:qm~FeatureExtractorNumeric

 

The numeric type. Adds the numeric value as a feature.

categorical

module:qm~FeatureExtractorCategorical

 

The categorical type.

multinomial

module:qm~FeatureExtractorMultinomial

 

The multinomial type.

text

module:qm~FeatureExtractorText

 

The text type. Creates the bag-of-words text representation.

join

module:qm~FeatureExtractorJoin

 

The join type.

pair

module:qm~FeatureExtractorPair

 

The pair type.

jsfunc

module:qm~FeatureExtractorJsfunc

 

The jsfunc type. Allows creating a custom feature extractor.

dateWindow

module:qm~FeatureExtractorDateWindow

 

The date window type.

sparseVector

module:qm~FeatureExtractorSparseVector

 

The sparse vector type.

inner
FeatureExtractorCategorical  Object

The feature extractor of type 'categorical'. Used for constructing module:qm.FeatureSpace objects.

Example

var qm = require('qminer');
// create a simple base, where each record contains the student name and it's study group
// here we know the student is part of only one study group
var base = new qm.Base({
   mode: 'createClean',
   schema: [{
      name: "Class",
      fields: [
         { name: "Name", type: "string" },
         { name: "StudyGroup", type: "string" }
      ]
   }]
});
// create a feature space containing the categorical extractor, where it's values
// are taken from the field "StudyGroup": "A", "B", "C" and "D"
var ftr = new qm.FeatureSpace(base, { type: "categorical", source: "Class", field: "StudyGroup" });
// add a few records to the store
base.store("Class").push({ Name: "Fred", StudyGroup: "A" });
base.store("Class").push({ Name: "Wilma", StudyGroup: "B" });
base.store("Class").push({ Name: "Barney", StudyGroup: "C" });
// update the feature space to get the categories 
ftr.updateRecords(base.store("Class").allRecords);
// get the feature vector for the first record
var vec = ftr.extractVector(base.store("Class")[0]); // returns vector [1, 0, 0]
base.close();

Properties

Name Type Optional Description

type

string

 

The type of the extractor. Important: It must be equal 'categorical'.

values

Array of Object

Yes

A fixed set of values, which form a fixed feature set. No dimensionality changes if new values are seen in the upgrades.

hashDimension

number

Yes

A hashing code to set the fixed dimensionality. All values are hashed and divided modulo hasDimension to get the corresponding dimension.

field

string

 

The name of the field form which to take the values.

source

string

 

The store name.

inner
FeatureExtractorConstant  Object

The feature extractor of type 'contant'. Used for constructing module:qm.FeatureSpace objects.

Example

var qm = require('qminer');
// create a simple base, where each record contains only a persons name
var base = new qm.Base({
  mode: 'createClean',
  schema: [{
     name: "Person",
     fields: [{ name: "Name", type: "string" }]
  }]
});
// create a feature space containing the constant extractor, where the constant is equal 5
var ftr = new qm.FeatureSpace(base, { type: "constant", source: "Person", const: 5 });
// add a new record to the base
base.store("Person").push({ Name: "Peterson" });
// get the features of the record
var vec = ftr.extractVector(base.store("Person")[0]); // the vector [5]
base.close();

Properties

Name Type Optional Description

type

string

 

The type of the extractor. Important: It must be equal 'constant'.

const

number

Yes

A constant number.

Defaults to 1.0.

source

string

 

The store name.

inner
FeatureExtractorDateWindow  Object

The feature extractor of type 'dateWindow'. Used for constructing module:qm.FeatureSpace objects.

Example

// import qm module
var qm = require('qminer');

Properties

Name Type Optional Description

type

string

 

The type of the extractor. Important: It must be equal 'dateWindow'.

unit

string

Yes

How granular is the time window. Possible options are 'day', 'week', 'month', 'year', '12hours', '6hours', '4hours', '2hours', 'hour', '30minutes', '15minutes', '10minutes', 'minute', 'second'.

Defaults to 'day'.

window

number

Yes

The size of the window.

Defaults to 1.

normalize

boolean

Yes

Normalize the resulting vector of the extractor to have L2 norm 1.0.

Defaults to 'false'.

start

number

 

//TODO

end

number

 

//TODO

source

string

 

The store name.

inner
FeatureExtractorJoin  Object

The feature extractor of type 'join'. Used for constructing module:qm.FeatureSpace objects.

Example

// import qm module
var qm = require('qminer');

Properties

Name Type Optional Description

type

string

 

The type of the extractor. Important: It must be equal 'join'.

bucketSize

number

Yes

The size of the bucket in which we group consecutive records.

Defaults to 1.

source

string

 

The store name.

inner
FeatureExtractorJsfunc  Object

The feature extractor of type 'jsfunc'. Used for constructing module:qm.FeatureSpace objects.

Example

var qm = require('qminer');
// create a simple base, where each record contains the name of the student and his study groups
// each student is part of multiple study groups
var base = new qm.Base({
   mode: 'createClean',
   schema: [{
      name: "Class",
      fields: [
         { name: "Name", type: "string" },
         { name: "StudyGroups", type: "string_v" }
      ]
   }]
});
// create a feature space containing the jsfunc extractor, where the function counts the number
// of study groups each student is part of. The functions name is "NumberOFGroups", it's dimension
// is 1 (returns only one value, not an array)
var ftr = new qm.FeatureSpace(base, {
    type: "jsfunc", source: "Class", name: "NumberOfGroups", dim: 1,
    fun: function (rec) { return rec.StudyGroups.length; }
});
// add a record to the store
base.store("Class").push({ Name: "Gaben", StudyGroups: ["A", "B", "D"] });
// get the feature vector of the record
var vec = ftr.extractVector(base.store("Class")[0]); // returns vector [3]
base.close();

Properties

Name Type Optional Description

type

string

 

The type of the extractor. Important: It must be equal 'jsfunc'.

name

string

 

The features name.

fun

function()

 

The javascript function callback. It takes one parameter:
1. rec - The record. Type module:qm.Record. It returns number or module:la.Vector.

dim

number

Yes

The dimension of the feature extractor.

Defaults to 1.

source

string

 

The store name.

inner
FeatureExtractorMultinomial  Object

The feature extractor of type 'multinomial'. Used for constructing module:qm.FeatureSpace objects.

Example

var qm = require('qminer');
// create a simple base, where each record contains the student name and an array of study groups
// here we know a student can be part of multiple study groups
var base = new qm.Base({
   mode: 'createClean',
   schema: [{
      name: "Class",
      fields: [
         { name: "Name", type: "string" },
         { name: "StudyGroups", type: "string_v" }
      ]
   }]
});
// create a feature space containing the multinomial extractor, where the values are normalized,
// and taken from the field "StudyGroup": "A", "B", "C", "D", "E", "F"
var ftr = new qm.FeatureSpace(base, {
    type: "multinomial", source: "Class", field: "StudyGroups", normalize: true
});
// add a few records to the store
base.store("Class").push({ Name: "Fred", StudyGroups: ["A", "B"] });
base.store("Class").push({ Name: "Wilma", StudyGroups: ["B", "C"] });
base.store("Class").push({ Name: "Barney", StudyGroups: ["C", "A"] });
// update the feature space to get the categories
ftr.updateRecords(base.store("Class").allRecords);
// get the feature vector for the first record
var vec = ftr.extractVector(base.store("Class")[0]); // returns vector [0.707, 0.707, 0]
base.close();

Properties

Name Type Optional Description

type

string

 

The type of the extractor. Important: It must be equal 'multinomial'.

normalize

boolean

Yes

Normalize the resulting vector of the extractor to have L2 norm 1.0.

Defaults to 'false'.

transform

string

Yes

Transformation to apply to each dimension of the feature vector.
Transformation options are:
- 'log' - Compute a logarithm of the frequencies using the following formula: log(1+x)
- 'binary' - Do not compute frequencies, but only a binary indicator whether the category appears

values

Array of Object

Yes

A fixed set of values, which form a fixed feature set, no dimensionality changes if new values are seen in the updates. Cannot be used the same time as datetime.

hashDimension

number

Yes

A hashing code to set the fixed dimensionality. All values are hashed and divided modulo hashDimension to get the corresponding dimension.

datetime

Object

Yes

Same as 'values', only with predefined values which are extracted from date and time (month, day of month, day of week, time of day, hour).
This fixes the dimensionality of feature extractor at the start, making it not dimension as new dates are seen. Cannot be used the same time as values.

Defaults to false.

field

(string or Array of String)

 

The name of the field from which to take the key value.

valueField

(string or Array of String)

Yes

The name of the field from which to take the numeric value.
Defaults to 1.0 for non-zero elements in the vector.

source

string

 

The store name.

inner
FeatureExtractorNumeric  Object

The feature extractor of type 'numeric'. Used for constructing module:qm.FeatureSpace objects.

Example

var qm = require('qminer');
// create a simple base, where each record contains the student name and it's grade
var base = new qm.Base({
   mode: 'createClean',
   schema: [{
      name: "Class",
      fields: [
         { name: "Name", type: "string" },
         { name: "Grade", type: "int" }
      ]
   }]
});
// create a feature space containing the numeric extractor, where the values are
// normalized, the values are taken from the field "Grade"
var ftr = new qm.FeatureSpace(base, { type: "numeric", source: "Class", normalize: "scale", field: "Grade" });
// add a new record to the base
base.store("Class").push({ Name: "Peterson", Grade: 9 });
base.store("Class").push({ Name: "Ericsson", Grade: 8 });
// update the feature space for scaling 
ftr.updateRecords(base.store("Class").allRecords);
// get the features of the first record
var vec = ftr.extractVector(base.store("Class")[0]); // the vector with the random value
base.close();

Properties

Name Type Optional Description

type

string

 

The type of the extractor. Important: It must be equal 'numeric'.

normalize

string

Yes

Normalize values between 0.0 and 1.0 if set to 'scale'. Standardize values to mean = 0 and sdiv = 1 if set to 'var'.

Defaults to 'none'.

min

number

Yes

The minimal value used to form the normalization.

max

number

Yes

The maximal value used to form the normalization.

field

string

 

The name of the field from which to take the value.

source

string

 

The store name.

inner
FeatureExtractorPair  Object

The feature extractor of type 'pair'. Used for constructing module:qm.FeatureSpace objects.

Example

var qm = require('qminer');

Properties

Name Type Optional Description

type

string

 

The type of the extractor. Important: It must be equal 'pair'.

first

module:qm~FeatureExtractors

 

The first feature extractor.

second

module:qm~FeatureExtractors

 

The second feature extractor.

source

source

 

The store name.

inner
FeatureExtractorRandom  Object

The feature extractor of type 'random'. Used for constructing module:qm.FeatureSpace objects.

Example

var qm = require('qminer');
// create a simple base, where each record contains only a persons name
var base = new qm.Base({
  mode: 'createClean',
  schema: [{
     name: "Person",
     fields: [{ name: "Name", type: "string" }]
  }]
});
// create a feature space containing the random extractor
var ftr = new qm.FeatureSpace(base, { type: "random", source: "Person" });
// add a new record to the base
base.store("Person").push({ Name: "Peterson" });
// get the features of the record
var vec = ftr.extractVector(base.store("Person")[0]); // the vector with the random value
base.close();

Properties

Name Type Optional Description

type

string

 

The type of the extractor. Important: It must be equal 'random'.

seed

number

Yes

The seed number used to construct the random number.

Defaults to 0.

source

string

 

The store name.

inner
FeatureExtractorSparseVector  Object

The feature extractor of type 'num_sp_v'. Used for constructing module:qm.FeatureSpace objects.

Example

var qm = require('qminer');
// create a simple base, where each record contains the student name and it's grade
var base = new qm.Base({
   mode: 'createClean',
   schema: [{
      "name": "Class",
      "fields": [
         { "name": "Name", "type": "string" },
         { "name": "Features", "type": "num_sp_v" }
      ]
   }]
});
// create a feature space containing the numeric extractor, where the values are
// normalized, the values are taken from the field "Grade"
var ftr = new qm.FeatureSpace(base, { type: "num_sp_v", source: "Class", normalize: false, field: "Features" });
base.close();

Properties

Name Type Optional Description

type

string

 

The type of the extractor. Important: It must be equal 'num_sp_v'.

dimension

number

Yes

Dimensionality of sparse vectors.

Defaults to 0.

normalize

boolean

Yes

Normalize vectors to L2 norm of 1.0.

Defaults to false.

field

string

 

The name of the field from which to take the value.

source

string

 

The store name.

inner
FeatureExtractorText  Object

The feature extractor of type 'text'. Used for constructing module:qm.FeatureSpace objects.

Example

var qm = require('qminer');
// create a simple base, where each record contains the title of the article and it's content
var base = new qm.Base({
   mode: 'createClean',
   schema: [{
      name: "Articles",
      fields: [
         { name: "Title", type: "string" },
         { name: "Text", type: "string" }
      ]
   }]
});
// create a feature spave containing the text (bag of words) extractor, where the values are normalized,
// weighted with 'tfidf' and the tokenizer is of 'simple' type, it uses english stopwords.
var ftr = new qm.FeatureSpace(base, {
    type: "text", source: "Articles", field: "Text", normalize: true, weight: "tfidf",
    tokenizer: { type: "simple", stopwords: "en"}
});
// add a few records to the store
base.store("Articles").push({ Title: "Article1", Text: "The last time we drew up the league table for bad banks, UBS was on top." });
base.store("Articles").push({ Title: "Article2", Text: "Barclays dropped a bombshell on its investment bankers last week." });
// update the feature space to get the the terms for the bag-of-words text representation
ftr.updateRecords(base.store("Articles").allRecords);
// extract the feature vector for the first article
var vec = ftr.extractSparseVector(base.store("Articles")[0]);
base.close();

Properties

Name Type Optional Description

type

string

 

The type of the extractor. Important: It must be equal 'text'.

normalize

boolean

Yes

Normalize the resulting vector of the extractor to have L2 norm 1.0.

Defaults to 'true'.

weight

string

Yes

Type of weighting used for scoring terms. Possible options are:
1. 'none' - Sets 1 if the term occurs and 0 otherwise.
2. 'tf' - Sets the term frequency in the document.
3. 'idf' - Sets the inverse document frequency in the document.
4. 'tfidf' - Sets the product of the tf and idf frequency.

Defaults to 'tfidf'.

hashDimension

number

Yes

A hashing code to set the fixed dimensionality. All values are hashed and divided modulo hashDimension to get the corresponding dimension.

hashTable

boolean

Yes

If true, stores the hash table which is used for for feature naming.

Defaults to false.

field

string

 

The name of the field from which to take the value.

tokenizer

module:qm~FeatureTokenizer

 

The settings for extraction of text.

mode

string

Yes

How are multi-record cases combined into single vector. Possible options:
1. 'concatenate' - Multi-record cases are merged into one document.
2. 'centroid' - Treat each case as a seperate document.
3. 'tokenized' - use the tokenizer option.

stream

string

Yes

Details on forgetting old IDFs when running on stream. Possible options:
1. 'field' - Field name which is providing timestamp. If missing, system time is used (optional).
2. 'factor' - Forgetting factor, by which the olf IDFs are multiplied after each iteration.
3. 'interval' - The time between iterations when the factor is applied, standard JSON time format is used to specify the interval duration.

source

string

 

The store name.

inner
FeatureTokenizer  Object

The settings for extraction of text used in module:qm~FeatureExtractorText.

Properties

Name Type Optional Description

type

string

Yes

The type of the encoding text. Possible options:
1. 'simple' - The simple encoding.
2. 'html' - The html encoding.
3.B2'unicode' - The unicode encoding.

Defaults to 'simple'.

stopwords

(string or Array of string)

Yes

The stopwords used for extraction. Possible options:
1. 'none' - No pre-defined stopword list. Type string.
2. 'en' - The english pre-defined stopword list. Type string.
3. 'si' - The slovene pre-defined stopword list. Type string.
4. 'es' - The spanish pre-defined stopword list. Type string.
5. 'de' - The german pre-defined stopword list. Type string.
6. array - An array of stopwords. The array must be given as a parameter. Type Array of strings.

Defaults to 'en'.

stemmer

string

Yes

The stemmer used for extraction. Possible options:
1. 'true' - Using the porter stemmer.
2. 'porter' - Using the porter stemmer.
3. 'none' - Using no stemmer.

Defaults to 'none'.

uppercase

boolean

Yes

Changing all words to uppercase.

Defaults to 'true'.

inner
PerformanceStat  object

The performance statistics used to describe module:qm~PerformanceStatBase and module:qm~PerformanceStatStore.

Properties

Name Type Optional Description

alloc_count

number

 

\ TODO: Add the description

alloc_size

number

 

\ TODO: Add the description

alloc_unused_size

number

 

\ TODO: Add the description

avg_get_len

number

 

\ TODO: Add the description

avg_put_len

number

 

\ TODO: Add the description

avg_get_new_len

number

 

\ TODO: Add the description

dels

number

 

\ TODO: Add the description

gets

number

 

\ TODO: Add the description

puts

number

 

\ TODO: Add the description

puts_new

number

 

\ TODO: Add the description

released_count

number

 

\ TODO: Add the description

released_size

number

 

\ TODO: Add the description

size_changes

number

 

\ TODO: Add the description

inner
PerformanceStatBase  object

The performance statistics that is returned by module:qm.Base#getStats.

Properties

Name Type Optional Description

stores

Array of module:qm~PerformanceStatStore

 

The performance statistics of the stores in base. \ TODO: Check if this is right

gix_stats

object

 

The statistics of the base. \ TODO: Check if this is right

Values in gix_stats have the following properties:

Name Type Optional Description

avg_len

number

 

The average length. \ TODO: Check if this is right

cache_all

number

 

The number of cache. \ TODO: Check if this is right

cache_all_loaded_perc

number

 

\ TODO: Add the description

cache_dirty

number

 

\ TODO: Add the description

cache_dirty_loaded_perc

number

 

\ TODO: Add the description

mem_sed

number

 

\ TODO: Add the description

gix_blob

module:qm~PerformanceStat

 

\ TODO: Add the description

inner
PerformanceStatStore  object

The performance statistics of the store found in module:qm~PerformanceStatBase.

Properties

Name Type Optional Description

name

string

 

Store name.

blob_storage_memory

module:qm~PerformanceStat

 

\ TODO: Add the description

blob_storage_cache

module:qm~PerformanceStat

 

\ TODO: Add the description

inner
QMinerFlags  Object

The object containing the QMiner compile flags.

Properties

Name Type Optional Description

buildTime

string

 

The module build time.

win

boolean

 

True, if the module is compiled for Windows.

linux

boolean

 

True, if the module is compiled for Linux.

darwin

boolean

 

True, if the module is compiled for Darwin.

x86

boolean

 

True, if the module is compiled for x86 system.

x64

boolean

 

True, if the module is compiled for x64 system.

omp

boolean

 

True, if the module is compiled for omp.

debug

boolean

 

True, if the module is compiled for debug mode.

gcc

boolean

 

True, if the module is compiled for gcc.

clang

boolean

 

True, if the module is compiled for clang.

blas

boolean

 

True, if the module is compiled with Blas.

blas_intel

boolean

 

True, if the module is compiled with Intel Blas.

blas_amd

boolean

 

True, if the module is compiled with AMD Blas.

blas_openblas

boolean

 

True, if the module is compiled with OpenBLAS.

lapacke

boolean

 

True, if the module is compiled with Lapacke.

inner
QueryObject  object

The object used for querying records with module:qm.Base#search. How to construct a query is found on the QMiner Wiki page.

inner
SchemaDef  Object

Store schema definition used in module:qm~BaseConstructorParam.

Example

var qm = require('qminer');
// create a simple movies store, where each record contains only the movie title.
var base = new qm.Base({
    mode: 'createClean',
    schema: [{
      name: "Movies",
      fields: [{ name: "title", type: "string" }]
    }]
});
base.close();

Properties

Name Type Optional Description

name

string

 

The name of the store. Store name can be composed by from English letters, numbers, _ or $ characters. It can only begin with a character.

fields

Array of module:qm~SchemaFieldDef

 

The array of field descriptors.

joins

Array of module:qm~SchemaJoinDef

Yes

The array of join descriptors, used for linking records from different stores.

Defaults to [].

keys

Array of module:qm~SchemaKeyDef

Yes

The array of key descriptors. Keys define how records are indexed, which is needed for search using the query language.

Defaults to [].

timeWindow

module:qm~SchemaTimeWindowDef

Yes

Time window description. Stores can have a window, which is used by garbage collector to delete records once they fall out of the time window. Window can be defined by number of records or by time.

inner
SchemaFieldDef  Object

Store schema field definition used in module:qm~SchemaDef.

Example

var qm = require('qminer');
 var base = new qm.Base({
     mode: 'createClean',
     schema: [
       { name: 'NewsArticles',
         fields: [
           { name: "ID", primary: true, type: "string", shortstring: true },
           { name: "Source", type: "string", codebook: true },
           { name: "DateTime", type: "datetime" },
           { name: "Title", type: "string", store: "cache" },
           { name: "Tokens", type: "string_v", store: "cache", null: true },
           { name: "Vector", type: "num_sp_v", store: "cache", null: true }]
       }
    ]
 });
// add a record:
// - we set the date using the ISO string representation
// - we set the string vector Tokens with an array of strings
// - we set the numeric sparse vector Vector with an array of two element arrays
//   (index, value), see the sparse vector constructor {@link module:la.SparseVector}
base.store('NewsArticles').push({
  ID: 't12344',
  Source: 's1234',
  DateTime: '2015-01-01T00:05:00',
  Title: 'the title',
  Tokens: ['token1', 'token2'],
  Vector: [[0,1], [1,1]]});
base.close();

Properties

Name Type Optional Description

name

string

 

The name of the field.

type

string

 

The type of the field. Possible options:
1. 'int' - Signed 32-bit integer,
2. 'uint64' - Unsigned 64-bit integer,
3. 'int_v' - Array of signed 32-bit integers,
4. 'string' - String,
5. 'string_v' - Array of strings,
6. 'bool' - Boolean,
7. 'float' - Double precision flating point number,
8. 'float_pair' - A pair of floats, useful for storing geo coordinates,
9. 'float_v' - Array of floats,
10. 'datetime' - Date and time format, stored in a form of miliseconds since 1600,
11. 'num_sp_v' - Array of [int, float] pairs. See constructor array for module:la.SparseVector,
12. 'json' - this field can be an arbitrary object and will be internally serialized into string using JSON notation,
13. 'blob' - Binary buffer, used for storing binary data,

primary

boolean

Yes

Field which can be used to identify record. There can be only one primary field in a store. There can be at most one record for each value of the primary field. Currently following fields can be marked as primary: int, uint64, string, float, datetime. Primary fields of type string are also used for record querying using module:qm.Store#recordByName.

Defaults to false.

null

boolean

Yes

When set to true, null is a possible value for a field (allow missing values).

Defaults to false.

store

string

Yes

Defines where to store the field. Possible options
1. 'memory' - Stores the values in RAM.
2 'cache' - Stores the values on disk, with a layer of FIFO cache in RAM, storing the most recently used values.

Defaults to 'memory'.

default

Object

Yes

Default value for field when not given for a new record.

codebook

boolean

Yes

Useful when many records have only few different values of this field. If set to true, then a separate table of all values is kept, and records only point to this table (replacing variable string field in record serialisation with fixed-length integer). Useful to decrease memory footprint, and faster to update. (STRING FIELD TYPE SPECIFIC).

Defaults to false.

shortstring

boolean

Yes

Useful for string shorter then 127 characters (STRING FIELD TYPE SPECIFIC).

Defaults to false.

inner
SchemaJoinDef  Object

Store schema join definition used in module:qm~SchemaDef.

Example

var qm = require('qminer');
// Create two stores: People which stores only names of persons and Movies, which stores only titles.
// Each person can direct zero or more movies, so we use an index join named 'directed' and
// each movie has a single director, so we use a field join 'director'. The joins are
// inverses of each other. The inverse join simplifies the linking, since only one join needs
// to be specified, and the other direction can be linked automatically (in the example
// below we specify only the 'director' link and the 'directed' join is updated automatically).
//
var base = new qm.Base({
    mode: 'createClean',
    schema: [
      { name: 'People',
        fields: [{ name: 'name', type: 'string', primary: true }],
        joins: [{ name: 'directed', 'type': 'index', 'store': 'Movies', 'inverse': 'director' }] },
      { name: 'Movies',
        fields: [{ name: 'title', type: 'string', primary: true }],
        joins: [{ name: 'director', 'type': 'field', 'store': 'People', 'inverse': 'directed' }] }
    ]
});
// Adds a movie, automatically adds 'Jim Jarmusch' to People, sets the 'director' join (field join)
// and automatically updates the index join 'directed', since it's an inverse join of 'director'
base.store('Movies').push({ title: 'Broken Flowers', director: { name: 'Jim Jarmusch' } });

// Adds a movie, sets the 'director' join, updates the index join of 'Jim Jarmusch'
base.store('Movies').push({ title: 'Coffee and Cigarettes', director: { name: 'Jim Jarmusch' } });
// Adds movie, automatically adds 'Lars von Trier' to People, sets the 'director' join
// and 'directed' inverse join (automatically)
base.store('Movies').push({ title: 'Dogville', director: { name: 'Lars von Trier' } });

// Adds a person, sets the 'directed' join with multiple movies ('directed' is of type 'index', movies must be given in an array)
base.store('People').push({ name: 'Christopher Nolan', directed: [{ title: 'Inception' }, { title: 'Interstellar' }] });

var movie = base.store('Movies')[0]; // get the first movie (Broken Flowers)
// Each movie has a property corresponding to the join name: 'director'.
// Accessing the property returns a {@link module:qm.Record} from the store People.
var person = movie.director; // get the director
var personName = person.name; // get person's name ('Jim Jarmusch')

// Each person has a property corresponding to the join name: 'directed'.
// Accessing the property returns a {@link module:qm.RecSet} from the store People.
var movies = person.directed; // get all the movies the person directed.
movies.each(function (movie) { var title = movie.title; });
// Gets the following titles:
//   'Broken Flowers'
//   'Coffee and Cigarettes'
base.close();

Properties

Name Type Optional Description

name

string

 

The name of the join.

type

string

 

Join types. Possible options:
1. 'field' - Points to zero or one record and is implemented as an additional hidden field of type uint64, which can hold the ID of the record it links to. Accessing the records join returns a record.
2. 'index' - Point to any number of records and is implemented using the inverted index, where for each record a list (vector) of linked records is kept. Accessing the records join returns a record set. Important: The records given to this join field must be in an array.

store

string

 

The store name from which the linked records are.

inner
SchemaKeyDef  Object

Store schema key definition used in module:qm~SchemaDef.

Example

var qm = require('qminer');
// Create a store People which stores only names of persons.
var base = new qm.Base({
    mode: 'createClean',
    schema: [
        { name: 'People',
          fields: [{ name: 'name', type: 'string', primary: true }],
          keys: [
            { field: 'name', type: 'value'},
            { field: 'name', name: 'nameText', type: 'text'}
         ]
       }
    ]
});

base.store('People').push({name : 'John Smith'});
base.store('People').push({name : 'Mary Smith'});
// search based on indexed values
base.search({$from : 'People', name: 'John Smith'}); // Return the record set containing 'John Smith'
// search based on indexed values
base.search({$from : 'People', name: 'Smith'}); // Returns the empty record set
// search based on text indexing
base.search({$from : 'People', nameText: 'Smith'}); // Returns both records
base.close();

Properties

Name Type Optional Description

field

string

 

The name of the field that will be indexed.

type

string

 

Key type. Possible options:
1. 'value' - Indexes records using an inverted index using full value of the field (no processing). The key type supports 'string', 'string_v' and 'datetime' fields types.
2. 'text' - Indexes string fields by using a tokenizer and text processing. Supported by 'string' fields.
3. 'location'- Indexes records as points on a sphere and enables nearest-neighbour queries. Supported by 'float_pair' type fields.

name

string

Yes

Allows using a different name for the key in search queries. This allows for multiple keys to be put against the same field. Default value is the name of the field.

vocabulary

string

Yes

Defines the name of the vocabulary used to store the tokens or values. This can be used indicate to several keys to use the same vocabulary, to save on memory. Supported by 'value' and 'text' keys.

tokenize

string

Yes

Defines the tokenizer that is used for tokenizing the values stored in indexed fields. Tokenizer uses same parameters as in bag-of-words feature extractor. Default is english stopword list and no stemmer. Supported by 'text' keys.

inner
SchemaTimeWindowDef  Object

Stores can have a window, which is used by garbage collector to delete records once they fall out of the time window. Window can be defined by number of records or by time. Window defined by parameter window, its value being the number of records to be kept. Used in module:qm~SchemaDef.
Important: module:qm.Base#garbageCollect must be called manually to remove records outside time window.

Examples

Define window by number of records

var qm = require('qminer');
// create base
var base = new qm.Base({ mode: 'createClean' });
// create store with window
base.createStore({
    "name": "TestStore",
    "fields": [
        { "name": "DateTime", "type": "datetime" },
        { "name": "Measurement", "type": "float" }
    ],
    window: 3,
});

// push 5 records into created store
for (var i = 0; i < 5; i++) {
    var rec = {
        "DateTime": new Date().toISOString(),
        "Measurement": i
    };
    base.store("TestStore").push(rec);
}

// check number of records in store
base.store("TestStore").allRecords.length; // 5
// clean base with garbage collector
base.garbageCollect();
// check number of records in store
base.store("TestStore").allRecords.length; // 3
base.close();

Define window by time

var qm = require('qminer');
// create base
var base = new qm.Base({ mode: 'createClean' });
// create store with window
base.createStore({
    "name": "TestStore",
    "fields": [
        { "name": "DateTime", "type": "datetime" },
        { "name": "Measurement", "type": "float" }
    ],
    timeWindow: {
        duration: 2,
        unit: "hour",
        field: "DateTime"
    }
});

// push 5 records into created store
for (var i = 0; i < 5; i++) {
    var rec = {
        "DateTime": new Date(new Date().getTime() + i * 60 * 60 * 1001).toISOString(),
        "Measurement": i
    };
    base.store("TestStore").push(rec);
}

// check number of records in store
base.store("TestStore").allRecords.length; // 5
// clean base with garbage collector
base.garbageCollect();
// check number of records in store
base.store("TestStore").allRecords.length; // 2
base.close();

Properties

Name Type Optional Description

duration

number

 

The size of the time window (in number of units).

unit

string

 

Defines in which units the window size is specified. Possible options are 'second', 'minute', 'hour', 'day', 'week' or 'month'.

field

string

Yes

Name of the datetime field, which defines the time of the record. In case it is not given, the insert time is used in its place.

inner
StreamAggrAggrResampler  module:qm.StreamAggr

This stream aggregate resamples an input time series to a new time seris of equally spaced measurements. Each new measurement corresponds to an aggregate (sum,avg,min,max) computed over an interval. The aggregate exposes the following methods.
1. module:qm.StreamAggr#getFloat returns the last resampled value.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the last resampled value.
3. module:qm.StreamAggr#onStep reads from an input aggregate and tries to resample.
4. module:qm.StreamAggr#onTime updates the current time (no data has arrived, but time has passed) and tries to resample.
5. module:qm.StreamAggr#getParams returns a parameter object.
6. module:qm.StreamAggr#setParams used primarily for setting the out-aggregate.
The stream aggregate exposes its results through getFloat and getTimestamp methods (itself represents timeseries). The resampler has an input time-series aggregate (supports getFloat and getTimestamp), from where it reads time series values. The reading and resampling occourrs wehen resamplers onStep() or onTime() methods are called. When resampling succeeds (all the data needed for the computation becomes available), the resampler will trigger the onStep() method of an output stream aggregate that will read the resamplers state through getFloat and getTime.

Example

var qm = require('qminer');
// create a base with a simple timeseries store
var base = new qm.Base({
    mode: 'createClean',
    schema: [{
        name: 'default',
        fields: [
            { name: 'timestamp', type: 'datetime' },
            { name: 'value', type: 'float' }
        ]
    }]
});
var store = base.store('default');
// the tick aggregate reads from the store (provides time series input to other aggregates)
var raw = store.addStreamAggr({
    type: 'timeSeriesTick',
    timestamp: 'timestamp',
    value: 'value'
});

// will compute sums over 1 second intervals
var resampler = store.addStreamAggr({
    type: 'aggrResample',
    inAggr: raw.name,
    start: '1970-01-01T00:00:00.000',
    defaultValue: 0,
    aggType: 'sum',
    interval: 1000
});

// will print out resampler state on each resample
var resamplerOutput = new qm.StreamAggr(base, new function () {
    this.onStep = function () {
        console.log('Resampler emitted the sum: ' + resampler.getFloat() +
            ' for the interval [' + new Date(resampler.getTimestamp()).toISOString() +
            ' - ' + new Date(resampler.getTimestamp() + resampler.getParams().interval).toISOString() + ')');
    }
});

// IMPORTANT. After the output exists, connect it to resampler
resampler.setParams({ outAggr: resamplerOutput.name });

store.push({ timestamp:  1, value: 1 });
store.push({ timestamp: 10, value: 10 });
store.push({ timestamp: 500, value: 100 });
store.push({ timestamp: 2000, value: 1000 }); // triggers two resampling steps (two intervals complete)
// // three measurements for the first interval
// Resampler emitted the sum: 111 for the interval [1970-01-01T00:00:00.000Z - 1970-01-01T00:00:01.000Z)
// // zero measurements for the second interval (does not skip empty)
// Resampler emitted the sum: 0 for the interval [1970-01-01T00:00:01.000Z - 1970-01-01T00:00:02.000Z)
store.push({ timestamp: 2001, value: 10000 });
store.push({ timestamp: 3000, value: 100000 }); // triggers one resampling step (one interval complete)
// // one measurement for the third interval
// Resampler emitted the sum: 11000 for the interval [1970-01-01T00:00:02.000Z - 1970-01-01T00:00:03.000Z)

base.close();

Properties

Name Type Optional Description

type

string

 

The type of the stream aggregator. Important: It must be equal to 'aggrResampler'.

interval

number

 

Interval size in milliseconds

aggType

string

 

Must be one of the values: "sum", "avg", "min" or "max" - represents the function executed on the data values in the interval.

inAggr

(string or module:qm.StreamAggr)

 

The name of the input stream aggregate which must implement getFloat() and getTimestamp() methods.

start

(string or number)

Yes

Start time (linux timestamp or a web log date string like 1970-01-01T00:00:00.000)

roundStart

string

Yes

Must be one of the values: 'h', 'm' or 's' - represents rounding of the start time when it must be determined by the first observed record. 'h' will clip minutes, seconds and milliseconds, 'm' will clip seconds and milliseconds and 's' will clip only milliseconds.

defaultValue

number

Yes

default value for empty intervals (no data available).

Defaults to 0.

skipEmpty

boolean

Yes

If true, the resampler will not call the onStep method of the out-aggregate when the interval is empty (for example, average of an empty set is not defined).

Defaults to false.

name

string

Yes

The given name for the stream aggregator.

outAggr

(string or module:qm.StreamAggr)

Yes

The name of the output stream aggregate. Only useful when the outAggr is a javascript stream aggregate, otherwise the output must be set by calling setParam({outAggr: outAggregateName}).

inner
StreamAggrAnomalyDetectorNN  module:qmStreamAggr

This stream aggregator represents the anomaly detector using the Nearest Neighbor algorithm. It calculates the new incoming point's distance from its nearest neighbor and, depending on the input threshold values, it classifies the severity of the alarm. It implements the following methods:
1. module:qm.StreamAggr#getInteger returns the severity of the alarm.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the latest alarm.
3. module:qm.StreamAggr#saveJson returns the Json with the description and explanation of the alarm.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store named Cars with 4 fields
var base = new qm.Base({
    mode: 'createClean',
    schema: [{
        name: 'Cars',
        fields: [
            { name: 'NumberOfCars', type: 'float' },
            { name: 'Temperature', type: 'float' },
            { name: 'Precipitation', type: 'float' },
            { name: 'Time', type: 'datetime' }
        ]
    }]
});
// create the store
var store = base.store('Cars');
// define a feature space aggregator on the Cars store which needs at least 2 records to be initialized. Use three of the
// four fields of the store to create feature vectors with normalized values.
var aggr = {
   name: "ftrSpaceAggr",
   type: "featureSpace",
   initCount: 2,
   update: true, full: false, sparse: true,
   featureSpace: [
       { type: "numeric", source: "Cars", field: "NumberOfCars", normalize: "var" },
       { type: "numeric", source: "Cars", field: "Temperature", normalize: "var" },
       { type: "numeric", source: "Cars", field: "Precipitation", normalize: "var" }
   ]
};
//create the feature space aggregator
var ftrSpaceAggr = base.store('Cars').addStreamAggr(aggr);
// define a new time series tick stream aggregator for the 'Cars' store, that takes the values from the 'NumberOfCars' field
// and the timestamp from the 'Time' field.
var aggr = {
    name: "tickAggr",
    type: "timeSeriesTick",
    store: "Cars",
    timestamp: "Time",
    value: "NumberOfCars"
};
//create the tick aggregator
var tickAggr = base.store('Cars').addStreamAggr(aggr);

//define an anomaly detection aggregator using nearest neighbor on the cars store that takes as input timestamped features.
// The time stamp is provided by the tick aggregator while the feature vector is provided by the feature space aggregator.
var aggr = {
    name: 'AnomalyDetectorAggr',
    type: 'nnAnomalyDetector',
    inAggrSpV: 'ftrSpaceAggr',
    inAggrTm: 'tickAggr',
    rate: [0.7, 0.5, 0.15],
    windowSize: 2
};
//create the anomaly detection aggregator
var anomaly = base.store('Cars').addStreamAggr(aggr);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name for the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'nnAnomalyDetector'.

store

string

 

The name of the store from which it takes the data.

inAggr

string

 

The name of the stream aggregator to which it connects and gets data. It cannot be connect to the module:qm~StreamAggrTimeSeriesWindow.

inner
StreamAggregator  module:qm.StreamAggr

Stream aggregator types.

Properties

Name Type Optional Description

timeSeries

module:qm~StreamAggrTimeSeriesWindow

 

The time series type.

timeSeriesBufferVector

module:qm~StreamAggrTimeSeriesWindowVector

 

The time series buffer vector type.

tick

module:qm~StreamAggrTimeSeriesTick

 

The time series tick type.

record-buffer

module:qm~StreamAggrRecordBuffer

 

The record buffer type.

ftr-space

module:qm~StreamAggrFeatureSpace

 

The feature space type.

sum

module:qm~StreamAggrSum

 

The sum type. Calculates the sum of values.

min

module:qm~StreamAggrMin

 

The minimal type. Saves the minimal value in the buffer.

max

module:qm~StreamAggrMax

 

The maximal type. Saves the maximal value in the buffer.

sparse-vec-sum

module:qm~StreamAggrSparseVecSum

 

The sparse-vector-sum type.

ma

module:qm~StreamAggrMovingAverage

 

The moving average type. Calculates the average within the window.

ema

module:qm~StreamAggrEMA

 

The exponental moving average type. Calculates the exponental average of the values.

ema-sp-vec

module:qm~StreamAggrEMASpVec

 

The exponental moving average for sparse vectors type.

var

module:qm~StreamAggrMovingVariance

 

The moving variance type. Calculates the variance of values within the window.

cov

module:qm~StreamAggrMovingCovariance

 

The moving covariance type. Calculates the covariance of values within the window.

cor

module:qm~StreamAggrMovingCorrelation

 

The moving correlation type. Calculates the correlation of values within the window.

res

module:qm~StreamAggrResampler

 

The resampler type. Resamples the records so that they come in in the same time interval.

aggr-res

module:qm~StreamAggrAggrResampler

 

The aggregating (avg/sum) resampler type. Resamplers the records so that it takes the records in the time window and returns one sample.

mer

module:qm~StreamAggrMerger

 

The merger type. Merges the records from two stream series.

hist

module:qm~StreamAggrHistogram

 

The online histogram type.

slotted-hist

module:qm~StreamAggrSlottedHistogram

 

The online slotted-histogram type.

vec-diff

module:qm~StreamAggrVecDiff

 

The difference of two vectors (e.g. online histograms) type.

lin-reg

module:qm~StreamAggrSimpleLinearRegression

 

The linear regressor type.

detector-nn

module:qm~StreamAggrAnomalyDetectorNN

 

The anomaly detector type. Detects anomalies using the k nearest neighbour algorithm.

treshold

module:qm~StreamAggrThreshold

 

The threshold indicator type.

tdigest

module:qm~StreamAggrTDigest

 

The quantile estimator type. It estimates the quantiles of the given data using TDigest.

record-switch-aggr

module:qm~StreamAggrRecordSwitch

 

The record switch type.

inner
StreamAggrEMA  module:qmStreamAggr

This stream aggregator represents the exponential moving average window buffer. It calculates the weighted moving average of the values in the connected stream aggregator, where the weights are exponentially decreasing. It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the exponentional average of the values in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Heat",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});

// create a new time series tick stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window should be 1 hour.
var timeser = {
   name: 'TimeSeriesAggr',
   type: 'timeSeriesTick',
   store: 'Heat',
   timestamp: 'Time',
   value: 'Celsius'
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);

// add an exponentional moving average aggregator, that is connected with the 'TimeSeriesAggr' aggregator.
// It should interpolate with the previous value, the decay should be 3 seconds and the initWindow should be 2 seconds.
var ema = {
   name: 'emaAggr',
   type: 'ema',
   store: 'Heat',
   inAggr: 'TimeSeriesAggr',
   emaType: 'previous',
   interval: 3000,
   initWindow: 2000
};
var expoMovingAverage = base.store("Heat").addStreamAggr(ema);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name for the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'ema'.

store

string

 

The name of the store from which it takes the data.

inAggr

string

 

The name of the stream aggregator to which it connects and gets data. It cannot be connect to the module:qm~StreamAggrTimeSeriesWindow.

emaType

string

 

The type of interpolation. Possible options are:
1. 'previous' - Interpolates with the previous value.
2. 'next' - Interpolates with the next value.
3. 'linear' - Makes a linear interpolation.

interval

number

 

The time interval defining the decay. It must be greater than initWindow.

initWindow

number

 

The time window of required values for initialization.

inner
StreamAggrEMASpVec  module:qmStreamAggr

This stream aggregator represents the exponential moving average window buffer for sparse vectors. It calculates the weighted moving average of the values in the connected stream aggregator, where the weights are exponentially decreasing. It implements the following methods:
1. module:qm.StreamAggr#getValueVector returns the exponentional average of the sparse vector values in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Data",
       fields: [
           { name: "Text", type: "string" },
           { name: "Time", type: "datetime" }
       ]
   }]
});
var store = base.store("Data");
// create a new time series that emits data as sparse vector, based on text
var aggr = {
   name: 'featureSpaceWindow',
   type: 'timeSeriesWinBufFeatureSpace',
   store: store.name,
   timestamp: 'Time',
   featureSpace: {
      type: "categorical",
      source: store.name,
      field: "Text"
   },
   winsize: 1 // keep only most recent value in window
};
// attach sum
var sa = store.addStreamAggr(aggr);
var aggr2 = {
   name: 'sparseVectorSum',
   type: 'winBufSpVecSum',
   store: store.name,
   inAggr: aggr.name // this means that sum is equal to the most recent data
};
// ok, now attach EMA
var sa2 = store.addStreamAggr(aggr2);
var ema_def = {
   name: 'sparseVectorEma',
   type: 'emaSpVec',
   store: store.name,
   inAggr: aggr2.name,
   emaType: "next",
   interval: 2000,
   initWindow: 0
};
var ema = store.addStreamAggr(ema_def);
// add some data
store.push({ Time: 1000, Text: 'a' });
store.push({ Time: 2000, Text: 'b' });
store.push({ Time: 3000, Text: 'c' });
// display EMA data
ema.getValueVector().print();
base.close();

Properties

Name Type Optional Description

name

string

 

The given name for the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'ema'.

store

string

 

The name of the store from which it takes the data.

inAggr

string

 

The name of the stream aggregator to which it connects and gets data. It cannot be connect to the module:qm~StreamAggrTimeSeriesWindow.

emaType

string

 

The type of interpolation. Possible options:
1. 'previous' - Interpolates with the previous value.
2. 'next' - Interpolates with the next value.
3. 'linear' - Makes a linear interpolation.

interval

number

 

The time interval defining the decay. It must be greater than initWindow.

initWindow

number

Yes

The time window of required values for initialization.

Defaults to 0.

cuttof

number

Yes

Minimal value for any dimension. If value of certain dimension falls bellow this value, the dimension is pruned from average.

Defaults to 0.001.

inner
StreamAggrFeatureSpace  module:qmStreamAggr

This stream aggregator creates the feature space and stores the specified features of the last input. It implements the following methods:
1. module:qm.StreamAggr#getFloatVector returns the dense feature vectors.
2. module:qm.StreamAggr#getFeatureSpace returns the feature space.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store named Cars with 4 fields
var base = new qm.Base({
    mode: 'createClean',
    schema: [{
        name: 'Cars',
        fields: [
            { name: 'NumberOfCars', type: 'float' },
            { name: 'Temperature', type: 'float' },
            { name: 'Precipitation', type: 'float' },
            { name: 'Time', type: 'datetime' }
        ]
    }]
});
// create the store
var store = base.store('Cars');
// define a feature space aggregator on the Cars store which needs at least 2 records to be initialized. Use three of the
// four fields of the store to create feature vectors with normalized values.
var aggr = {
   name: "ftrSpaceAggr",
   type: "featureSpace",
   initCount: 2,
   update: true, full: false, sparse: true,
   featureSpace: [
       { type: "numeric", source: "Cars", field: "NumberOfCars", normalize: "var" },
       { type: "numeric", source: "Cars", field: "Temperature", normalize: "var" },
       { type: "numeric", source: "Cars", field: "Precipitation", normalize: "var" }
   ]
};
//create the feature space aggregator
var ftrSpaceAggr = base.store('Cars').addStreamAggr(aggr);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name for the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'featureSpace'.

store

string

 

The name of the store from which it takes the data.

initCount

number

 

The number of records needed before it initializes.

update

boolean

 

If true, updates the feature space.

full

boolean

 

If true, saves the full vector of features.

sparse

boolean

 

If true, saves the sparse vector of features.

FeatureSpace

Array of module:qm~FeatureExtractor

 

Array of feature extractors.

inner
StreamAggrHistogram  module:qm.StreamAggr

This stream aggregator represents an online histogram. It can connect to a buffered aggregate (such as module:qm~StreamAggrTimeSeriesWindow) or a time series (such as module:qm~StreamAggregateEMA). The aggregate defines an ordered set of points p(0), ..., p(n) that define n bins. Infinites at both ends are allowed. A new measurement is tested for inclusion in the left-closed right-opened intervals [p(i), p(i+1)) and the corresponding bin counter is increased for the appropriate bin (or decreased if the point is outgoing from the buffer). It implements the following methods:
1. module:qm.StreamAggr#getFloatLength returns the number of bins.
2. module:qm.StreamAggr#getFloatAt returns the count for a bin index.
3. module:qm.StreamAggr#getFloatVector returns the vector of counts, the length is equal to the number of bins.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Heat",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});

// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 1 day.
var timeser = {
   name: 'TimeSeriesBuffer',
   type: 'timeSeriesWinBuf',
   store: 'Heat',
   timestamp: 'Time',
   value: 'Celsius',
   winsize: 86400000 // one day in miliseconds
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);

// add a histogram aggregator, that is connected with the 'TimeSeriesAggr' aggregator
var aggrJson = {
   name: 'Histogram',
   type: 'onlineHistogram',
   store: 'Heat',
   inAggr: 'TimeSeriesBuffer',
   lowerBound: 0,
   upperBound: 10,
   bins: 5,
   addNegInf: false,
   addPosInf: false
};
var hist = base.store("Heat").addStreamAggr(aggrJson);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name of the stream aggregator.

type

string

 

The type for the stream aggregator. Important: It must be equal to 'onlineHistogram'.

store

string

 

The name of the store from which it takes the data.

inAggr

string

 

The name of the stream aggregator to which it connects and gets data.

lowerBound

number

 

The lowest non-infinite bin point.

upperBound

number

 

The highest non-infinite bin point.

bins

number

Yes

The number of bins bounded by lowerBound and upperBound.

Defaults to 5.

addNegInf

boolean

Yes

Include a bin [-Inf, lowerBound].

Defaults to false.

addPosInf

boolean

Yes

Include a bin [upperBound, Inf].

Defaults to false.

autoResize

boolean

Yes

The histogram will be empty at the beginning and double its size on demand (resize only when incrementing counts). The unbounded bins are guaranteed to stay between lowerBound and upperBound and in all cases the bin size equals (upperBound - lowerBound)/bins.

Defaults to false.

inner
StreamAggrMax  module:qm.StreamAggr

This stream aggregator represents the maximum moving window buffer. It monitors the maximal value in the connected stream aggregator. It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the maximal value of the records in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Heat",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});

// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 1 day.
var timeser = {
   name: 'TimeSeriesAggr',
   type: 'timeSeriesWinBuf',
   store: 'Heat',
   timestamp: 'Time',
   value: 'Celsius',
   winsize: 86400000 // one day in miliseconds
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);

// add a max aggregator, that is connected with the 'TimeSeriesAggr' aggregator
var max = {
   name: 'MaxAggr',
   type: 'winBufMax',
   store: 'Heat',
   inAggr: 'TimeSeriesAggr'
};
var maximal = base.store("Heat").addStreamAggr(max);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name of the stream aggregator.

type

string

 

The type for the stream aggregator. Important: It must be equal to 'winBufMax'.

store

string

 

The name of the store from which it takes the data.

inAggr

string

 

The name of the stream aggregator to which it connects and gets data.

inner
StreamAggrMerger  module:qm.StreamAggr

This stream aggregator represents the merger aggregator. It merges records from two or more stores into a new store depending on the timestamp. No methods are implemented for this aggregator.

Merger Animation

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Cars",
       fields: [
           { name: "NumberOfCars", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   },
   {
       name: "Temperature",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   },
   {
       name: "Merged",
       fields: [
           { name: "NumberOfCars", type: "float" },
           { name: "Celsius", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});
// create a new merger stream aggregator that mergers the records of the 'Cars' and 'Temperature' stores.
// The records are interpolated linearly and stored in the 'Merged' store.
var mer = {
   name: 'MergerAggr',
   type: 'merger',
   outStore: 'Merged',
   createStore: false,
   timestamp: 'Time',
   fields: [
       { source: 'Cars', inField: 'NumberOfCars', outField: 'NumberOfCars', interpolation: 'linear', timestamp: 'Time' },
       { source: 'Temperature', inField: 'Celsius', outField: 'Celsius', interpolation: 'linear', timestamp: 'Time' }
   ]
};
var merger = new qm.StreamAggr(base, mer);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name for the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'merger'.

outStore

string

 

The name of the store where it saves the merged records.

createStore

boolean

 

If the outStore must be created.

timestamp

string

 

The store field of outStore, where the timestamp is saved.

fields

Array of Object

 

An array of field objects. The field object contain the properties:
field.source - The name of the store, from which it takes the values. Type string.
field.inField - The field name of source, from which it takes the values. Type string.
field.outField - The field name of outStore, into which it saves the values. Type string.
field.interpolation - The type of the interpolation. The options are: 'previous', 'next' and 'linear'. Type string.
field.timestamp - The field name of source, where the timestamp is saved. Type string.

inner
StreamAggrMin  module:qm.StreamAggr

This stream aggregator represents the minimum moving window buffer. It monitors the minimal value in the connected stream aggregator. It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the minimal value of the records in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Heat",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});

// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 1 day.
var timeser = {
   name: 'TimeSeriesAggr',
   type: 'timeSeriesWinBuf',
   store: 'Heat',
   timestamp: 'Time',
   value: 'Celsius',
   winsize: 86400000 // 1 day in miliseconds
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);

// add a min aggregator, that is connected with the 'TimeSeriesAggr' aggregator
var min = {
   name: 'MinAggr',
   type: 'winBufMin',
   store: 'Heat',
   inAggr: 'TimeSeriesAggr'
};
var minimal = base.store("Heat").addStreamAggr(min);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name of the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'winBufMin'.

store

string

 

The name of the store from which it takes the data.

inAggr

string

 

The name of the stream aggregator to which it connects and gets data.

inner
StreamAggrMovingAverage  module:qmStreamAggr

This stream aggregator represents the moving average window buffer. It calculates the moving average value of the connected stream aggregator values. It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the average of the values in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Heat",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});

// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window should be 1 day.
var timeser = {
   name: 'TimeSeriesAggr',
   type: 'timeSeriesWinBuf',
   store: 'Heat',
   timestamp: 'Time',
   value: 'Celsius',
   winsize: 86400000
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);

// add a moving average aggregator, that is connected with the 'TimeSeriesAggr' aggregator
var ma = {
   name: 'movingAverageAggr',
   type: 'ma',
   store: 'Heat',
   inAggr: 'TimeSeriesAggr'
};
var movingAverage = base.store("Heat").addStreamAggr(ma);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name of the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'ma'.

store

string

 

The name of the store from which it takes the data.

inAggr

string

 

The name of the stream aggregator to which it connects and gets data.

inner
StreamAggrMovingCorrelation  module:qm.StreamAggr

This stream aggregator represents the moving covariance window buffer. It calculates the moving correlation of the three stream aggregators, that it's connected to. It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the correlation of the values in it's buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in it's buffer window.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Heat",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "WaterConsumption", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});

// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 1 day
var Celsius = {
   name: 'CelsiusAggr',
   type: 'timeSeriesWinBuf',
   store: 'Heat',
   timestamp: 'Time',
   value: 'Celsius',
   winsize: 86400000
}; base.store("Heat").addStreamAggr(Celsius);

// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'WaterConsumption' field
// and the timestamp from the 'Time' field. The size of the window is 1 day
var water = {
   name: 'WaterAggr',
   type: 'timeSeriesWinBuf',
   store: 'Heat',
   timestamp: 'Time',
   value: 'WaterConsumption',
   winsize: 86400000
}; base.store("Heat").addStreamAggr(water);

// add a covariance aggregator, that is connected with the 'CelsiusAggr' and 'WaterAggr' aggregators
var covariance = {
   name: 'covarianceAggr',
   type: 'covariance',
   store: 'Heat',
   inAggrX: 'CelsiusAggr',
   inAggrY: 'WaterAggr'
}; base.store("Heat").addStreamAggr(covariance);

// add the two variance aggregators, that take from the 'Celsius' and 'WaterConsumption' fields, respectively
var celVar = {
   name: 'CelsiusVarAggr',
   type: 'variance',
   store: 'Heat',
   inAggr: 'CelsiusAggr'
}; base.store("Heat").addStreamAggr(celVar);

var waterVar = {
   name: 'waterVarAggr',
   type: 'variance',
   store: 'Heat',
   inAggr: 'WaterAggr'
}; base.store("Heat").addStreamAggr(waterVar);

// add a correlation aggregator, that is connected to 'CovarianceAggr', 'CelsiusVarAggr' and 'WaterValAggr' aggregators
var corr = {
   name: 'corrAggr',
   type: 'correlation',
   store: 'Heat',
   inAggrCov: 'covarianceAggr',
   inAggrVarX: 'CelsiusVarAggr',
   inAggrVarY: 'waterVarAggr'
};
var correlation = base.store("Heat").addStreamAggr(corr);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name for the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'correlation'.

store

string

 

The name of the store from which it takes the data.

inAggrCov

string

 

The name of the covariance stream aggregator.

inAggrVarX

string

 

The name of the first variance stream aggregator.

inAggrVarY

string

 

The name of the second variance stream aggregator.

inner
StreamAggrMovingCovariance  module:qm.StreamAggr

This stream aggregator represents the moving covariance window buffer. It calculates the moving covariance of the two stream aggregators, that it's connected to. It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the covariance of the values in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Heat",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "WaterConsumption", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});

// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 1 day.
var Celsius = {
   name: 'CelsiusAggr',
   type: 'timeSeriesWinBuf',
   store: 'Heat',
   timestamp: 'Time',
   value: 'Celsius',
   winsize: 86400000
}; base.store("Heat").addStreamAggr(Celsius);

// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'WaterConsumption' field
// and the timestamp from the 'Time' field. The size of the window is 1 day.
var water = {
   name: 'WaterAggr',
   type: 'timeSeriesWinBuf',
   store: 'Heat',
   timestamp: 'Time',
   value: 'WaterConsumption',
   winsize: 86400000
}; base.store("Heat").addStreamAggr(water);

// add a covariance aggregator, that is connected with the 'CelsiusAggr' and 'WaterAggr' stream aggregators
var covariance = {
   name: 'covAggr',
   type: 'covariance',
   store: 'Heat',
   inAggrX: 'CelsiusAggr',
   inAggrY: 'WaterAggr'
};
var covarianceAggr = base.store("Heat").addStreamAggr(covariance);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name for the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'covariance'.

store

string

 

The name of the store from which it takes the data.

inAggrX

string

 

The name of the first stream aggregator to which it connects and gets data.

inAggrY

string

 

The name of the recond stream aggregator to which it connects and gets data.

inner
StreamAggrMovingVariance  module:qm.StreamAggr

This stream aggregator represents the moving variance window buffer. It calculates the moving variance of the stream aggregator, that it's connected to. It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the variance of the values in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Heat",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});

// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 1 day
var timeser = {
   name: 'TimeSeriesAggr',
   type: 'timeSeriesWinBuf',
   store: 'Heat',
   timestamp: 'Time',
   value: 'Celsius',
   winsize: 86400000
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);

// add a variance aggregator, that is connected with the 'TimeSeriesAggr' aggregator
var variance = {
   name: 'varAggr',
   type: 'variance',
   store: 'Heat',
   inAggr: 'TimeSeriesAggr'
};
var varianceAggr = base.store("Heat").addStreamAggr(variance);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name for the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'variance'.

store

string

 

The name of the store from which it takes the data.

inAggr

string

 

The name of the stream aggregator to which it connects and gets data.

inner
StreamAggrRecordBuffer  module:qm.StreamAggr

This stream aggregator represents record buffer. It stores the values inside a moving window. It implements all the stream aggregate methods except module:qm.StreamAggr#getFloat and module:qm.StreamAggr#getTimestamp.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Heat",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});

// create a new time series stream aggregator for the 'Heat' store. The size of the window is 3 records.
var aggr = {
   name: 'Delay',
   type: 'recordBuffer',
   size: 3
};
base.store("Heat").addStreamAggr(aggr);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name of the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'recordBuffer'.

size

number

 

The size of the window.

inner
StreamAggrRecordSwitch  module:qm.StreamAggr

This stream aggregate enables switching control flow between stream aggregates based on string keys. It is based on a hash table from keys (strings read from records) to stream aggregates. When OnAdd of a record switch is called, the aggregate looks for an appropriate target aggregate and triggers its onAdd. The aggregate exposes the following methods.
1. module:qm.StreamAggr#getInteger takes a string input and returns returns 1 if the string is a known key and null if it's unknown.
2. module:qm.StreamAggr#onAdd reads the appropriate field as a key and triggers the onAdd of a target aggregate if the key is known.
3. module:qm.StreamAggr#getParams returns a parameter object.
4. module:qm.StreamAggr#setParams used primarily for adding (using $add) new targets or seting (using $set) the internal hashmap with new targets.

Example

// main library
var qm = require('qminer');

// create a store
var base = new qm.Base({
    mode: 'createClean',
    schema: [{
        name: 'testStore',
        fields: [
            { name: 'switchField', type: 'string' }
        ]
    }]
});
// select store
var store = base.store('testStore');

// first JS aggregate
var outAggr1 = new qm.StreamAggr(base, new function () {
    this.onAdd = function (rec) {
        console.log('first');
    }
});

// second JS aggregate
var outAggr2 = new qm.StreamAggr(base, new function () {
    this.onAdd = function (rec) {
        console.log('second');
    }
});

// switcher aggregate: calls outAggr1 when rec.switchField == 'a' and outAggr2 when rec.switchField == 'b'
var switcher = store.addStreamAggr({
    type: 'recordSwitchAggr',
    store: 'testStore',
    fieldName: 'switchField',
    $set: [{ key: 'a', aggrName: outAggr1.name },
           { key: 'b', aggrName: outAggr2.name }],
    throwMissing: false
});

store.push({ switchField: 'a' });
// outAggr1 prints `first`
store.push({ switchField: 'b' });
// outAggr2 prints `second`
store.push({ switchField: 'b' });
// outAggr2 prints `second`
store.push({ switchField: 'c' });
// nothing happens
store.push({ switchField: 'a' });
// outAggr1 prints `first`

// clean up
base.close();

Properties

Name Type Optional Description

name

string

Yes

The given name of the stream aggregator (autogenerated by default).

type

string

 

The type for the stream aggregator. Important: It must be equal to 'recordSwitchAggr'.

store

string

 

The name of the store consistent with the records that it processes.

fieldName

string

 

The name of the field whose values are used for switching

throwMissing

boolean

Yes

If true, the aggregate will throw an exception when the record's key is not recognized (no appropirate target aggregate exists).

Defaults to false.

$set

Array of Object

Yes

An array with objects like {'key': string, 'aggrName': string}. Each object is a switch key and the name of the target aggregate.

inner
StreamAggrResampler  module:qm.StreamAggr

This stream aggregator represents the resampler window buffer. It creates new values that are interpolated by using the values from an existing store. No methods are implemented for this aggregator.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Heat",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   },
   {
       name: "interpolatedValues",
       fields: [
           { name: "Value", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});
// create a new resampler stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The interpolated values are stored in the 'interpolatedValues' store.
// The interpolation should be linear and the interval should be 2 seconds
var res = {
   name: 'resamplerAggr',
   type: 'resampler',
   store: 'Heat',
   outStore: 'interpolatedValues',
   timestamp: 'Time',
   fields: [{
       name: 'Celsius',
       interpolator: 'linear'
   }],
   createStore: false,
   interval: 2000
};
var resampler = base.store("Heat").addStreamAggr(res);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name for the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'resampler'.

store

string

 

The name of the store from which it takes the data.

outStore

string

 

The store in which the samples are stored.

timestamp

string

 

The store field from which it takes the timestamps.

fields

Array of Object

 

The array off field objects from which it takes the values. The field object contain the properties:
field.name - The store field from which it takes the values. Type string.
field.interpolator - The type of the interpolation. The options are 'previous', 'next' and 'linear'. Type string.

createStore

boolean

 

If true, outStore must be created.

interval

number

 

The interval size. The frequency on which it makes the interpolated values.

inner
StreamAggrSimpleLinearRegression  module:qm.StreamAggr

This stream aggregator computes a simple linear regression given two stream aggregates that support the module:qm.StreamAggr#getFloatVector method and represent variates (input) and covariates (output). Optionally the aggregate computes quantile bands: for each quantile q a parallel line to the fitted line is found, so that q fraction of (x,y) datapoints fall below the line. For example, under Gaussian noise, if Y = a + k X + N(0,sig) then the 0.95 quantile band will equal a + 2 sig. This means that 95% of (x,y) pairs lie below the line Y = a + 2 sig + k X. The results are returned as a JSON by calling module:qm.StreamAggr#saveJson and are of type module:qm~StreamAggrSimpleLinearRegressionResult.

Example

// import the qm module
var qm = require('qminer');
// create a base with X,Y measurements
var base = new qm.Base({
    mode: 'createClean',
    schema: [{
        name: 'Function',
        fields: [
            { name: 'Time', type: 'datetime' },
            { name: 'X', type: 'float' },
            { name: 'Y', type: 'float' }
        ]
    }]
});

var store = base.store('Function');

// 1 second buffer for X values
var winX = store.addStreamAggr({
    type: 'timeSeriesWinBuf',
    timestamp: 'Time',
    value: 'X',
    winsize: 1000
});
// 1 second buffer for Y values
var winY = store.addStreamAggr({
    type: 'timeSeriesWinBuf',
    timestamp: 'Time',
    value: 'Y',
    winsize: 1000
});

// the will find regression line, as well as two parallel quartile lines
var linReg = store.addStreamAggr({
    type: 'simpleLinearRegression',
    inAggrX: winX.name,
    inAggrY: winY.name,
    storeX: "Function",
    storeY: "Function",
    quantiles: [0.25, 0.75]
});

store.push({ Time: '2015-06-10T14:13:32.001', X: 0, Y: -2 });
store.push({ Time: '2015-06-10T14:13:32.002', X: 0, Y: -1 });
store.push({ Time: '2015-06-10T14:13:32.003', X: 0, Y: 1 });
store.push({ Time: '2015-06-10T14:13:32.004', X: 0, Y: 2 });
store.push({ Time: '2015-06-10T14:13:32.005', X: 1, Y: -1 });
store.push({ Time: '2015-06-10T14:13:32.006', X: 1, Y: -0 });
store.push({ Time: '2015-06-10T14:13:32.007', X: 1, Y: 2 });
store.push({ Time: '2015-06-10T14:13:32.008', X: 1, Y: 3 });

var res = linReg.saveJson();
res.bands[0]; // -1.5
res.bands[1]; // 1.5

base.close();

Properties

Name Type Optional Description

name

string

 

The given name of the stream aggregator.

type

string

 

The type for the stream aggregator. Important: It must be equal to 'onlineVecDiff'.

storeX

string

 

The name of the store from which it takes the data for the first vector.

storeY

string

 

The name of the store from which it takes the data for the second vector.

inAggrX

string

 

The name of the first stream aggregator to which it connects and gets data.

inAggrY

string

 

The name of the second stream aggregator to which it connects and gets data.

quantiles

Array of number

 

An array of numbers between 0 and 1 for which the quantile bands will be computed.

inner
StreamAggrSimpleLinearRegressionResult  Object

Simple linear regression result JSON returned by module:qm~StreamAggrSimpleLinearRegression.

Properties

Name Type Optional Description

intercept

number

 

Regressor intercept.

slope

number

 

Regressor slope.

quantiles

Array of number

Yes

Quantiles for which bands will be computed.

bands

Array of number

Yes

Computed band intercepts.

inner
StreamAggrSlottedHistogram  module:qm.StreamAggr

This stream aggregator represents an online slotted histogram. It can connect to a buffered aggregate (such as module:qm~StreamAggrTimeSeriesWindow) or a time series (such as module:qm~StreamAggregateEMA). It maps historical values into single period (e.g. into hours of the week). The aggregate defines an ordered set of points p(0), ..., p(n) that define n bins. Infinites at both ends are NOT allowed. A new measurement is tested for inclusion in the left-closed right-opened intervals [p(i), p(i+1)) and the corresponding bin counter is increased for the appropriate bin (or decreased if the point is outgoing from the buffer). It implements the following methods:
1. module:qm.StreamAggr#getFloatLength returns the number of bins.
2. module:qm.StreamAggr#getFloatAt returns the count for a bin index.
3. module:qm.StreamAggr#getFloatVector returns the vector of counts, the length is equal to the number of bins.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Heat",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});

// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 4 weeks
var timeser = {
   name: 'TimeSeriesBuffer',
   type: 'timeSeriesWinBuf',
   store: 'Heat',
   timestamp: 'Time',
   value: 'Celsius',
   winsize: 2419200000 // 4 weeks
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);

// add a slotted-histogram aggregator, that is connected with the 'TimeSeriesAggr' aggregator
// it will present accumulated histogram for the last 2 hours (window) of the week (period) for the last 4 weeks (see aggregate above)
var aggrJson = {
   name: 'Histogram',
   type: 'onlineSlottedHistogram',
   store: 'Heat',
   inAggr: 'TimeSeriesBuffer',
   period: 604800000, // 1 week
   window: 7200000, // 2h
   bins: 5, // 5 possible clusters
   granularity: 300000  // 5 min
};
var hist = base.store("Heat").addStreamAggr(aggrJson);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name of the stream aggregator.

type

string

 

The type for the stream aggregator. Important: It must be equal to 'onlineHistogram'.

store

string

 

The name of the store from which it takes the data.

inAggr

string

 

The name of the stream aggregator to which it connects and gets data.

period

number

 

Cycle length in miliseconds.

window

number

 

Window length that is reported when aggregate is queried.

bins

number

 

The number of bins - input data is expected to be withing interval [0, bins-1].

granularity

number

 

Storage granularity in miliseconds. History is stored in slots with this length. Number of slots is equal to period/granularity.

inner
StreamAggrSparseVecSum  module:qm.StreamAggr

This stream aggregator represents the sparse-vector-sum moving window buffer. It sums all the sparse-vector values, that are in the connected stream aggregator. It implements the following methods:
1. module:qm.StreamAggr#getValueVector returns the sum of the values of the records in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.

Example

var qm = require('qminer');
var base = new qm.Base({
  mode: 'createClean',
  schema: [{
      name: 'Docs',
      fields: [
          { name: 'Time', type: 'datetime' },
          { name: 'Text', type: 'string' }
      ]
  }]
});
var store = base.store('Docs');

var aggr = {
  name: 'featureSpaceWindow',
  type: 'timeSeriesWinBufFeatureSpace',
  store: 'Docs',
  timestamp: 'Time',
  featureSpace: {
      type: "categorical",
      source: "Docs",
      field: "Text"
  },
  winsize: 1000
};
var sa = store.addStreamAggr(aggr);

var aggr2 = {
  name: 'sparseVectorSum',
  type: 'winBufSpVecSum',
  store: 'Docs',
  inAggr: 'featureSpaceWindow'
};
var sa2 = store.addStreamAggr(aggr2);

store.push({ Time: '2015-06-10T14:13:32.0', Text: 'a' }); // 0
store.push({ Time: '2015-06-10T14:13:33.0', Text: 'b' }); // 1
store.push({ Time: '2015-06-10T14:14:34.0', Text: 'c' }); // 2
store.push({ Time: '2015-06-10T14:15:35.0', Text: 'd' }); // 3
store.push({ Time: '2015-06-10T14:15:36.0', Text: 'e' }); // 4
store.push({ Time: '2015-06-10T14:15:37.0', Text: 'f' }); // 5

var valVec2 = sa2.getValueVector(); // [0, 0, 0, 0, 1, 1] - only vectors 4 and 5 remain in window

base.close();

Properties

Name Type Optional Description

name

string

 

The given name of the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'winBufSpVecSum'.

store

string

 

The name of the store from which it takes the data.

inAggr

string

 

The name of the stream aggregator to which it connects and gets data.

inner
StreamAggrSum  module:qm.StreamAggr

This stream aggregator represents the sum moving window buffer. It sums all the values, that are in the connected stream aggregator. It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the sum of the values of the records in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Income",
       fields: [
           { name: "Amount", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});

// create a new time series stream aggregator for the 'Income' store, that takes the values from the 'Amount' field
// and the timestamp from the 'Time' field. The size of the window should 1 week.
var timeser = {
   name: 'TimeSeriesAggr',
   type: 'timeSeriesWinBuf',
   store: 'Income',
   timestamp: 'Time',
   value: 'Amount',
   winsize: 604800000 // 7 days in miliseconds
};
var timeSeries = base.store("Income").addStreamAggr(timeser);

// add a sum aggregator, that is connected with the 'TimeSeriesAggr' aggregator
var sum = {
   name: 'SumAggr',
   type: 'winBufSum',
   store: 'Heat',
   inAggr: 'TimeSeriesAggr'
};
var sumAggr = base.store("Income").addStreamAggr(sum);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name of the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'winBufSum'.

store

string

 

The name of the store from which it takes the data.

inAggr

string

 

The name of the stream aggregator to which it connects and gets data.

inner
StreamAggrTDigest  module:qm.StreamAggr

This stream aggregator computes the quantile estimators using the TDigest algorithm. The quantile values are returned using module:qm.StreamAggr#getFloatVector.

Example

// import the qm module
var qm = require('qminer');
// create a base with the Time and Value fields
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Processor",
       fields: [
           { name: "Value", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});
var store = base.store('Processor');

// create a new time series stream aggregator for the 'Processor' store, that takes the value of the processor
// and the timestamp from the 'Time' field. The size of the window is 1 second.
var tick = {
    name: 'TickAggr',
    type: 'timeSeriesTick',
    store: 'Processor',
    timestamp: 'Time',
    value: 'Value',
    winsize: 1000 // one day in miliseconds
};
var timeSeries = store.addStreamAggr(tick);

// create the TDigest stream aggregator
var aggr = {
    name: 'TDigest',
    type: 'tdigest',
    store: 'Processor',
    inAggr: 'TickAggr',
    quantiles: [0.90, 0.95, 0.99, 0.999],
    minCount: 5
};
// add the stream aggregator to the 'Processor' store
var td = store.addStreamAggr(aggr);
store.push({ Time: '2015-12-01T14:20:32.0', Value: 0.9948628368 });
store.push({ Time: '2015-12-01T14:20:33.0', Value: 0.1077458826 });
store.push({ Time: '2015-12-01T14:20:34.0', Value: 0.9855685823 });
store.push({ Time: '2015-12-01T14:20:35.0', Value: 0.7796449082 });
// with this record the aggregator will initialize becuase it is the 5th record
store.push({ Time: '2015-12-01T14:20:36.0', Value: 0.0844943286 });

store.push({ Time: '2015-12-01T14:20:37.0', Value: 0.187490856 });
store.push({ Time: '2015-12-01T14:20:38.0', Value: 0.0779815107 });
store.push({ Time: '2015-12-01T14:20:39.0', Value: 0.8945312691 });
store.push({ Time: '2015-12-01T14:20:40.0', Value: 0.5574567409 });

// get the quantile estimations
var result = td.getFloatVector();
base.close();

Properties

Name Type Optional Description

name

string

 

The given name of the stream aggregator.

type

string

 

The type for the stream aggregator. Important: It must be equal to 'tdigest'.

store

string

 

The name of the store from which it takes the data.

inAggr

string

 

The name of the stream aggregator to which it connects and gets data.

quantiles

Array of number

 

An array of numbers between 0 and 1 for which the quantile bands will be computed.

minCount

Number

 

The minimal number of values given before it start to compute the quantiles.

inner
StreamAggrThreshold  module:qmStreamAggr

This stream aggregator represents a threshold indicator. It outputs 1 if the current value in the data streams is above the threshold and 0 otherwise. It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the exponentional average of the values in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Heat",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});

// create a new time series tick stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window should be 1 hour.
var timeser = {
   name: 'TimeSeriesTickAggr',
   type: 'timeSeriesTick',
   store: 'Heat',
   timestamp: 'Time',
   value: 'Celsius'
};
var timeSeries = base.store("Heat").addStreamAggr(timeser);

// a threshold aggregator, that is connected wit hthe 'TimeSeriesAggr' aggregator.
// It should output 1 when the temperature is over 3 degrees Celsius and 0 otherwise
var thresholdAggr = {
   name: 'thresholdAggr1',
   type: 'threshold',
   store: 'Heat',
   inAggr: 'TimeSeriesTickAggr',
   threshold: 3
};
var expoMovingAverage = base.store("Heat").addStreamAggr(thresholdAggr);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name for the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'ema'.

store

string

 

The name of the store from which it takes the data.

inAggr

string

 

The name of the stream aggregator to which it connects and gets data. It cannot be connect to the module:qm~StreamAggrTimeSeriesWindow.

threshold

string

 

The threshold mentioned above.

inner
StreamAggrTimeSeriesTick  module:qm.StreamAggr

This stream aggregator represents the time series tick window buffer. It exposes the data to other stream aggregators (similar to module:qm~StreamAggrTimeSeriesWindow). It implements the following methods:
1. module:qm.StreamAggr#getFloat returns the last value added in its buffer window.
2. module:qm.StreamAggr#getTimestamp returns the timestamp of the newest record in its buffer window.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Students",
       fields: [
           { name: "Id", type: "float" },
           { name: "TimeOfGraduation", type: "datetime" }
       ]
   }]
});

// create a new time series tick stream aggregator for the 'Students' store, that takes the values from the 'Id' field
// and the timestamp from the 'TimeOfGraduation' field.
var tick = {
   name: 'TimeSeriesTickAggr',
   type: 'timeSeriesTick',
   store: 'Students',
   timestamp: 'TimeOfGraduation',
   value: 'Id',
};
var timeSeriesTick = base.store("Students").addStreamAggr(tick);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name for the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'timeSeriesTick'.

store

string

 

The name of the store from which it takes the data.

value

string

 

The name of the store field, from which it takes the values.

timestamp

string

 

The name of the store field, from which it takes the timestamp.

inner
StreamAggrTimeSeriesWindow  module:qm.StreamAggr

This stream aggregator represents the time series window buffer. It stores the values inside a moving window. It implements all the stream aggregate methods except module:qm.StreamAggr#getFloat and module:qm.StreamAggr#getTimestamp.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [
   {
       name: "Heat",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});

// create a new time series stream aggregator for the 'Heat' store, that takes the values from the 'Celsius' field
// and the timestamp from the 'Time' field. The size of the window is 2 seconds (2000ms).
var aggr = {
   name: 'TimeSeriesAggr',
   type: 'timeSeriesWinBuf',
   store: 'Heat',
   timestamp: 'Time',
   value: 'Celsius',
   winsize: 2000
};
base.store("Heat").addStreamAggr(aggr);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name of the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'timeSeriesWinBuf'.

store

string

 

The name of the store from which to takes the data.

timestamp

string

 

The field of the store, where it takes the timestamp.

value

string

 

The field of the store, where it takes the values.

winsize

number

 

The size of the window, in milliseconds.

delay

number

 

Delay in milliseconds.

inner
StreamAggrTimeSeriesWindowVector  module:qm.StreamAggr

This stream aggregator represents the values read from a time series window buffer. It implements module:qm.StreamAggr#getFloatVector, module:qm.StreamAggr#getFloatAt and module:qm.StreamAggr#getFloatLength.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
var base = new qm.Base({
   mode: "createClean",
   schema: [{
       name: "Heat",
       fields: [
           { name: "Celsius", type: "float" },
           { name: "Time", type: "datetime" }
       ]
   }]
});

var store = base.store("Heat");
var tick = store.addStreamAggr({
    type: 'timeSeriesTick',
    timestamp: 'Time',
    value: 'Celsius'
});

var winbufvec = store.addStreamAggr({
    type: 'timeSeriesWinBufVector',
    inAggr: tick.name,
    winsize: 2000
});

store.push({ Time: '2015-06-10T14:13:32.0', Celsius: 1 });
winbufvec.getFloatVector().print(); // prints 1
store.push({ Time: '2015-06-10T14:33:30.0', Celsius: 2 });
winbufvec.getFloatVector().print(); // prints 2
store.push({ Time: '2015-06-10T14:33:31.0', Celsius: 3 });
winbufvec.getFloatVector().print(); // prints 2,3
store.push({ Time: '2015-06-10T14:33:32.0', Celsius: 4 });
winbufvec.getFloatVector().print(); // prints 2,3,4

base.close();

Properties

Name Type Optional Description

name

string

 

The given name of the stream aggregator.

type

string

 

The type of the stream aggregator. Important: It must be equal to 'timeSeriesWinBuf'.

store

string

 

The name of the store from which to takes the data.

inAggr

string

 

The name of the window buffer aggregate that represents the input.

inner
StreamAggrVecDiff  module:qm.StreamAggr

This stream aggregator represents difference between two vectors (e.g. online histograms). It implements the following methods:
1. module:qm.StreamAggr#getFloatLength returns the number of bins.
2. module:qm.StreamAggr#getFloatAt returns the count for a bin index.
3. module:qm.StreamAggr#getFloatVector returns the vector of counts, the length is equal to the number of bins.

Example

// import the qm module
var qm = require('qminer');
// create a base with a simple store
// the store records results of clustering
var base = new qm.Base({
mode: "createClean",
schema: [
{
  name: "Rpm",
  fields: [
      { name: "ClusterId", type: "float" },
      { name: "Time", type: "datetime" }
  ]
}]
});

var store = base.store('Rpm');

// create a new time series stream aggregator for the 'Rpm' store that takes the recorded cluster id
// and the timestamp from the 'Time' field. The size of the window is 4 weeks.
var timeser1 = {
  name: 'TimeSeries1',
  type: 'timeSeriesWinBuf',
  store: 'Rpm',
  timestamp: 'Time',
  value: 'ClusterId',
  winsize: 7200000 // 2 hours
};
var timeSeries1 = base.store("Rpm").addStreamAggr(timeser1);

// add a histogram aggregator, that is connected with the 'TimeSeries1' aggregator
var aggrJson1 = {
  name: 'Histogram1',
  type: 'onlineHistogram',
  store: 'Rpm',
  inAggr: 'TimeSeries1',
  lowerBound: 0,
  upperBound: 5,
  bins: 5,
  addNegInf: false,
  addPosInf: false
};
var hist1 = base.store("Rpm").addStreamAggr(aggrJson1);

// create a new time series stream aggregator for the 'Rpm' store that takes the recorded cluster id
// and the timestamp from the 'Time' field.
var timeser2 = {
  name: 'TimeSeries2',
  type: 'timeSeriesWinBuf',
  store: 'Rpm',
  timestamp: 'Time',
  value: 'ClusterId',
  winsize: 21600000 // 6 hours
};
var timeSeries2 = base.store("Rpm").addStreamAggr(timeser2);

// add a histogram aggregator, that is connected with the 'TimeSeries1' aggregator
var aggrJson2 = {
  name: 'Histogram2',
  type: 'onlineHistogram',
  store: 'Rpm',
  inAggr: 'TimeSeries2',
  lowerBound: 0,
  upperBound: 5,
  bins: 5,
  addNegInf: false,
  addPosInf: false
};
var hist2 = base.store("Rpm").addStreamAggr(aggrJson2);

// add diff aggregator that subtracts Histogram1 with 2h window from Histogram2 with 6h window
var aggrJson3 = {
  name: 'DiffAggr',
  type: 'onlineVecDiff',
  storeX: 'Rpm',
  storeY: 'Rpm',
  inAggrX: 'Histogram2',
  inAggrY: 'Histogram1'
}
var diff = store.addStreamAggr(aggrJson3);
base.close();

Properties

Name Type Optional Description

name

string

 

The given name of the stream aggregator.

type

string

 

The type for the stream aggregator. Important: It must be equal to 'onlineVecDiff'.

storeX

string

 

The name of the store from which it takes the data for the first vector.

storeY

string

 

The name of the store from which it takes the data for the second vector.

inAggrX

string

 

The name of the first stream aggregator to which it connects and gets data.

inAggrY

string

 

The name of the second stream aggregator to which it connects and gets data.

inner
StreamAggrWindowQuantiles  module:qm.StreamAggr

This stream aggregate computes approximate quantiles on a sliding time window using the SW-GK algorithm proposed in: http://dl.acm.org/citation.cfm?id=2954329

The quantile values are returned using module:qm.StreamAggr#getFloatVector.

Example

// variables
var batchSize = 1000;
var nbatches = 10;

var dt = 10;
var windowMSec = batchSize*dt;

var quantileEps = 0.01;
var countEps = 0.0001;

var maxRelErr = quantileEps + 2*countEps;

var targetQuants = (function () {
    var quants = [];
    for (var prob = 0; prob <= 1; prob += 0.001) {
        quants.push(prob);
    }
    return quants;
})();

// create a base with a simple store
// the store records results of throwing two independent fair dices
var base = new qm.Base({
    mode: "createClean",
    schema: [
    {
        name: "hugeDie",
        fields: [
            { name: "value", type: "float" },
            { name: "time", type: "datetime" }
        ]
    }]
});
var store = base.store('hugeDie');

// create a new time series stream aggregator for the 'Dice' store, that takes the expected values of throwing a dice
// and the timestamp from the 'Time' field. The size of the window is 1 day.
var windowAggr = store.addStreamAggr({
    name: 'TimeSeries1',
    type: 'timeSeriesWinBuf',
    store: store,
    timestamp: 'time',
    value: 'value',
    winsize: windowMSec
});

var gk = store.addStreamAggr({
type: 'windowQuantiles',
    inAggr: windowAggr,
    quantileEps: quantileEps,
    countEps: countEps,
    quantiles: targetQuants
})

var vals = [];
for (var i = 0; i < batchSize; i++) {
    vals.push(i);
}
for (var batchN = 0; batchN < nbatches; batchN++) {
    // shuffle the array
    for (var i = 0; i < batchSize; i++) {
        var swapN = Math.floor(Math.random()*batchSize);
        var temp = vals[i];
        vals[i] = vals[swapN];
        vals[swapN] = temp;
    }

    for (var i = 0; i < batchSize; i++) {
        var time = (batchN*batchSize + i)*dt;
        store.push({ time: time, value: vals[i] })
    }

    var result = gk.getFloatVector();
    for (var i = 0; i < targetQuants.length; i++) {
        var pval = targetQuants[i];
        var quant_hat = result[i];
        assert(Math.floor((pval - maxRelErr)*batchSize) <= quant_hat);
        assert(Math.ceil((pval + maxRelErr)*batchSize) >= quant_hat);
    }
}

Properties

Name Type Optional Description

name

string

 

The given name of the stream aggregator.

type

string

 

Must use type 'windowQuantiles'.

inAggr

string

 

The name of the stream aggregate which defines the time window.

quantiles

Array of number

 

An array of p-values for which the algorithm will return quantiles.

quantileEps

number

 

Maximal relative error of the quantile estimation procedure.

countEps

number

 

Maximal relative error of the count procedure.

If the number of items in the window is N, then the algorithms error is bound by N(quantileEps + 2countEps + O(countEps^2)). However in practice, the error should be less.