SysDB — System DataBase

Fix VPATH (out of tree) builds.

Fixed a build issue (configure failure) when using RRDtool 1.5.

Enable silent build rules by default (if supported).

Fix build issues with GCC 5.

Add a new, standalone parser module based on a generic (public) AST (abstract syntax tree). This replaces the (internal) parser of the frontend module.

Improved error reporting for syntactic and semantic errors by propagating error messages in the parser and analyzer.

Fully support array access and comparison (all element types).

Service/metric parent host names are now available via the "hostname" attribute (populated automatically).

Rebase the store query interface on top of the new parser: store implementations prepare actual queries from an AST, thus, allowing multiple implementations of store readers.

Querying the store (through the new interface) will emit data to an arbitrary writer implementation allowing for more flexible post- processing.

Fixed a memory leak in the TIMESERIES implementation.

data: Added support for boolean values; this is a backward incompatible change which breaks the network encoding.

Include nano-second fraction of datatime values (if any) in the string format (including JSON query results).

Add new callback type for store-readers (query plugins) based on the new generic query interface.

memstore: Refactor the in-memory store implementation into a separate data-type. It implements the writer and query/reader APIs.

The main, in-memory store is no longer enabled by default (see store::memory plugin). Instead, all updates go through the plugin API and are distributed to all registered writers.

Handle hostname canonicalization in the generic plugin module instead of requiring each writer plugin to do so.

Do not canonicalize hostnames when fetching time-series.

Determine an object’s backends and update interval in the generic plugin module and pass it on to writer plugins which are expected to merge it with existing entries.

Let the JSON formatter implement the writer API to make it more flexible and uncouple it from the memstore module; let it support arbitrary top-level objects (besides hosts).

FETCH (in the core) now supports all object types and child objects as well as fetching the base object without children / siblings.

Added the SERVER_VERSION command; this allows clients to check for supported servers.

Fixed a memory leak in failed FETCH commands.

Switch to the new, pluggable query interface: use registered plugins to write to or query the SysDB store.

Move protocol-specific (query) logic out of the core and into the frontend (based on the new query API).

A generic TIMESERIES implementation based on the query API replaces the in-core implementation.

Check types of comparison operands and arithmetic expression and treat mismatches as errors; previously those comparisons would fail silently.

Added the NOT IN operator.

Support iterators (ANY/ALL) of arbitrary array expressions.

Change child-object iterator syntax to ANY/ALL <type>.name, that is, require the field name and add support for accessing arbitrary fields.

Added new field "value" for attribute values.

Added new field "timeseries" (boolean) for metrics indicating whether a time-series store is known to SysDB.

Support attribute and backend iterators in FILTER expressions.

Enforce that TIMESERIES end times are greater than start times.

Added unary IS TRUE / IS FALSE operators.

Iterator expressions may now reference a chain of sibling objects as long as only one such reference is an iterator. For example, LOOKUP services MATCHING ANY host.backend =~ 'foo'.

Use check "loop" tests; this will run all test-cases even if there are failures.

Create separate binaries for each test; this allows to run more in parallel.

Make it possible to run unit and integration tests separately; see README.

Added support for C++.

Fixed terminal resets on exit.

Don’t exit if the server closed the connection; instead reconnect automatically on the next user input.

Fixed string parsing: previously, quoted strings were not handled at all and might have been terminated by an embedded semicolon.

Also store failed commands in the history to enable history editing in such cases.

Improved program output: don’t mix log messages with prompts after sending a command; print log messages to the standard error channel.

Fixed lookup of the .sysdb_history file for cases where the SysDB user does not match the current operating system user.

SSL options can be specified inside a <Listen> block.

Unload all plugins on exit to ensure a clean shutdown.

Simplified RPC handling by providing a helper function.

Added support for TCP connections using SSL/TLS and supporting, both, IPv4 and IPv6. Remote peer verification is required for any such connection.

New command line flags -K, -C, and -A to specify the SSL private key, client certificate, and CA certificates file names.

Fully support arrays of all types.

Fixed (implemented) reloading of time-series fetcher plugins.

Added new callback type for "store-writers", making that part of the store pluggable (queries are not supported).

Added support for querying an object’s siblings.

Send all stored objects to store-writer plugins in addition to storing them in the in-memory store.

Fixed parsing of time values.

Add support for storing hosts, services, metrics, and attributes using the new STORE command.

Fixed handling of subsequent requests on the same connection by avoiding unnecessary pauses of up to one second, thus, greatly speeding up request handling.

Added support for TCP connections supporting, both, IPv4 and IPv6. SSL/TLS is enforced for any such connection.

Added support for and require actual client authentication: UNIX socket connections require peer authentication. TCP connections require client SSL certificate authentication.

Added support for expressions accessing an object’s siblings by using the <object_type>.<sibling_type> syntax.

Fixed handling of special characters which are now being escaped.

Include a flag indicating whether time-series data are available for a metric.

os: This utility module provides some Operating System related helper functions: access user information (current user, homedir), manage directories (create, delete, resolve), I/O helpers (select, read, write), networking helpers (DNS resolution).

proto: Added support for marshaling and unmarshaling messages encoding a datum of any type, hosts, services, metrics, and attributes.

ssl: This new module provides convenience functions for SSL/TLS servers, clients, and sessions using OpenSSL.

Specify the object-type by name as first word in the query.

Query by name simply by specifying an (optionally quoted) string which is treated as a regex.

Query by attribute using the <name>:<value> syntax.

Added support for arrays. Elements may be of any other supported data-type but only integers, decimals, and strings are fully supported.

Refactored matchers: all matchers now operate on one or two expressions or other matchers, replacing the old and less flexible conditional matchers.

Attribute values may now be used anywhere in an expression.

On data-type mismatch when comparing attribute values, the values will be cast to string (previously only done for regex matches).

Added support for querying an object’s backends (array of strings).

Added support for iterating child objects and array values.

All matchers now return false if any operand is NULL (except for IS NULL).

All operations now return NULL if any operand is NULL.

Fixed behavior of not-operators (!=, !~) and let them not match if any of the operands is NULL.

Add support for looking up all object types (hosts, services, metrics).

Refactored the JSON formatter to be more flexible and powerful.

Record parent objects for each stored object.

Added a separate parser mode for expressions.

Added a new queryable field name to be used in place of the rather magical keyword host when looking up hosts.

Improved error reporting in the parser and the (newly introduced) analyzer.

Apply filters to hosts in the FETCH command.

All operators may now be used in all places if the involved data-types support the respective operation. For example, regular expressions may be used to match any value. Non-string values are cast to strings for that purpose. IS NULL / IS NOT NULL may be applied to arbitrary expressions.

Introduced the IN operator to check if a value or an array of values is included in an array.

Added support for arrays: [<elem1>,<elem2>,…]

Added support for string and array concatenation using the || operator.

Attribute names have to be strings instead of identifiers now.

Access object fields using <field> rather than .<field>.

Let the LOOKUP and FETCH commands support services and metrics.

Introduced FETCH service|metric <host>.<name for fetching services and metrics.

Added support for ANY and ALL operators which apply a matcher to each value emitted by an iterator (a host’s services and metrics or arrays). MATCHING ANY service|metric|attribute <cmp> <expr> replaces MATCHING service|metric|attribute <cmp> <value> expressions.

Include priority when printing log messages.

Fixed handling of empty queries.

When serializing the entire store (LIST command), return an array of host objects (rather than wrapping it into another object).

Fixed JSON format of time-series data.

Fixed error checks of invalid compare expressions and reject them.

Fixed lookups by back-end name.

Added support for comparing attributes with different types by comparing their string values.

Clarified details about asynchronous messages and authentication options and made client implementation more robust.

Introduced a new message type encoding the JSON serialized response of a query and its data type.

Include the priority in log messages.

Include object types in FETCH, LIST, and LOOKUP messages.

Added support for including services and metrics in LIST responses and skip hosts without the respective children.

Added support for services and metrics to LIST command.

Changed syntax for attribute lookup to attributes[<name>].

Changed syntax for accessing queryable fields to .<field>.

strbuf: Improved memory management.

Added checks for format strings and arguments.

Changed configure option for collectd::unixsock to --enable-collectd-unixsock.

Added support for “metric” objects which may be assigned to a host.

Introduced support for handling time-series data: added new data structures describing time-series data and a new type of plugins called “time-series fetchers” which are used to access time-series information.

Fixed a memory leak.

Apply (very) simple optimizations to lookup expressions.

Added support for metrics and their “data-stores” (describing how to access the actual data references by a metric).

Added support for querying time-series data through the newly introduced “time-series fetcher” plugins.

Fixed a bug that might have caused reading too much data from the connection buffer in certain situations.

The ‘MATCHING’ clause is now optional in ‘LOOKUP’ queries.

The ‘LIST’ and ‘FETCH’ commands support ‘FILTER’ clauses as well.

‘LIST’ / ‘FETCH’ were changed to ‘LIST hosts’ / ‘FETCH host’.

Added the ‘TIMESERIES’ command to query arbitrary time-series which fetches time-series data from a backend’s data-store for a given period of time.

Added support for date/time values.

Added support for single quotes in string values: two adjacent single quotes may be used to specify a single quote.

Documented the frontend protocol in frontend/proto.h.

Documented changes to the query language and new / updated plugins.

Documented all supported data types.

Fixed weird behavior when collector callbacks were registered without any plugin context information (as it would, e.g., happen when using LoadPlugin instead of LoadBackend in sysdbd).

Introduced service attributes — similar to host attributes but assigned to a service object.

Don’t log “value too old” messages if an updated object uses the same timestamp as the old object.

Switched from linked-lists to AVL trees for storing all object types to reduce lookup times from O(n) to O(log n).

Added infrastructure for handling and evaluating arithmetic expressions and filters (see also the changes to the query language below).

Export various fields describing base attributes of all stored object types: last_update, age, interval, and backend. These fields may be used when querying the store.

Added support for ‘IS NULL’ attribute checks matching on non-existent attributes.

Changed ‘LOOKUP … WHERE’ queries to ‘LOOKUP … MATCHING’ to avoid confusion with SQL semantics.

Switched from ‘<obj>.name’ to ‘<obj>’ in MATCHING clauses, thus, no longer shadowing attributes called “name”.

Added support for arithmetic expressions in places that used simple data values before. Arithmetic expressions support addition, subtraction, multiplication, division, modulo, and concatenation operations. The expressions may be based on constant values or queryable fields of the stored objects (specified as ‘:last_update’, ‘:age’, ‘:interval’, and ‘:backend’).

Added ‘FILTER’ support to ‘LOOKUP’ queries. Filters are conditional expressions which may optionally be used to limit the query and the result to object (of any type) matching the filter condition.

Added support to specify date and time values.

avltree: Added an AVL tree (self-balancing binary tree) implementation.

Always log to the standard output stream if no user-provided logging callbacks had been specified. Previously this was disabled after enabling logging to clients in the frontend.

Simplified plugin naming information and determine configuration callback names automatically to ensure better consistency.

Simplified internal data management.

Store the list of backends along with each stored object which provide the respective object.

Simplified and improved the internal architecture matching / looking up objects.

Fixed parser to correctly reject invalid object types.

Automatically create the base directory of a listening UNIX socket.

data collection

hostname canonicalization

logging

generic channels for asynchronous, multi-threaded I/O

DBI wrapper to ease common database access operations

thread-aware error handling and reporting

doubly linked-lists

dynamic sized memory buffers supporting formatted strings and binary

data

UNIX socket client implementation