Exploring the Backlog Functionality of the SmartInspect Libraries
This article demonstrates and explains the new backlog functionality and its related features of the SmartInspect libraries introduced in version 2.0. With the new backlog feature the SmartInspect libraries are capable of managing a packet queue and flushing it with a specified trigger. The packet queue is size-limited and automatically removes old packets if needed.
The backlog functionality is a protocol specific feature and can be configured for each used connection in the connections string. Likewise, each protocol can have an associated log level. With this log level you can limit the amount of data being logged by a protocol by excluding packets which don't have a certain minimum log level.
If the backlog option of a protocol is enabled, all packets whose log level is less than a specified trigger level and equal to or higher than the general log level of a protocol, are written to a queue rather than directly to the protocol specific destination. When a packet arrives with a log level of at least the same value as the trigger level, the current content of the queue is written. If the packet queue has been filled up with packets and a new packet is about to be stored, old packets are discarded.
The backlog functionality for a connection can be enabled and configured with the "backlog" and the trigger level with the "flushon" protocol option. If the "backlog" option of a connection is 0 (the default), the backlog functionality is disabled. Otherwise, it specifies the maximum memory size of the packet queue. The "flushon" option can be set to the desired trigger log level. By default it is set to the error log level. Additionally, there is the "keepopen" option which specifies if a connection should always be kept open or only during the actual flush operation. When specifying false for this option (the default), a connection attempt is only initiated when absolutely necessary.
The following examples demonstrate the usage of the backlog feature and its related options in the connections string.
SiAuto.Si.Connections = "file(backlog=2MB, keepopen=true)"; SiAuto.Si.Connections = "file(backlog=512KB, flushon=warning)"; SiAuto.Si.Connections = "tcp(level=message, backlog=2MB, flushon=error)";
The backlog functionality is not tied to a particular protocol. It can be used with all built-in and even custom protocols. Each connection in the connections string which uses the backlog functionality manages its own queue and can have its own set of flushon, keepopen and level options.
There are typical uses for the backlog functionality. Imagine a production server which wants to have logging enabled 24/7 but only needs the logging information after an error occurred. This can partially be achieved with log levels, that means, setting the log level to error and having only errors logged. The problem with this approach is that the context is lost; all other packets with a log level less than error are ignored and it can be difficult to track down the causes for the error.
An elegant solution to this problem is to use the new backlog functionality. By setting the "flushon" trigger option to error and the "backlog" option to a reasonable size, not only the error but also its entire context is logged. This way you minimize the performance costs involved in logging (especially I/O costs) and still get all the information needed to track down, analyze and solve errors.
To summarize this article we can say that the new backlog functionality can be quite helpful. Especially on production systems where the logging performance costs play an important role, the backlog functionality is useful to reduce the logging output to the relevant parts without losing crucial information. For questions about this topic, feel free to contact me via mail at firstname.lastname@example.org.