SmartInspect Log Formats and Protocols
This article describes the SmartInspect log file format and the SmartInspect TCP/IP log protocol. This article is mainly intended for advanced SmartInspect users or third-party vendors who want to build additional tools and libraries around SmartInspect. Normal SmartInspect users generally do not need to know the exact log file format and log protocols to work with SmartInspect.
Please note that this specification is valid for SmartInspect version 1.x, 2.x and higher. Whenever the formats and protocols change, this article will be updated or replaced by a newer version.
We changed the log formats and protocols for SmartInspect 2.x . The following list is a summary of changes in the new version:
- All string fields are now encoded in UTF-8
- New packet IDs are used for compatibility
- Watch packets now contain a timestamp field
- New log entry types have been added
The new log format has been designed to work with current tools and libraries that were based on the 1.x specification. To update any tool and library to work with the current format, the above changes must be incorporated.
Before we can discuss the log file format and the TCP protocol, we need to look at the so-called SmartInspect packets. Both the log files and the TCP protocol use packets to transfer the logging data. There are different kinds of packets that contain log entries, watches and process flow information. The different packet types are discussed below.
A packet consists of a header that specifies the packet type and the size of the packet data, and of a body that contains the actual data of the packet (a log entry or watch, for example).
The table below describes the field types and sizes. Please note that all fields are stored and transferred in little-endian format.
|Packet Type||Unsigned Integer||16 Bit||The type of the packet. Please see below for the supported packet types.|
|Body Size||Signed Integer||32 Bit||The size of the body in bytes.|
|Body||<variable>||<variable>||The actual data of the packet (the log entry data, for example).|
The following packet types are defined and supported by SmartInspect. If a third-party vendor wants to define his own packet types, they should start with 0x8000 (32768).
- 0 = Log Entry
- 1 = Control Command
- 2 = Watch
- 3 = Process Flow
- 4 = Log Entry v2
- 5 = Watch v2
- 6 = Process Flow v2
The v2 packet types were introduced with SmartInspect 2.x. If you only want to support log files and libraries that are based on the 2.x specification, you only have to support these packet types. The main difference is that the new packet types contain UTF-8 encoded strings, whereas the old packet types contain ANSI strings.
If you want to process SmartInspect log files or receive data from a SmartInspect library, you do not need to support all packet types. You can skip packets you don't want to process. You just need to read the packet header and skip as many bytes as specified in the Body Size field.
SmartInspect currently knows four different packet types. As noted above, you don't have to support all packet types if you want to write a tool that reads or receives logging information. The following sections describe the different packet types, structures and formats.
Log entries are the most visible packets. They are displayed in the Views of the SmartInspect Console and are basically the main logging information you process or generate.
There are many different log entry types like messages, warnings and errors. Additionally, every log entry can contain data like files, source code or images. The attached data is displayed in the Viewer Toolbox in the Console and can be further analyzed or saved by the user. The following image shows the structure of a log entry.
A log entry packet consists of three parts: the log entry header, the variable-size log entry strings and the (optional) data part. The following table describes the fields of a log entry.
|Log Entry Type||Signed Integer||32 Bit||The type of the log entry. Please see below for the supported log entry types.|
|Viewer ID||Signed Integer||32 Bit||Specifies the Viewer that is used to display the data part.|
|Application Name Length||Signed Integer||32 Bit||The length of the application name string.|
|Session Length||Signed Integer||32 Bit||The length of the session string.|
|Title Length||Signed Integer||32 Bit||The length of the title string.|
|Hostname Length||Signed Integer||32 Bit||The length of the hostname.|
|Data Size||Signed Integer||32 Bit||The size of the data part in bytes.|
|Process ID||Unsigned Integer||32 Bit||The ID of the process that generated the log entry.|
|Thread ID||Unsigned Integer||32 Bit||The ID of the thread that generated the log entry.|
|Timestamp||Float (Double)||64 Bit||Contains the date and time of the log entry (see below for the format).|
|Color||Signed Integer||32 Bit||The background color of the log entry (see below for the format).|
|Application Name||UTF-8 String (ANSI in v1)||<variable>||The name of the application that generated this log entry.|
|Session||UTF-8 String (ANSI in v1)||<variable>||The name of the session that generated this log entry.|
|Title||UTF-8 String (ANSI in v1)||<variable>||The title of the log entry. This will be displayed in the View of the Console.|
|Hostname||UTF-8 String (ANSI in v1)||<variable>||The hostname of the computer that generated this log entry.|
|Data||<variable>||<variable>||The attached data of a log entry. The format depends on the Viewer.|
Most of the fields are straightforward and do not need any further explanations. However, some fields like the log entry ID, viewer ID, color and timestamp need to be discussed in more detail. This happens in the next sections.
Log Entry Type
The Log Entry Type is used by the Console to allow users to filter for specific log entries. Additionally, each Log Entry Type has a unique icon that is displayed next to all log entries of a type. Please note that the Log Entry Type has no effect on which Viewer is used to display the data.
- 0 = Separator
- 1 = EnterMethod
- 2 = LeaveMethod
- 3 = ResetCallstack
- 100 = Message
- 101 = Warning
- 102 = Error
- 103 = InternalError
- 104 = Comment
- 105 = VariableValue
- 106 = Checkpoint
- 107 = Debug
- 108 = Verbose
- 109 = Fatal (new in v2)
- 110 = Conditional (new in v2)
- 111 = Assert (new in v2)
- 200 = Text
- 201 = Binary
- 202 = Graphic
- 203 = Source
- 204 = Object
- 205 = WebContent
- 206 = System
- 207 = MemoryStatistic
- 208 = DatabaseResult
- 209 = DatabaseStructure
The Viewer ID specifies which Viewer will be used by the Console to display the attached data part of a log entry. The format of the data depends on the chosen Viewer. The following list contains the IDs of all supported Viewers. Please consult the SmartInspect online help for more information about the different Viewers and the Data formats (Topic: Working with SmartInspect » Understanding the Console Viewers). If no data is attached to a log entry, the Title Viewer should be used.
- 0 = Title Viewer
- 1 = Data Viewer
- 2 = List Viewer
- 3 = Value List Viewer
- 4 = Inspector Viewer
- 5 = Table Viewer
- 100 = Web Viewer
- 200 = Binary Viewer
- 300 = HTML Viewer (Source Viewer)
- 302 = VbScript Viewer (Source Viewer)
- 303 = Perl Viewer (Source Viewer)
- 304 = SQL Viewer (Source Viewer)
- 305 = INI File Viewer (Source Viewer)
- 306 = Python Viewer (Source Viewer)
- 307 = XML Viewer (Source Viewer)
- 400 = Bitmap Viewer (Image Viewer)
- 401 = JPEG Viewer (Image Viewer)
- 402 = Icon Viewer (Image Viewer)
- 403 = Metafile Viewer (Image Viewer)
The Timestamp field contains the time and the date of the log entry. This field is a 64 bit float (double). The integral part of the value is the number of days that have passed since 12/30/1899. The fractional part of the value is the fraction of a 24 hour day that has elapsed. This value is compatible with Delphi's native TDateTime type.
The Color field specifies the background color that the Console uses to display the log entry. It consists of a null byte, followed by one byte for each of the RGB colors in the order of blue, green and red. If no background color should be used, specify 0xFF000005 in the Color field. The Color field is compatible with Delphi's native TColor type.
Control commands are currently used to notify the Console to clear all logging information or just specific parts of the current log (only watches or log entries, for example). Additional control commands might be added in the future. The following image describes the structure of a control command packet.
Please note that currently only a few control commands are supported by the Console. None of the existing control commands use the data part at the moment.
|Control Command Type||Signed Integer||32 Bit||The type of the control command. See below for supported types.|
|Data Size||Signed Integer||32 Bit||The size of the data in bytes.|
|Data||<variable>||<variable>||The data of the control command (currently not used).|
The following control commands are defined and supported by the SmartInspect Console. Third-party tools do not need to support all or any of the control command types.
- 0 = Clear all log entries
- 1 = Clear all watches
- 2 = Clear all AutoViews
- 3 = Reset the Console
- 4 = Clear the Process Flow
Watch packets contain a snapshot of a variable and its value. Watch packets are used to monitor and view variable values in the Console. The following image describes the structure of a watch packet.
All watch values are stored as an UTF-8 string representation of the value. This means that numbers and other values have to be converted to an UTF-8 string before it's stored in a watch packet (ANSI in v1).
|Name Length||Signed Integer||32 Bit||The length of the watch name string.|
|Value Length||Signed Integer||32 Bit||The length of the value string.|
|Watch Type||Signed Integer||32 Bit||The type of the watch. See below for a list of supported watch types.|
|Timestamp||Float (Double)||64 Bit||Contains the date and time of the watch (see the log entry timestamp for the format).|
|Name||UTF-8 String (ANSI in v1)||<variable>||The name of the watch.|
|Value||UTF-8 String (ANSI in v1)||<variable>||The UTF-8 string representation of the value (ANSI in v1).|
The following watch types are supported by the Console. Please note that the watch type has no effect on the interpretation of the value field. The value field is always the UTF-8 string representation of the value (ANSI in v1).
- 0 = Character
- 1 = String
- 2 = Integer
- 3 = Float
- 4 = Boolean
- 5 = Memory Address
- 6 = Timestamp
- 7 = Object
The process flow packets are used to track threads, processes and method execution. At the moment, this data is only used for the Process Flow Toolbox and is not used to the full extent possible. It could be used, for example, to develop a profiling tool based on the logging information generated by the SmartInspect libraries. The following image describes the process flow packet structure.
Depending on the process flow type, the title contains the method, thread or process name. Please also note that additional log entry packets are generated for specific log events like Enter Method and Leave Method by the SmartInspect libraries.
|Process Flow Type||Signed Integer||32 Bit||The type of the process flow (see below).|
|Title Length||Signed Integer||32 Bit||The length of the title string.|
|Hostname Length||Signed Integer||32 Bit||The length of the hostname string.|
|Process ID||Unsigned Integer||32 Bit||The ID of the process that generated this packet.|
|Thread ID||Unsigned Integer||32 Bit||The ID of the thread that generated this packet.|
|Timestamp||Float (Double)||64 Bit||Contains the date and time of the packet.|
|Title||UTF-8 String (ANSI in v1)||<variable>||The name of the method, thread or process in question.|
|Hostname||UTF-8 String (ANSI in v1)||<variable>||The name of the computer that generated this packet.|
The timestamp field has the same format as the timestamp field of log entry packets. Please see the section about log entries for the format description.
The following process flow packets are defined and generated by the SmartInspect libraries.
- 0 = Enter Method
- 1 = Leave Method
- 2 = Enter Thread
- 3 = Leave Thread
- 4 = Enter Process
- 5 = Leave Process
SmartInspect Log Files
A SmartInspect log file consists of a header and 0..n SmartInspect packets. The header currently only contains a magic number to make it easier to identify valid SmartInspect log files. The following image shows the structure of a log file.
The magic number of the current SmartInspect version is the ASCII representation of "SILF" (4 bytes, all characters are upper case). Future log file formats might use a different magic number and add a version number to the file format.
A typical log file looks like the following image:
SmartInspect TCP Protocol
The TCP/IP protocol used to send logging information to the Console is simple. A client application or library ("client") connects to the TCP port 4228 (can be changed in the Console) and waits for the Console banner. The Console banner is a line of text that contains the Console name and the version information and is terminated with carriage return and line feed characters (0xD and 0xA).
After the client received the Console banner, the client sends its own banner with its name and version information (this must be terminated with carriage return and line feed characters). This finalizes the so-called initialization stage of the protocol.
After the initialization ended, the client can now send packets to the Console (the packet has the same format as in log files). After the Console has received and processed a packet, it acknowledges the packet by sending the ASCII representation of "OK" back to the client. This is done to make sure that the Console doesn't receive more packets than it is able to process. After the client received the acknowledgement that the packet was received and processed, it can send an additional packet.
Both parties can disconnect at any time. The following image shows the communication between the client and the Console.
The following banners are currently used by the Console and the different libraries (with varying version numbers and platforms, depending on the SmartInspect version and library platform used).
SmartInspect Professional Console v220.127.116.114 SmartInspect Java Library v18.104.22.1684
Third-party developers should use banners like the following examples in new SmartInspect compatible libraries and receiver applications:
ABC Receiver Version 1.0, compatible with SmartInspect v22.214.171.124 ABC Python Library Version 1.0, compatible with SmartInspect v126.96.36.199
After reading this article, you should now be able to implement additional SmartInspect compatible tools and libraries. If you have any questions about SmartInspect or the protocols and file formats published in this article, please let us know at firstname.lastname@example.org.