FieldLog file format specification

FieldLog writes the log items in a binary format that minimises the amount of bytes per information and reuses any previous text strings for storage efficiency. This document describes the file structure for the current and past format versions.

All integer values are stored in big endian format, independent of the underlying hardware architecture.

Each record starts with its total length, so it can be quickly skipped when reading the file. This also works if the file reader does not know the exact internal structure of the record. Record length values are technically 32 bits long, but the first (most-significant) 4 bits are used as the record type indicator. So a record can effectively be no longer than 2²⁸ = 256 MiB. Also, there can only be 2⁴ − 1 = 15 different record types (0 is reserved for the invalid value). The record type values are defined in the source file FieldLog\Enums.cs.

Each log file cannot be larger than 2³¹ = 2 GiB due to the signed Int32 addressing of offsets. FieldLog limits the maxfilesize configuration option to 1 GiB. But normally such large single log files are not practical anyway.

File header

The log file header is read and written in the FieldLogFileWriter constructor and also read in the FieldLogFileReader constructor.

The highest supported file format version, which is always the version for writing new files, is defined in the public FL.FileFormatVersion constant. The file format version of a file being read by the FieldLogFileReader class is available in its public FormatVersion property. If an existing file has a different format version than the current, it cannot be appended to and a new log file will be started.

Length	Data type	Description
9	Text	Constant string "FieldLog\0", ASCII encoded, with final 0 byte.
1	Integer	File format version, currently 2.

File header length: 10 bytes.

After the file header, the data records are stored. A record can be a shared string or a log item.

Shared strings

Shared strings store the actual string data used in log items. Each time a string should be stored, a shared string record is written and the file offset of its beginning is stored as an integer value to reference the string in a log item. If a shared string already exists in the current file, it is reused. Strings cannot be shared across multiple files, so each file remains self-containing. To fetch the string data from a record field, a simple reader could just seek to the indicated position in the file and read the string record from there. The FieldLog library uses a cache to keep string instances that were already read and written.

If a string in a data structure has the value null, it is stored as the offset value -1 (FFFFFFFFh). No shared string record is created for null values, but a record may be created for an empty string.

The text cache is read and written in the FieldLogFileWriter.GetText method and read in the FieldLogFileWriter.ReadTextCache and FieldLogFileReader.ReadLogItem methods.

Length	Data type	Description
4	Integer	Record length, excluding the length value itself. The first 4 bits indicate the string record type (value: 1).
var.	Text	String content, UTF-8 encoded.

Record length: variable (depends on the string length).

Log items

Log items come in different types. The type is indicated in the first 4 bits, as introduced above. All log item records begin with a number of common fields, followed by the type-specific structure.

The log item base data is written in the FieldLogFileWriter.WriteBuffer method and read in the FieldLogFileReader.ReadLogItem method (continued in FieldLogItem.Read). The base data fields are prepared for the buffer in the FieldLogItem.Write method and read in the FieldLogItem.ReadBaseData method.

Length	Data type	Description
4	Integer	Record length, excluding the length value itself. The first 4 bits indicate the string record type (value: > 1).
8	Integer	Ticks of the event time, from `DateTime.Ticks`.
4	Integer	Event counter, sequence number within the session.
1	Integer	Priority, defined in the `FieldLogPriority` enum in the source file FieldLog\Enums.cs.
16	GUID	Application session ID, uniquely identifies each execution of the application.
4	Integer	Managed thread ID from which the log item was created.
4	Integer	Web request ID (unsigned), sequence number for each web request, 0 otherwise. Added in file format version 2.

Record base length: 41 bytes.

Text log item

The type value for exception log items is 2.

The text log item data is written in the FieldLogTextItem.Write method and read in the FieldLogTextItem.Read method.

Length	Data type	Description
4	String offset	Text.
4	String offset	Details, if specified.

Record total length: 49 bytes.

Data log item

The type value for exception log items is 3.

The data log item data is written in the FieldLogDataItem.Write method and read in the FieldLogDataItem.Read method.

Length	Data type	Description
4	String offset	Name of the variable being logged.
4	String offset	Value.

Record total length: 49 bytes.

Exception log item

The type value for exception log items is 4.

The exception log item data is written in the FieldLogExceptionItem.Write method and read in the FieldLogExceptionItem.Read method.

Length	Data type	Description
var.	Structure	Exception data, see below.
4	String offset	Context in which the exception occured, if specified.
138	Structure	Environment data, see below.

Record total length: variable (depends on the number of stack frames and inner exceptions).

Scope log item

The type value for scope log items is 5. If a scope item is repeated in a following log file, it is marked as repeated scope item with the type value 6. Such items are kept available as long as the scope is still relevant, in case a new log file was started in the middle of a scope and older log files (where the scope was originally started) were already purged away.

WebRequestStart type scope items can also be repeated when more data about a request becomes available. They are not an exact copy of the original item but contain additional data in the WebRequestData structure. The latest instance of such an item is the one with the most data and should be considered exclusively, replacing the older instances of the item.

The scope log item data is written in the FieldLogScopeItem.Write method and read in the FieldLogScopeItem.Read method.

Length	Data type	Description
1	Integer	Scope type, defined in the `FieldLogScopeType` enum in the source file FieldLog\Enums.cs.
4	Integer	New scope level after this log item, used for indenting. Only incremented and decremented for `Enter` and `Leave` scopes.
4	String offset	Scope name.
1	Integer	Only for type `ThreadStart` (2): Flags: 1: Background thread. 2: Pool thread.
138	Structure	Only for type `LogStart` (4): Environment data, see below.
44	Structure	Only for type `WebRequestStart` (6): Web request data, see below. Added in file format version 2.

Record total length: variable (depends on the scope type).

Data structures

These data structures are always part of a log item structure. They are referenced from the tables above. Some structures may also contain other instances of themselves to store recursive data.

Exception data structure

The exception data is written in the FieldLogException.Write method and read in the FieldLogException.Read method.

Length	Data type	Description
4	String offset	Exception type name.
4	String offset	Module that defines the exception type. Added in file format version 2.
4	Integer	Metadata token of the exception type. Added in file format version 2.
4	String offset	Exception message.
4	String offset	Error code.
4	String offset	Additional data from the exception object, formatted presentation of the `Data` dictionary and other public properties.
4	Integer	Number of stack frames.
Count * 36	Structure	Stack frames data, see below.
4	Integer	Number of inner exceptions on this level.
Count * var.	Structure	Inner exceptions data (this structure).

Structure length: variable (depends on the number of stack frames and inner exceptions).

The stack frame and inner exception structures are repeated as often as is indicated by the count value in the previous field.

Stack frame structure

The stack frame data is written in the FieldLogStackFrame.Write method and read in the FieldLogStackFrame.Read method.

Length	Data type	Description
4	String offset	Module file name.
4	Integer	Metadata token of the method. Added in file format version 2.
4	Integer	IL offset in the method body. Added in file format version 2.
4	String offset	Type name of the executing method.
4	String offset	Method name.
4	String offset	Method signature.
4	String offset	Source code file name (this is only available if the .pdb file is present at runtime).
4	Integer	Line number in the source code file (this is only available if the .pdb file is present at runtime).
4	Integer	Column number in the source code file (this is only available if the .pdb file is present at runtime).

Structure length: 36 bytes.

Environment data structure

The environment data is written in the FieldLogEventEnvironment.Write method and read in the FieldLogEventEnvironment.Read method.

Length	Data type	Description
1	Integer	OS type, defined in the `OSType` enum in the source file FieldLog\OSInfo.cs.
1	Integer	OS version, defined in the `OSVersion` enum in the source file FieldLog\OSInfo.cs.
1	Integer	OS edition, defined in the `OSEdition` enum in the source file FieldLog\OSInfo.cs.
4	String offset	OS service pack.
4	Integer	OS build number.
4	Integer	OS service pack build number.
4	String offset	OS product name.
4	String offset	OS language culture code.
8	Integer	Last system boot time, from `DateTime.Ticks`.
4	String offset	Application compatibility layer, the entire string as read from the registry. Values explained in this blog post.
4	String offset	CLR type. Either "Microsoft .NET" or "Mono".
1	Integer	Number of mouse buttons.
1	Integer	Number of touch points.
2	Integer	Logical screen resolution in dpi.
4	String offset	Current thread culture code.
4	String offset	Current working directory.
4	String offset	Environment variables.
2	Integer	Number of CPU cores available on the system.
4	String offset	Host name.
4	String offset	User name, including domain.
4	String offset	Application executable file path.
4	String offset	Command line.
4	String offset	Application version.
4	String offset	Application assembly configuration. Added in file format version 2.
4	String offset	CLR version.
2	Integer	Local time zone offset, in minutes from UTC.
8	Integer	Private memory currently used by the process, in bytes.
8	Integer	Peak working set memory used by the process, in bytes.
8	Integer	Total visible memory on the computer, in bytes.
8	Integer	Available memory on the computer, in bytes.
4	Integer	Process ID.
2	Integer	Primary screen width, in pixels.
2	Integer	Primary screen height, in pixels.
1	Integer	Primary screen colour depth, in bits per pixel.
2	Integer	Primary screen working area left start, in pixels.
2	Integer	Primary screen working area top start, in pixels.
2	Integer	Primary screen working area width, in pixels.
2	Integer	Primary screen working area height, in pixels.
1	Integer	Number of screens on the computer.
1	Integer	Flags: 1: OS is 64 bit. 2: OS is application server. 4: OS is booted in fail-safe mode. 8: Session is interactive. 16: Process is 64 bit. 32: User is local administrator.

Structure length: 138 bytes.

Web request data structure

This data structure was added in file format version 2.

The web request data is written in the FieldLogWebRequestData.Write method and read in the FieldLogWebRequestData.Read method.

Length	Data type	Description
4	String offset	Request URL.
4	String offset	HTTP method.
4	String offset	Client network address (IPv4 or IPv6, depending on connection type).
4	String offset	Client host name.
4	String offset	Referrer URL, from the HTTP request header.
4	String offset	User agent, as in the HTTP request header.
4	String offset	Accepted languages, from the HTTP request header.
4	String offset	Accepted content types, from the HTTP request header.
4	String offset	Web session ID.
4	String offset	Application-specific user ID.
4	String offset	Application-specific user name.

Structure length: 44 bytes.