Generic Log Adapter configuration file structure

Adapter Configuration File structure

An adapter configuration file contains a collection of contexts that are to be run by a Generic Log Adapter runtime instance. Each context describes how to process a single log file. Because the adapter configuration file can to contain many of these contexts, the adapter can process many log files at the same time.

Each context associated with a particular log file is constituted of a series of components. Each component has a role to play in processing a log file.

Context
Sensor
Extractor
Parser
Formatter
Outputter

Context

The context describes the ordered grouping of components as they are chained together for log file processing. Each log file has a separate context, and multiple contexts can be defined in a single configuration. The Generic Log Adapter provides one context:

org.eclipse.hyades.logging.adapter.impl.BasicContext - a context which runs each of the components in a single thread.

A context has the following properties:

Property Type Behavior
Description string A text description for the context. This value is used only by the Adapter Configuration Editor.
Unique Id string A unique identifier for this component. The context refers to this string to identify which component instance to use.
Is Continuous Operation boolean Whether or not this log file should be monitored for changes based upon the Maximum Idle Time and Pause Interval properties.
Maximum Idle Time integer (milliseconds) The maximum amount of time the context will wait for a log file to change before shutting itself down.
Pause Interval integer (milliseconds) The amount of time the context waits after reaching the end of the log file before trying to read again.

Property	Type	Behavior
Description	string	A text description for the context. This value is used only by the Adapter Configuration Editor.
Unique Id	string	A unique identifier for this component. The context refers to this string to identify which component instance to use.
Is Continuous Operation	boolean	Whether or not this log file should be monitored for changes based upon the Maximum Idle Time and Pause Interval properties.
Maximum Idle Time	integer (milliseconds)	The maximum amount of time the context will wait for a log file to change before shutting itself down.
Pause Interval	integer (milliseconds)	The amount of time the context waits after reaching the end of the log file before trying to read again.

Sensor

The sensor provides the mechanism to read the content for processing. The Generic Log Adapter provides the following sensors:

SingleFileSensor: A sensor that reads files from a local storage medium.
Executable Class: org.eclipse.hyades.logging.adapter.sensors.SingleOSFileSensor
StaticParserSensor: A sensor that integrates transforms that are performed by Java based implementations. It directly invokes the Java parser and exports the Common Base Event objects that the Java parser produces.
Executable Class: org.eclipse.hyades.logging.adapter.config.sensors.StaticParserSensor
AdapterCBESensor: A sensor that monitors the adapter's execution. It is used internally by Generic Log Adpater to log messages from the Generic Log Adapter itself. All internal logging within the adapter is sent to this sensor.
Executable Class: org.eclipse.hyades.logging.adapter.internal.util.AdapterSensor

The following table lists the attributes that are common among the sensor types.

Sensor attributes

Attribute name Type Behavior
Sensor Type String The sensor type to be used by the adapter.
Description (required) String A text description for the sensor. This value is used only by the Adapter Configuration Editor.
Unique ID (required) String A unique identifier for this component. The sensor refers to this string to identify which component instance to use.
Maximum Blocking (required) Integer The maximum number of lines the sensor reads before forwarding to the extractor for processing. For the StaticParserSensor type, this field indicates the maximum number of Common Base Event objects created by the static parser before forwarding to the outputter for processing.
Confidence Buffer Integer The last n bytes of a file to be saved and used for determining if an append has occured to the file since the last time it was read.
Footer size Integer The n bytes of a file that contains variable content. The footer portion of a file will not be used when determining if an append has occured since the last time the file was read.

Attribute name	Type	Behavior
Sensor Type	String	The sensor type to be used by the adapter.
Description (required)	String	A text description for the sensor. This value is used only by the Adapter Configuration Editor.
Unique ID (required)	String	A unique identifier for this component. The sensor refers to this string to identify which component instance to use.
Maximum Blocking (required)	Integer	The maximum number of lines the sensor reads before forwarding to the extractor for processing. For the StaticParserSensor type, this field indicates the maximum number of Common Base Event objects created by the static parser before forwarding to the outputter for processing.
Confidence Buffer	Integer	The last n bytes of a file to be saved and used for determining if an append has occured to the file since the last time it was read.
Footer size	Integer	The n bytes of a file that contains variable content. The footer portion of a file will not be used when determining if an append has occured since the last time the file was read.

The following table lists the properties that are specific for each type of sensor. The following properties are known by the component code and must be spelled exactly as indicated in the table for them to be recognized and used by the component.

Sensor type Properties Application
SingleFileSensor

directory (required)

The directory location of the file to be read by the SinlgeFileSensor.

fileName (required)

The name of the file to be read by the SingleFileSensor.

shell

The UNIX shell to run the converter command in. Used only on UNIX platforms

converter

A command line for a preprocessing instruction to run before starting the sensor.

Used when creating an adapter for parsing a log file with regular expression rules
StaticParserSensor

directory (required)

The directory location of the file to be read by the StaticParserSensor.

fileName (required)

The name of the file to be read by the StaticParserSensor.

parserClassName (required)

the name of the Java class that implements the static parser. The class must implement the org.eclipse.hyades.logging.parsers.IParser interface.

Used when creating an adapter for parsing a log file with a static parser Java class.

Sensor type	Properties	Application
SingleFileSensor	directory (required) The directory location of the file to be read by the SinlgeFileSensor. fileName (required) The name of the file to be read by the SingleFileSensor. shell The UNIX shell to run the converter command in. Used only on UNIX platforms converter A command line for a preprocessing instruction to run before starting the sensor.	Used when creating an adapter for parsing a log file with regular expression rules
StaticParserSensor	directory (required) The directory location of the file to be read by the StaticParserSensor. fileName (required) The name of the file to be read by the StaticParserSensor. parserClassName (required) the name of the Java class that implements the static parser. The class must implement the org.eclipse.hyades.logging.parsers.IParser interface.	Used when creating an adapter for parsing a log file with a static parser Java class.

Extractor

The extractor takes a collection of input lines provided by the sensor and separates them into message boundaries.

The Generic Log Adapter provides the following extractors:

org.eclipse.hyades.logging.adapter.extractors.RegularExpressionExtractor
This extractor uses regular expressions to describe the delimitation of the log file into messages.
org.eclipse.hyades.logging.adapter.extractors.SimpleExtractor.
This extractor uses simple string comparisons to describe the delimitation of the log file into messages. This extractor, a high-performance alternative to the regular expression extractor, should be used with log files that contain unchanging message delimiters.

The following properties are provided as part of the configuration for the extractor:

Property Type Behavior
Description string A text description for the extractor. This value is used only by the Adapter Configuration Editor.
Unique Id string A unique identifier for this component. The extractor refers to this string to identify which component instance to use.
Contains Line Breaks boolean Indicates whether the messages in the log file span multiple lines.
Replace Line Breaks boolean Replace the line breaks in this file by another string segment. The replacement is made before the extractor searches for Start Pattern or End Pattern.
Line Break Symbol string The string segment to replace line breaks with. This is used only if Replace Line Breaks is true.
Include Start Pattern boolean Whether to include the StartPattern as part of the message to be forwarded to the next component. If the value is false, the matching portion of the message is stripped.
Start Pattern string or regular expression The string segment that identifies the start of a message. You can always use the RegularExpressionExtractor class. However, if your start pattern is a simple string, the SimpleExtractor class incurs less overhead.
Include End Pattern boolean Whether to include End Pattern as part of the message to be forwarded to the next component. If the value is false, the matching portion of the message is stripped.
End Pattern string or regular expression The string segment that identifies the end of a message. If End Pattern is not specified, then the message is delimited by the next Start Pattern.

Property	Type	Behavior
Description	string	A text description for the extractor. This value is used only by the Adapter Configuration Editor.
Unique Id	string	A unique identifier for this component. The extractor refers to this string to identify which component instance to use.
Contains Line Breaks	boolean	Indicates whether the messages in the log file span multiple lines.
Replace Line Breaks	boolean	Replace the line breaks in this file by another string segment. The replacement is made before the extractor searches for Start Pattern or End Pattern.
Line Break Symbol	string	The string segment to replace line breaks with. This is used only if Replace Line Breaks is true.
Include Start Pattern	boolean	Whether to include the StartPattern as part of the message to be forwarded to the next component. If the value is false, the matching portion of the message is stripped.
Start Pattern	string or regular expression	The string segment that identifies the start of a message. You can always use the RegularExpressionExtractor class. However, if your start pattern is a simple string, the SimpleExtractor class incurs less overhead.
Include End Pattern	boolean	Whether to include End Pattern as part of the message to be forwarded to the next component. If the value is false, the matching portion of the message is stripped.
End Pattern	string or regular expression	The string segment that identifies the end of a message. If End Pattern is not specified, then the message is delimited by the next Start Pattern.

Parser

The parser takes the messages that have been delimited by the extractor and builds a set of mappings of string values to a data structure. In the case of the Generic Log Adapter the data structure is Common Base Event 1.0.1

The Generic Log Adapter provides the following parser class:

org.eclipse.hyades.logging.adapter.parsers.Parser
This parser uses regular expressions to parse values to build Common Base Event objects. The java.util.regex library is used for processing regular expressions.

The parser has two phases of execution:

The global processing phase, where a set of global regular expressions are executed against the message provided by the extractor.
The attribute processing phase, where specific sets of substitution rules are executed to determine the value to be assigned to each attribute in the data structure (Common Base Event 1.0.1).

The two phases of execution allow the message to be tokenized into a series of attribute values during the global processing phase. The attribute values can then be referred to by attribute name or index during the attribute processing phase.

The following global properties are specified as part of the parser:

Property Type Behavior
Description string A text description for the parser. This value is used only by the Adapter Configuration Editor.
Unique Id string A unique identifier for this component. The parser refers to this string to identify which component instance to use.
Separator Token string or regular expression A token, or a regular expression that describes a series of tokens, that breaks the message up into attribute-value pairs.
For example, if each new line of your log file contains an attribute and its value, you could specify a Separator Token of \n. The advantage of breaking the message up is that the string segments can be referred to in the attribute processing phase.
Designation Token string or regular expression A token, or a regular expression that describes a series of tokens, that breaks the string segments created by the Separator Token into attribute-value pairs. For example, if each attribute and value are separated by a = sign , and the value of Designation Token is = , the attribute and its value can be referred to in the attribute processing phase. If a separator token is specified but a designation token is not specified then the message is broken up into a list of attribute values which can be referred to in the substitution rules by index.

Property	Type	Behavior
Description	string	A text description for the parser. This value is used only by the Adapter Configuration Editor.
Unique Id	string	A unique identifier for this component. The parser refers to this string to identify which component instance to use.
Separator Token	string or regular expression	A token, or a regular expression that describes a series of tokens, that breaks the message up into attribute-value pairs. For example, if each new line of your log file contains an attribute and its value, you could specify a Separator Token of \n. The advantage of breaking the message up is that the string segments can be referred to in the attribute processing phase.
Designation Token	string or regular expression	A token, or a regular expression that describes a series of tokens, that breaks the string segments created by the Separator Token into attribute-value pairs. For example, if each attribute and value are separated by a = sign , and the value of Designation Token is = , the attribute and its value can be referred to in the attribute processing phase. If a separator token is specified but a designation token is not specified then the message is broken up into a list of attribute values which can be referred to in the substitution rules by index.

The individual attributes of the Common Base Event have the following properties:

Property Type Behavior
Use Previous Match Substitution As Default boolean If the current set of substitution rules for this attribute returns a positive match, then store the match value so that it can be used as a default to process the next record if there is no positive match.
Default value String If none of the substitution rules for this attribute provide a positive match or there are no substitution rules for this attribute then this attribute will be assigned the default value specified here. If the same value is desired for all records for a particular attribute, this property can be used instead of creating a default substitution rule for that attribute. Note, the Use Previous Match Substitution as Default will take precedence if there is a saved value available.

Property	Type	Behavior
Use Previous Match Substitution As Default	boolean	If the current set of substitution rules for this attribute returns a positive match, then store the match value so that it can be used as a default to process the next record if there is no positive match.
Default value	String	If none of the substitution rules for this attribute provide a positive match or there are no substitution rules for this attribute then this attribute will be assigned the default value specified here. If the same value is desired for all records for a particular attribute, this property can be used instead of creating a default substitution rule for that attribute. Note, the Use Previous Match Substitution as Default will take precedence if there is a saved value available.

Substitution rules are run as part of the attribute processing phase. They are processed in order from top to bottom. Once a rule is a match, no further rules are run against this property. If no match is found, then no value is placed in the attribute unless the Use Previous Match Substitution as Default or Default Value properties of the attribute are specified.

Substitution rules contain the following properties:

Property Type Behavior
Match regular expression The regular expression that identifies the value to be applied to this attribute if a match occurs in the scope of the string segment being processed by this Substitution Rule. If you do not provide a regular expression, the rule is considered a match.
Positions name-value attribute pairs Positions let you refer to the name-value attribute pairs created during the global processing phase.

If both a separator token and a designation tokens are specified, the Positions string can refer to many attributes by name using the following grammar: $h('Attr1')@@$h('Attr2')...
$h('Attrn') identifies a previously created attribute name.
where @@ is the separator between the attributes.
If a separator token is specified but not a designation token, then the Positions string can refer to one or more attributes by index instead of name using the following grammar: n1@@n2..
where n1 and n2 are integers representing the indicies. The index starts at 1. If the message string begins with a separator token, the attribute value at index 1 will be null. The first attribute value will begin at index 2.

In the Match property, you can add the separator to indicate which regular expressions should be applied to which attribute value. The Substitute property can refer to these attribute atoms using $1, $2, and so on..
Substitute string The value to place in the attribute. This can be a string or it can refer to the regular expression in the Match property. The string segments that are extracted by the Match property can be referred to using the groupings in the regular expression. $1 indicates the first grouping, $2 the second, and so on.
Use built in function check box Use built-in function when selected for an attribute, if none of the previous substitution rules, if any exist, are matched, will cause the Generic Log Adapter runtime to generate a value for that attribute using its own internal functions. This feature is only supported the followingn CBE attributes:

localInstanceId
globalInstanceId
componentIdentification.location
componentIdentification.locationType

Substitution Class string Specifies the name of a user created Java class that can perform custom parsing logic on the value matched by the Positions, Match and Substitute specifications of the rule. If the substitution rule matches data in the record, this class will be called and the result will be used to create the Common Base Event object.
Specifications:

This class must implement the interface class org.eclipse.hyades.logging.adapter.parsers.ISubstitutionExtension or the extension class org.eclipse.hyades.logging.adapter.parsers.IStatefulSubstitutionExtension in order for it to be executed by the Generic Log Adapter.
Use the extension class when you want to save state data between log file records. The class org.eclipse.hyades.logging.adapter.parsers.IStatefulSubstitutionExtension causes the Generic Log Adapter to only allocate one instance of this class to parse all records of the log file.
The user class must be included in the classpath of the Generic Log Adapter runtime.
If the Generic Log Adapter is running in an Eclipse environment, the class can be created as an eclipse plug-in extension of the Generic Log Adapter by using the substitutionExtension extension point of the org.eclipse.hyades.logging.adapter plugin.

Time Format string This field only applies to substitution rules for the creationTime element and is used to simplify date time parsing. For example, some log files contain timestamps that include the names of a month instead of its numerical representation. This makes it difficult to create substituion rules to generate the string value in the form of the XML Schema dateTime data type required by the CBE creationTime element. In this case the user can specify a java simple date format string as documented in the javadoc for java.text.SimpleDateFormat (java.sun.com/j2se/1.4.2/docs/api/java/text/SimpleDateFormat.html) in the Time Format field. The value matched by the Positions, Match and Substitute specifications of the rule will be parsed based on the Time Format string and the resulting value will be used to generate the CBE creationTime value.

Property	Type	Behavior
Match	regular expression	The regular expression that identifies the value to be applied to this attribute if a match occurs in the scope of the string segment being processed by this Substitution Rule. If you do not provide a regular expression, the rule is considered a match.
Positions	name-value attribute pairs	Positions let you refer to the name-value attribute pairs created during the global processing phase. If both a separator token and a designation tokens are specified, the Positions string can refer to many attributes by name using the following grammar: $h('Attr1')@@$h('Attr2')... $h('Attrn') identifies a previously created attribute name. where @@ is the separator between the attributes. If a separator token is specified but not a designation token, then the Positions string can refer to one or more attributes by index instead of name using the following grammar: n1@@n2.. where n1 and n2 are integers representing the indicies. The index starts at 1. If the message string begins with a separator token, the attribute value at index 1 will be null. The first attribute value will begin at index 2. In the Match property, you can add the separator to indicate which regular expressions should be applied to which attribute value. The Substitute property can refer to these attribute atoms using $1, $2, and so on..
Substitute	string	The value to place in the attribute. This can be a string or it can refer to the regular expression in the Match property. The string segments that are extracted by the Match property can be referred to using the groupings in the regular expression. $1 indicates the first grouping, $2 the second, and so on.
Use built in function	check box	Use built-in function when selected for an attribute, if none of the previous substitution rules, if any exist, are matched, will cause the Generic Log Adapter runtime to generate a value for that attribute using its own internal functions. This feature is only supported the followingn CBE attributes: localInstanceId globalInstanceId componentIdentification.location componentIdentification.locationType
Substitution Class	string	Specifies the name of a user created Java class that can perform custom parsing logic on the value matched by the Positions, Match and Substitute specifications of the rule. If the substitution rule matches data in the record, this class will be called and the result will be used to create the Common Base Event object. Specifications: This class must implement the interface class org.eclipse.hyades.logging.adapter.parsers.ISubstitutionExtension or the extension class org.eclipse.hyades.logging.adapter.parsers.IStatefulSubstitutionExtension in order for it to be executed by the Generic Log Adapter. Use the extension class when you want to save state data between log file records. The class org.eclipse.hyades.logging.adapter.parsers.IStatefulSubstitutionExtension causes the Generic Log Adapter to only allocate one instance of this class to parse all records of the log file. The user class must be included in the classpath of the Generic Log Adapter runtime. If the Generic Log Adapter is running in an Eclipse environment, the class can be created as an eclipse plug-in extension of the Generic Log Adapter by using the substitutionExtension extension point of the org.eclipse.hyades.logging.adapter plugin.
Time Format	string	This field only applies to substitution rules for the creationTime element and is used to simplify date time parsing. For example, some log files contain timestamps that include the names of a month instead of its numerical representation. This makes it difficult to create substituion rules to generate the string value in the form of the XML Schema dateTime data type required by the CBE creationTime element. In this case the user can specify a java simple date format string as documented in the javadoc for java.text.SimpleDateFormat (java.sun.com/j2se/1.4.2/docs/api/java/text/SimpleDateFormat.html) in the Time Format field. The value matched by the Positions, Match and Substitute specifications of the rule will be parsed based on the Time Format string and the resulting value will be used to generate the CBE creationTime value.

Formatter

The formatter takes the mappings of attributes to their values provided by the parser and builds the correct Java object instance. The adapter runtime contains a single formatter that creates Common Base Event objects conforming to the Common Base Event version 1.0.1 specification.

The formatter contains the following properties:

Property Type Behavior
Unique Id string A unique identifier for this component. The formatter refers to this string to identify which component instance to use.
Description string A text description for the formatter. This value is used only by the Adapter Configuration Editor.

Property	Type	Behavior
Unique Id	string	A unique identifier for this component. The formatter refers to this string to identify which component instance to use.
Description	string	A text description for the formatter. This value is used only by the Adapter Configuration Editor.

Outputter

The outputter externalizes the resulting Common Base Event records provided by the formatter. Outputters provide, or wrap, the mechanism for storing the final outputs of the context. The Hyades Generic Log Adapter provides the following outputters:

org.eclipse.hyades.logging.adapter.outputters.CBEFileOutputter: receives Common Base Event instances and writes the externalized records to a file.
org.eclipse.hyades.logging.adapter.outputters.CBEStdoutOutputter: receives Common Base Event instances and writes the externalized records to standard output.
org.eclipse.hyades.logging.adapter.outputters.CBELogOutputter: This outputter receives Common Base Event records, creates a logging agent and writes the externalized records to the logging agent. The CBELogOutputter should only be included in adapter configuration files that will be used outside of eclipse. The user can specify the logging agent name as an Outputter proprerty in the adapter configuration file and must manually attach and start monitoring the agent within Log and Trace Analyzer in order to see the data in the Log View.
com.ibm.etools.logging.adapter.outputters.CBENotifyOutputter: receives Common Base Event instances and sends the externalized records via a Notification Source to a Notification Sink.
org.eclipe.hyades.logging.adapter.outputters.StaticParserOutputter: This outputter receives Common Base Event instances and writes the externalized records to a logging agent. The StaticParserOutputter should only be included in adapter files that will be used to import logs into the Log and Trace Analyzer. It should only be used for parsers that are extensions of the org.eclipse.hyades.logging.parsers.logParser extension point. The logging agent is known to the Log and Trace Analyzer and the Log and Trace Analyzer will attach and start monitoring it automatically during the import action.

Properties of the outputter:

Property Type Behavior
Unique Id string A unique identifier for this component. The outputter refers to this string to identify which component instance to use.
Description string A text description for the outputter. This value is used only by the Adapter Configuration Editor.
Outputter Type enumeration The type of outputter this is.

Property	Type	Behavior
Unique Id	string	A unique identifier for this component. The outputter refers to this string to identify which component instance to use.
Description	string	A text description for the outputter. This value is used only by the Adapter Configuration Editor.
Outputter Type	enumeration	The type of outputter this is.

Outputter types and required properties:

Outputter type Required properties
StandardOutOutputter None
SingleFileOutputter directory, fileName
LoggingAgentOutputter agentName - name given to the logging agent when it is created
waitUntilLoggingTime(optional)- the time the outputter will wait for the user to attach and start monitoring the agent before it starts writing to the agent. If this property is not specified, the outputter will not wait for the agent to be monitored before writing to it.
NotificationOutputter resourceURL - URL of the Notification Sink which will receive the events

Outputter type	Required properties
StandardOutOutputter	None
SingleFileOutputter	directory, fileName
LoggingAgentOutputter	agentName - name given to the logging agent when it is created waitUntilLoggingTime(optional)- the time the outputter will wait for the user to attach and start monitoring the agent before it starts writing to the agent. If this property is not specified, the outputter will not wait for the agent to be monitored before writing to it.
NotificationOutputter	resourceURL - URL of the Notification Sink which will receive the events