Since Log Insight version 8.4, a new tab has been added to the user interface called “Log Sources”.When we look further we see two agents there, besides the new FluentD agent, the well-known Log Insight Agent (LI Agent). The LI Agent is available for Windows and Linux platforms and can forward events from log files to a Log Insight server or other syslog destinations.
By the way, did you know that the LI Agent is also installed by default on a vRealize Operations Manager (vROPS) node? When you configure Log Forwarding via the GUI (Administration > Management > Log Forwarding), you are actually configuring the LI agent.
The reason to pay attention to the LI Agent was a recent situation regarding auditing of applications. Because Log Insight plays a central role in a SDDC environment, auditing of components such as ESXi, vCenter Server ed. can take place via Log Insight, but what about the auditing of Log Insight itself?
A pretty old KB 53123 is a good starting point. Just like vROPS mentioned before, each Log Insight node has the LI Agent installed by default. LI Agents can be centrally configured from the Log Insight GUI or by manually editing the configuration file liagent.ini in folder /var/lib/loginsight-agent. After editing the configuration, the new configuration will automatically become active and can be observed in the file liagent-effective.ini.
<fig1>
The following test setup was created; vRLI-1 is the log source, a one node Log Insight instance where the Log insight Agent is configured.
A second Log Insight instance, vRLI-2 serves as a destination host. For our auditing purposes we start by forwarding successful and unsuccessful login attempts on the vRLI-1, these are logged in the file ui_runtime.log in the /storage/var/loginsight folder, among others.
In the first example, the agent is configured to send the events to a host using the syslog protocol over port 514.
Note: Comments have been added to the loginsight-agent.ini file. In these examples, the comment lines have been omitted for clarity.
[server]
hostname=vrli-2.virtual.local
proto=syslog
port=514
ssl=no
[logging]
debug_level=0
[storage]
max_disk_buffer=200
[filelog|ui_runtime]
directory=/storage/var/loginsight
include=ui_runtime.log
event_marker=\[\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}\+\d{4}\]
[update]
package_type=rpm
auto_update=no
Line 1: Marks the start of the receiving Log Insight or Syslog server.
Line 3: The protocol usewd by the agent: cfapi or syslog.
Line 4-5: syslog uses port 514 when ssl is disabled.
Line 13: Marks the start of the first log file that will be forwarded. For Log Insight Linux Agents, this section always start with filelog. The second part must be unique, in this case: ui_runtime.
Line 14: The full path to the log file folder.
Line 15: the include=ui_runtime.log means, this log will be forwarded.
Other log files in this folder are ignored. The LI agent supports the use of regular expressions in defining the log files, which offers a lot of flexibility. More information can be found in the LI Agent manual.
Line 16: the event marker is a regular expression that denotes the start of an event in the log file.
Tip: Copy a few lines of the ui_runtime.log and the regex and go to https://regex101.com/ to see how it works.
After the configuration is saved, it becomes active immediately. The log file in liagent_yyyy-mm-dd_nn.log in the folder /var/log/loginsight-agent can be helpful for troubleshooting in case of connectivity issues.
<fig2>
As a first test, logout from vRLI-1 and logon as a not authorized user will create the first two events that can be observed in vRLI-2.
<fig3>
If the syslog target is another Log Insight instance like in this example, the cfapi protocol is the better option. The cfapi protocol uses the Log Insight Ingestion API which comes with several advantages, like statistical info and tagging. More info in this post by Steve Flanders.
The second example, liagent.ini is now configured for the cfapi protocol and additional log files were added.
[server]
hostname=vrli-2.virtual.local
proto=cfapi
port=9543
ssl=yes
reconnect=30
central_config=no
[logging]
debug_level=0
[storage]
max_disk_buffer=200
[filelog|ui_runtime]
directory=/storage/var/loginsight
include=ui_runtime.log
tags={"vmw_vr_log_logtype":"UI_RUNTIME"}
event_marker=\[\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}\+\d{4}\]
[filelog|runtime]
directory=/storage/var/loginsight
include=runtime.log;runtime.log.*
tags={"vmw_vr_log_logtype":"RUNTIME"}
event_marker=\[\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}\+\d{4}\]
[filelog|audit]
directory=/storage/var/loginsight
include=audit.log
tags={"vmw_vr_log_logtype":"AUDIT"}
event_marker=\[\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}\+\d{4}\]
[update]
package_type=rpm
auto_update=no
Line 1-7: Receiving host is configured for cfapi protocol with ssl.
Line 21-31: the runtime.log and audit.log have been added.
For rotating log files like the runtime.log, a pattern must be added to include the primary file and the rotated file(s), like: include=runtime.log;runtime.log.*
Line 18, 24, 30: A Tag with tag name “vmw_vr_log_logtype” has been added. The tag value represents the name of the log file in this example. This can be useful for filtering log files, see figure 4.
<fig4>
Until now, all events were sent to a single destination, however the LI agent supports to up to three destinations. A destination can include a Log Insight server or another syslog capable solution.
The following example is an extension of the previous example. With the addition of these six lines, the ui_runtime.log and the runtime.log are sent to a vRLI-1.
Yes, a Log Insight node can ingest its own log files.
Again, the cfapi protocol is configured. The last line starting with filter determines which log files as defined for the primary destination (vRLi-2) are sent to the secondary destination (vRLI-1).
Therefore, no log files that are not configured for the first destination can be sent.
Filters have the form: { collector_type ; collector_filter ; event_filter }, where collector_type is again filelog.
The collector_filter *runtime matches with two out of three collector sections; [filelog|ui_runtime] and [filelog|runtime]. So log files associated with section [filelog|audit] will not be processed.
The event_filter is not used in this example, but allows to filter on specific events by defining and allowlist or a denylist.
[server|runtime-audit]
hostname=vrli-1.virtual.local
proto=cfapi
port=9543
ssl=yes
filter={filelog; *runtime; }
These examples give a first impression of the capabilities of the LI Agent and are already sufficient to configure the auditing mentioned in the introduction.
The agent has many more features, such as the already mentioned filtering of events and a log parser with extensive possibilities. Furthermore, the GUI of Log Insight offers the possibility to group and centrally configure agents from different sources and to update the agents. All options are described in detail in the documentation.
