This article describes how to set up and use the logging and monitoring feature.
Introduction
To keep track of the resource usage for a Spotfire Web Player Server, it is possible to enable logging of document cache, open files, user sessions, and performance counters. As a complement, the library log from the TIBCO Spotfire Server can give information about what is actually loaded from the library. All log files are formatted for easy import into TIBCO Spotfire Professional. This logging makes it much easier to find out how often and at what times the different analyses are used and how well the servers handle the work load.
The Web Player uses log4net for the logging, so it is possible to log to a database instead of files, but this article describes how to work with log files.
Configuration
The configuration has to be done in two places: web.config and log4net.config. This is a good thing because log4net.config may be changed without restarting the Web Player Server. If you will need logging in the future, you should therefore do at least the web.config configuration, including setting enabled="true", to be able to enable logging later without restarting the Web Player Server.
Configuring web.config
The performance counter logging has to be enabled, and the set of counters to log is also decided here, for example:
<performanceCounterLogging
enabled="true"
logInterval="120"
counters="
TIBCO Spotfire Webplayer;# cached documents;,
TIBCO Spotfire Webplayer;# open documents;,
TIBCO Spotfire Webplayer;# concurrent users;,
PhysicalDisk;Avg. Disk Queue Length;_Total,
Network Interface;Bytes Total/sec;,
Memory;Available Bytes;,
Processor;% Processor Time;_Total
ASP.NET;Application Restarts;,"
debugLogInterval="15"
debugCounters="
TIBCO Spotfire Webplayer;# accumulated cached documents;,
TIBCO Spotfire Webplayer;# accumulated open documents;,
PhysicalDisk;Disk Bytes/sec;_Total,
PhysicalDisk;% Disk Time;_Total,
Memory;Committed Bytes;,
Process;Private Bytes;w3wp,
System;Context Switches/sec;,
Web Service;ISAPI Extension Requests/sec;_Total,
.NET CLR LocksAndThreads;# of current physical Threads Thread;_Global_,
.NET CLR LocksAndThreads;Contention Rate / Sec;_Global_,
.NET CLR Memory;# Total committed Bytes;_Global_,
.NET CLR Memory;% Time in GC;_Global_,
ASP.NET Applications;Requests/Sec;__Total__,"
/>
"counters" is a comma-separated list of performance counter specification, each consisting of three parts (separated by ';') : Category, Counter name, and Instance. For the non TIBCO specific counters, the Counter name and/or Instance may be omitted to be interpreted as a wildcard. For some counters the Instance is not used and therefore omitted, for example the TIBCO specific counters above.
The "counters" are logged at the INFO level but from version 3.3, it is also possible to give a list of counters "debugCounters" that are logged at the DEBUG level. If the logging level is DEBUG, the time between each logging is given by the debugLogInterval. Note that "counters" and "logInterval" (instead of adding the info prefix to them) are still used to keep the compatibility with 3.2 config files.
Then the data collection of users, sessions, etc. has to be enabled:
<statistics enabled="true" flushInterval="60"/>
These block are located inside spotfire.dxp.web/performance so just update them:
<!-- ************************************************************
This section contains setting for tuning performance.
Be careful when making changes. -->
<performance>
<!-- Enable performance counter logging. Add logger and appender to log4net.config to log the values.
The list of counters is a comma separated list of triples (group, counter, instance)
where each part is separated by semicolon. Each item should therefore contain exactly two
semicolons. Other items will be skipped. For the standard counters (not starting with TIBCO)
Empty strings (for counter or instance) is interpreted as wildcards. -->
<performanceCounterLogging
enabled="false"
logInterval="15"
counters="TIBCO Spotfire Webplayer;# cached documents;, Processor;% Processor Time;_Total"
/>
<!-- Enable data collection used for logging and monitoring.
Loggers and appenders have to be added to log4net.config -->
<statistics flushInterval="60" enabled="false"/>
Configuring log4net.config
The default log4net file configures the Spotfire.Dxp.Web.log file. What has to be done is to add loggers and appenders for each log file. The logging level may be set for each logger but in the following example, it is set in one place for all loggers:
<!-- WebLogger logging level (ALL=enabled, NONE=disabled) -->
<logger name="WebLogger">
<level value="DEBUG" />
</logger>
The levels used are: OFF, INFO, DEBUG.
The first appender is for the AuditLog. Note the rolling settings and header rows.
<!-- Audit log for successful and failed authentications -->
<appender name="AuditLog" type="log4net.Appender.RollingFileAppender">
<file value="Logs\AuditLog.txt"/>
<appendToFile value="true"/>
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="100MB" />
<staticLogFileName value="false" />
<layout type="log4net.Layout.PatternLayout">
<header value="Host Name;DateTime;Session Id;IP Address;User Name;Operation;Status String;DateTime;String;String;String;String;String "/>
<conversionPattern value="%message%newline"/>
</layout>
</appender>
<logger name="WebLogger.WebAuditLog" additivity="false">
<appender-ref ref="AuditLog"/>
</logger>
Here are example configurations for the rest of the log files:
<!-- Statistics log for document cache information -->
<appender name="DocumentCacheStatisticsLog" type="log4net.Appender.RollingFileAppender">
<file value="Logs\DocumentCacheStatisticsLog.txt"/>
<appendToFile value="true"/>
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="100MB" />
<staticLogFileName value="false" />
<layout type="log4net.Layout.PatternLayout">
<header value="Host Name;DateTime;Analysis;Modified;Reference Count String;DateTime;String;DateTime;Integer "/>
<conversionPattern value="%message%newline"/>
</layout>
</appender>
<logger name="WebLogger.DocumentCacheStatisticsLog" additivity="false">
<appender-ref ref="DocumentCacheStatisticsLog"/>
</logger>
<!-- Statistics log for open files information -->
<appender name="OpenFilesStatisticsLog" type="log4net.Appender.RollingFileAppender">
<file value="Logs\OpenFilesStatisticsLog.txt"/>
<appendToFile value="true"/>
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="100MB" />
<staticLogFileName value="false" />
<layout type="log4net.Layout.PatternLayout">
<header value="Host Name;DateTime;Session Id;Analysis;Modified;File Id;Time Since Opened;Inactivity Time String;DateTime;String;String;DateTime;String;TimeSpan;TimeSpan "/>
<conversionPattern value="%message%newline"/>
</layout>
</appender>
<logger name="WebLogger.OpenFilesStatisticsLog" additivity="false">
<appender-ref ref="OpenFilesStatisticsLog"/>
</logger>
<!-- Performance counter values -->
<appender name="PerformanceCounterLog" type="log4net.Appender.RollingFileAppender">
<file value="Logs\PerformanceCounterLog.txt" />
<appendToFile value="true" />
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="100MB" />
<staticLogFileName value="false" />
<layout type="log4net.Layout.PatternLayout">
<header value="Host Name;DateTime;Category;Counter;Instance;Value String;DateTime;String;String;String;Real " />
<conversionPattern value="%message%newline" />
</layout>
</appender>
<logger name="WebLogger.PerformanceCounterLog" additivity="false">
<appender-ref ref="PerformanceCounterLog"/>
</logger>
<!-- Statistics log for user session information -->
<appender name="UserSessionStatisticsLog" type="log4net.Appender.RollingFileAppender">
<file value="Logs\UserSessionStatisticsLog.txt"/>
<appendToFile value="true"/>
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="100MB" />
<staticLogFileName value="false" />
<layout type="log4net.Layout.PatternLayout">
<header value="Host Name;DateTime;Session Id;IP Address;User Name;Browser Type;Cookies;Logged In Time;Max Open Files;Open Files String;DateTime;String;String;String;String;String;TimeSpan;Integer;Integer " />
<conversionPattern value="%message%newline"/>
</layout>
</appender>
<logger name="WebLogger.UserSessionStatisticsLog" additivity="false">
<appender-ref ref="UserSessionStatisticsLog"/>
</logger>
<!-- A file that contains all DateTimes from the other WebLogger log files.
Use this file to join the other log files. -->
<appender name="DateTimesLog" type="log4net.Appender.RollingFileAppender">
<file value="Logs\DateTimesLog.txt"/>
<appendToFile value="true"/>
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="100MB" />
<staticLogFileName value="false" />
<layout type="log4net.Layout.PatternLayout">
<header value="Host Name;DateTime String;DateTime " />
<conversionPattern value="%message%newline"/>
</layout>
</appender>
<logger name="WebLogger.DateTimesLog" additivity="false">
<appender-ref ref="DateTimesLog"/>
</logger>
Make sure to set the paths to the log files correctly – just use the folder path from the Spotfire.Dxp.Web.log logger.
The appenders can be replaced with AdoNetAppenders to log to a database instead of text files (see log4net Config Examples for more info).
Using the Log Files
The most obvious way to look at the log files is to create a Spotfire analysis and create tables linked to copies of the log files. Here is an example analysis with embedded data tables: Logging_and_Monitoring_example.dxp
Just replace the data tables with linked data from the log files and make sure to re-save using linked data. The analysis contains linked data from a set of log files:
And for the Web Player Server:
-
AuditLog.txt: All login/logout attempts
-
UserSessionStatisticsLog.txt: The existing sessions sampled regularly
-
OpenFilesStatisticsLog.txt: The open analyses sampled regularly
-
DocumentCacheStatisticsLog.txt: The cached analyses sampled regularly
-
PerformanceCounterLog.txt: Standard and custom performance counters logged regularly
-
DateTimesLog.txt: Used to connect the time axis between tables – one slider filters all data tables
Notes About the Example Analysis
- The DateTimesLog has relations to all other Web Player log tables to get the single DateTime slider that filters the other tables instead of one slider per table.
- The analysis should be seen as an example. How the visualizations should be set up depends on what we want to show and also on the server load, the number of analyses, the number of users, etc.
- The library log from the TIBCO Spotfire Server is not related to DateTimesLog so it needs its own DateTime slider.
Setting up the Analysis for Real Time Monitoring
It is possible to get almost real time logs from a running Web Player by following some setup steps:
- Copy the analysis to the server where the Web Player is running and open it in Spotfire Professional
- Replace all data tables and link them to the Web Player's log files
- Publish the analysis to the local Web Player's library
- From version 3.3 you have to change the Web Player setting that allows linking to files inside the webroot folder (if the log files are located there), i.e. add a value for the setting AllowedWebRootFiles inside web.config:
<setting name="AllowedWebRootFiles" serializeAs="String">
<value>..\Logfiles\AuditLog.txt;..\Logfiles\DateTimesLog.txt;
..\Logfiles\DocumentCacheStatisticsLog.txt;
..\Logfiles\OpenFilesStatisticsLog.txt;..\Logfiles\PerformanceCounterLog.txt;
..\Logfiles\UserSessionStatisticsLog.txt;..\Logfiles\Spotfire.Dxp.Web.log;
</value>
</setting>
- Optionally, enable scheduled updates to refresh the analysis for example every 10 minutes
Now, the fresh logs can be browsed in the published analysis.
It is possible to avoid installing Spotfire Professional on the Web Player Server. The trick is to put copies of the log files on a client machine at the same local paths as they are located on the server. Publishing the analysis to the Web Player will cause it to point to the server's local log files instead of the client copies.
Various Issues
Duplicated header lines in log files
A bug in log4net sometimes generates multiple copies of the header rows in the log files. If the generated log files are used as linked data in a Spotfire analysis, the header rows may be removed from the log4net.config file to avoid import problems, but they are nice to have when creating the analysis to get the names and types correct.
Rolling the Files
The reason to roll the files is that they may become too big over time. If possible they should not be rolled at all but especially the performance counter log tends to grow too much. One solution may be to roll the files once a month (or day) instead of after size. This will keep them in sync. The example configuration above limits the file sizes to 100 MB each.