Logging for Apache Tomcat and Velocity using Log4j

Written by Geoff Mottram (geoff at minaret dot biz).

Placed in the public domain on August 28, 2004 by the author.

Last updated: January 27, 2005.

This document is provided "as is", without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. In no event shall the author be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with this document or the use or other dealings in this document.

Contents
Introduction
Tomcat 5.0 Logging
Tomcat 5.5 Logging
Tomcat Commons Logging
Log4j
Log4j Properties File
Caveat
Tomcat Bootstrapping
Tomcat and Jakarta Velocity
WEB-INF Logging
Logger Identifiers in Apache Code
Appendix 1 - Logger Identifiers in Tomcat 5.0.27
Appendix 2 - Logger Identifiers in Jakarta Struts 1.1

Introduction
It took a number of hours to get Apache Tomcat version 5.0.27 and Apache Velocity 1.4 to create log files the way I wanted them for a production environment. This document describes some of the things I discovered and provides step by step instructions for configuring Tomcat to generate log files that rotate nightly. This document covers both Tomcat 5.0 and 5.5.

Please note: The new alpha version of Log4j 1.3 has added support for a daily rolling log file. See Log4j Version 1.3 and Apache Tomcat for instructions on how to install and configure this new feature.

Tomcat 5.0 Logging
Tomcat 5.0 has two primary channels for logging information. The first is usually output to the catalina.out file and the other is the Servlet log, which typically saves its messages to files with a date as part of their name (i.e. catalina.2004-08-27.log).

The data in catalina.out comes from three sources: data that is written to standard output and standard error, and information that is logged via the Apache Commons Logging interface. This last category includes all manner of output from the Tomcat server about its state of affairs and is also used by other many other systems (like Jakarta Struts). Packages that use the Commons Logging interface can output varying amounts of log data according to how you configure your logging system (more on that in a moment).

The data in the Servlet log is generated by calls to the log() method of the ServletContext class. For example: 

httpServletRequest.getSession().getServletContext().log("Some message");
To save the output from the Server log in a file, all you need is a FileLogger directive in your Tomcat server.xml file (in the Tomcat conf directory). It typically looks something like this:

      <!-- Global logger unless overridden at lower levels -->
      <Logger className="org.apache.catalina.logger.FileLogger"
              prefix="servlet." suffix=".log" timestamp="true" />
The Servlet log is either on (by including a FileLogger directive) or off (by not having a FileLogger) but there is no way to specify a logging level (i.e. include only INFO messages and higher). While the Tomcat documentation mentions that all of its Loggers support a verbosity value, setting it doesn't seem to do anything.

When configuring your Tomcat server, you want to capture both streams of logging data into files that "roll over" every day (i.e. change their name to reflect the date they were made). Using the FileLogger directive in your Tomcat configuration provides a standard way to capture the Servlet log stream to a file that changes each day. This allows you to backup, remove, analyze or whatever else you like to do with your server logs.

It would be nice if there was a Catalina Logger that utilized the Commons Logging interface. That would give you more control over the formatting of the output stream and such a module would be fairly easy to write, but who's got the time? The next section describes how to tame the catalina.out log stream.

Tomcat 5.5 Logging
Tomcat 5.5 has two avenues for logging data. Anything written to standard out and standard error is always sent to the catalina.out file. Everything else has gone through the Apache Commons Logging interface. If you do not configure an alternate logger (like log4j), all logged messages (including everything that goes through commons-logging) ends up in an endlessly growing catalina.out.

One improvement in Tomcat 5.5 over version 5.0 is that the Servlet log no longer exists. This used to receive any messages that were passed to the log() method of the ServletContext class. For example:

httpServletRequest.getSession().getServletContext().log("Some message");

Tomcat now sends this information through the commons-logging interface as INFO level messages. By configuring a logger like log4j, you can redirect virtually all of your logging messages into files that "roll over" every day (i.e. change their name to reflect the date they were made). This allows you to backup, remove, analyze or whatever else you like to do with your server logs.

Tomcat Commons Logging
As mentioned above, much of the data in the catalina.out file is produced by calls to the Apache Commons Logging interface. Commons Logging gives you run-time control over the level of logging detail and permits you to use any number of underlying logging systems. The standard Tomcat distribution uses SimpleLog as the underlying logger. SimpleLog sends all of its output to standard error, which in Tomcat, is redirected to the file catalina.out by default. The biggest problem with this arrangement is that the catalina.out file grows forever -- there is no file rotation.

Log4j
Log4j is a powerful log manager that supports the Commons Logging interface. It can be dropped into your Tomcat installation as a replacement for SimpleLog to provide log rotation and all manner of configuration options. In fact, you are not limited to saving your logs to a file. They can be sent to the syslog, a remote port or other destinations. You can also send them to more than one place. To use Log4j as your log manager in Tomcat (see the separate instructions on how to install and configure the new alpha version of Log4j 1.3 with Apache Tomcat):

  1. Shutdown Tomcat if it is currently running.
  2. Download the Commons Logging package from the Apache web site (unless you already have it).
  3. Copy the commons-logging.jar file from the distribution into your Tomcat common/lib directory.
  4. Download the Log4j package from the Apache web site (unless you already have it).
  5. Copy the log4j.jar file from the distribution into your Tomcat common/lib directory.
  6. Create a log4j.properties file in your Tomcat common/classes directory (see next section).
  7. Restart Tomcat.
The configuration described here results in the creation of two log files for Tomcat 5.5 and three log files for Tomcat 5.0: the Servlet log file (only with Tomcat 5.0), which will roll over to a new file every night; the Commons Logging data via log4j (which will also roll over every night) and the catalina.out file, which will only contain messages that have been printed to standard output and standard error. The last file will grow forever but well behaved applications within your Tomcat server should not be printing to standard out or error, so this should not really be an issue (in general, the file should remain zero length).

Note that the commons-logging.jar and log4j.jar files are installed in the Tomcat common/lib directory and the log4j.properties file is installed in the common/classes directory. This arrangement allows both the Tomcat server and your web applications access to the log4j system. If your web applications use the Commons Logging interface, their log data will be output to the Tomcat server log. You can configure the system to output your application data to a different file by changing the configuration of the Log4j properties file (for help with that, you will have to look elsewhere).

Log4j Properties File
Your log4j.properties file should look something like this: 

#
# Configures Log4j as the Tomcat system logger
#

#
# Configure the logger to output info level messages into a rolling log file.
#
log4j.rootLogger=INFO, R

#
# To continue using the "catalina.out" file (which grows forever),
# comment out the above line and uncomment the next.
#
#log4j.rootLogger=ERROR, A1

#
# Configuration for standard output ("catalina.out").
#
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
#
# Print the date in ISO 8601 format
#
log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c - %m%n

#
# Configuration for a rolling log file ("tomcat.log").
#
log4j.appender.R=org.apache.log4j.DailyRollingFileAppender
log4j.appender.R.DatePattern='.'yyyy-MM-dd
#
# Edit the next line to point to your logs directory.
# The last part of the name is the log file name.
#
log4j.appender.R.File=/usr/local/tomcat/logs/tomcat.log
log4j.appender.R.layout=org.apache.log4j.PatternLayout
#
# Print the date in ISO 8601 format
#
log4j.appender.R.layout.ConversionPattern=%d [%t] %-5p %c - %m%n

#
# Application logging options
#
#log4j.logger.org.apache=DEBUG
#log4j.logger.org.apache=INFO
#log4j.logger.org.apache.struts=DEBUG
#log4j.logger.org.apache.struts=INFO
The only change you have to make is to edit the line with log4j.appender.R.File=/usr/local/tomcat/logs/tomcat.log to point to your Tomcat logs directory.

The above log4j configuration will log info messages as well as those that are more important (warning, error and fatal) to the tomcat.log file in your Tomcat logs directory. The first message that is logged after midnight every day will cause the log file to be renamed with the current date appended to the name tomcat.log and a new tomcat.log file to be created.

The one fly in the ointment with the log4j DailyRollingFileAppender is that in order for the log file to be renamed, a message must be logged to trigger the change. This is a problem if you run a cron job every night to do something with your log files. The Tomcat FileLogger solves this problem by always using the current date as part of the current log's file name. I have written a custom Log4j appender that mimics the behavior of the Tomcat FileLogger which you are free to use (source code included). A separate technical tip describes the DatedFileAppender for Log4j along with installation and configuration instructions.

Please note: The new alpha version of Log4j 1.3 has added support for a daily rolling log file. See Log4j Version 1.3 and Apache Tomcat for instructions on how to install and configure this new feature.

You can override the default logging level for specific applications with the commands at the end of the configuration file. In production, you may want even less logging and you could change the log4j.rootLogger=INFO, R line at the top of the file to read log4j.rootLogger=ERROR, R instead.

Caveat
Be careful when setting the logging level to DEBUG (particularly something like log4j.logger.org.apache, as an enormous amount of information can be generated, slowing your system down considerably.

Tomcat Bootstrapping
Please note, you MUST install commons-logging.jar into your common/lib directory. You might have noticed that there is a commons-logging-api.jar file in the bin (bootstrap) directory of Tomcat. This jar file appears to be a stripped down version of the full commons-logging.jar file which only implements SimpleLog or something like it. Basically, just enough of a logging system to get bootstrapped. While you can't boot without the API version of the logging system, you also can't replace it with a full blown commons-loging.jar file without adding your logging system (i.e. Log4j) to the bootstrap classpath.

Through some classloading voodoo during bootstrapping, if you have the full commons-logging.jar file in your common/lib directory, it replaces the classes from the commons-logging-api.jar file and will reinitialize the logging system and attempt to locate log4j or whatever other logging system you may be using.

Application Logging
You can log messages in your own Servlet applications by either writing to the Servlet log (i.e. servletContext.log("Some message")) or by utilizing the Commons Logging interface. This section describes the latter and far superior method of logging and demonstrates how it integrates with the logging enhancements to Tomcat as described above.

A sample application: 

package com.acme.webapp;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

public class MyApplication {

    private Log myLog;

    public MyApplication() {
        myLog = LogFactory.getLog(MyApplication.class);
    }

    public void process() {
        // Do some processing

        myLog.debug("A debug message");
	
        // Do something else

        myLog.info("An information message");

	// etc.
    }
}
The Log object can be shared between classes and may be declared as a static class variable to facilitate sharing. In other words, your entire application can share a single Log object or you can use multiple Log objects to provide more run-time control over log generation. You pass the LogFactory.getLog() method a class, from which the fully qualified name of that class is used as a unique name for that logging object. In the above example, the name would be com.acme.webapp.MyApplication. By giving each log object a name, it can be controlled in the log4j.properties file using that same identifier. In this case, you could add the following line to your properties file to log DEBUG messages in your application: 
log4j.logger.com.acme.webapp.MyApplication=DEBUG

The name you give each Log object can be any arbitrary string. However, the general convention is to use the name of the class that creates the Log object. The log4j FAQ describes an alternate naming system.

Appendix 1 contains a list of Log objects used by Tomcat. One of the more powerful features of log4j is the ability to control hierarchies of Log objects with a single command. For example, the following entry in the log4j.properties file would produce DEBUG level output from all of the Tomcat login (or realm) handlers: 

log4j.logger.org.apache.catalina.realm=DEBUG
This would affect JAASMemoryLoginModule, JAASRealm, MemoryRealm and RealmBase.

Tomcat and Jakarta Velocity
Version 1.4 of the Velocity template engine supports the log4j logger, amongst others. By default, Velocity will try to use the Jakarta Avalon Logkit. To change this behavior, you must either:

The latter option is preferable.

That just gets Velocity using Log4j. Velocity will still send its log data to a file named velocity.log. To get Velocity to use the Tomcat system log instead, add the following directive to your velocity.properties file: 

runtime.log.logsystem.log4j.category=org.apache.velocity.runtime.log.SimpleLog4JLogSystem
This is the name of the Log object to use. Just about any class name would work here, but this is consistent with recommended practice. To change the log level of the Velocity logger to INFO, for example, add this line to your log4j.properties file: 
log4j.logger.org.apache.velocity.runtime.log.SimpleLog4JLogSystem=INFO

WEB-INF Logging
You can override the use of the system log by your applications and set up customized logging to other files or places (using remote sockets). This is done by installing the commons-logging.jar and log4j.jar files in the WEB-INF/lib directory of the application in question, along with a custom log4j.properties file in the corresponding WEB-INF/classes directory. However, I have not tried this, so you are on your own with this one.

Logger Identifiers in Apache Code
To see what classes are creating Log objects in an Apache project, run the following command in the corresponding source code directory:

grep 'LogFactory.getLog' `find . -name '*.java'`
Code that uses the older and deprecated log4j API can be scanned thus:

grep 'Category.getInstance' `find . -name '*.java'`

Assuming you know what you are looking for, you can use this information to enable more intense logging in a particular area of an Apache product to pinpoint your problem.

Appendix 1 - Logger Identifiers in Tomcat 5.0.27
The following is a list of logger names in Tomcat 5.0.27:

jakarta-tomcat-connectors:
    org.apache.jk.apr.AprImpl
    org.apache.jk.common.ChannelJni
    org.apache.jk.common.ChannelSocket
    org.apache.jk.common.ChannelUn
    org.apache.jk.common.HandlerDispatch
    org.apache.jk.common.HandlerRequest
    org.apache.jk.common.JkInputStream
    org.apache.jk.common.JkMX
    org.apache.jk.common.JniHandler
    org.apache.jk.common.ModJkMX
    org.apache.jk.common.ModJkMX
    org.apache.jk.common.MsgAjp
    org.apache.jk.common.Shm
    org.apache.jk.config.WebXml2Jk
    org.apache.jk.server.JkCoyoteHandler
    org.apache.jk.server.JkMain
    org.apache.coyote.http11.Http11Processor
    org.apache.coyote.http11.Http11Protocol
    org.apache.naming.ant.JndiProperties
    org.apache.naming.ant.JndiSet
    org.apache.naming.modules.cache.ProxyDirContext
    org.apache.naming.modules.fs.FileDirContext
    org.apache.naming.modules.fs.fsURLContextFactory
    org.apache.naming.modules.java.ContextBindings
    org.apache.naming.util.DomXml
    org.apache.tomcat.util.compat.Jdk14Compat
    org.apache.tomcat.util.compat.JdkCompat
    org.apache.tomcat.util.http.mapper.Mapper
    org.apache.tomcat.util.net.jsse.JSSE14Support
    org.apache.tomcat.util.net.jsse.JSSEImplementation
    org.apache.tomcat.util.net.jsse.JSSESocketFactory
    org.apache.tomcat.util.net.jsse.JSSESupport
    org.apache.tomcat.util.net.puretls.PureTLSSocketFactory
    org.apache.tomcat.util.net.puretls.PureTLSSupport
    org.apache.tomcat.util.net.PoolTcpEndpoint
    org.apache.tomcat.util.net.SSLImplementation
    org.apache.tomcat.util.threads.ThreadPool
    
jakarta-tomcat-catalina:
    org.apache.catalina.authenticator.AuthenticatorBase
    org.apache.catalina.authenticator.BasicAuthenticator
    org.apache.catalina.authenticator.DigestAuthenticator
    org.apache.catalina.authenticator.FormAuthenticator
    org.apache.catalina.core.ApplicationContextFacade
    org.apache.catalina.core.ApplicationDispatcher
    org.apache.catalina.core.ApplicationFilterConfig
    org.apache.catalina.core.ContainerBase
    org.apache.catalina.core.NamingContextListener
    org.apache.catalina.core.StandardContext
    org.apache.catalina.core.StandardContextValve
    org.apache.catalina.core.StandardEngine
    org.apache.catalina.core.StandardHost
    org.apache.catalina.core.StandardHostDeployer
    org.apache.catalina.core.StandardHostValve
    org.apache.catalina.core.StandardPipeline
    org.apache.catalina.core.StandardServer
    org.apache.catalina.core.StandardService
    org.apache.catalina.core.StandardWrapper
    org.apache.catalina.core.StandardWrapperValve
    org.apache.catalina.loader.WebappClassLoader
    org.apache.catalina.loader.WebappLoader
    org.apache.catalina.logger.LoggerBase
    org.apache.catalina.mbeans.GlobalResourcesLifecycleListener
    org.apache.catalina.mbeans.MBeanUtils
    org.apache.catalina.mbeans.ServerLifecycleListener
    org.apache.catalina.realm.JAASMemoryLoginModule
    org.apache.catalina.realm.JAASRealm
    org.apache.catalina.realm.MemoryRealm
    org.apache.catalina.realm.RealmBase
    org.apache.catalina.security.SecurityConfig
    org.apache.catalina.security.SecurityUtil
    org.apache.catalina.session.ManagerBase
    org.apache.catalina.session.ManagerBase
    org.apache.catalina.session.PersistentManagerBase
    org.apache.catalina.startup.Catalina
    org.apache.catalina.startup.ContextConfig
    org.apache.catalina.startup.Embedded
    org.apache.catalina.startup.HostConfig
    org.apache.catalina.startup.TldConfig
    org.apache.catalina.util.ExtensionValidator
    org.apache.catalina.valves.ExtendedAccessLogValve
    org.apache.catalina.valves.ValveBase
    org.apache.coyote.tomcat5.CoyoteAdapter
    org.apache.coyote.tomcat5.CoyoteConnector
    org.apache.coyote.tomcat5.MapperListener
    org.apache.naming.NamingContext
    org.apache.catalina.cluster.deploy.FarmWarDeployer
    org.apache.catalina.cluster.deploy.WarWatcher
    org.apache.catalina.cluster.io.XByteBuffer
    org.apache.catalina.cluster.mcast.McastService
    org.apache.catalina.cluster.mcast.McastServiceImpl
    org.apache.catalina.cluster.session.DeltaManager
    org.apache.catalina.cluster.session.DeltaSession
    org.apache.catalina.cluster.session.SimpleTcpReplicationManager
    org.apache.catalina.cluster.tcp.AsyncSocketSender
    org.apache.catalina.cluster.tcp.Jdk13ReplicationListener
    org.apache.catalina.cluster.tcp.PooledSocketSender
    org.apache.catalina.cluster.tcp.ReplicationListener
    org.apache.catalina.cluster.tcp.ReplicationTransmitter
    org.apache.catalina.cluster.tcp.ReplicationValve
    org.apache.catalina.cluster.tcp.SimpleTcpCluster
    org.apache.catalina.cluster.tcp.SocketSender
    org.apache.catalina.cluster.tcp.TcpReplicationThread
    org.apache.catalina.cluster.tcp.WorkerThread
    org.apache.catalina.cluster.util.SmartQueue
    
jakarta-tomcat-jasper:
    org.apache.jasper.compiler.Compiler
    org.apache.jasper.compiler.JspConfig
    org.apache.jasper.compiler.JspReader
    org.apache.jasper.compiler.JspRuntimeContext
    org.apache.jasper.compiler.TagLibraryInfoImpl
    org.apache.jasper.compiler.TldLocationsCache
    org.apache.jasper.runtime.JspFactoryImpl
    org.apache.jasper.runtime.PageContextImpl
    org.apache.jasper.security.SecurityClassLoad
    org.apache.jasper.servlet.JspServlet
    org.apache.jasper.servlet.JspServletWrapper
    org.apache.jasper.xmlparser.ParserUtils
    org.apache.jasper.EmbeddedServletOptions
    org.apache.jasper.JspC

Appendix 2 - Logger Identifiers in Jakarta Struts 1.1
The following is a list of logger names in Jakarta Struts 1.1:

example:
    org.apache.struts.webapp.example.memory.MemoryDatabasePlugIn
    org.apache.struts.webapp.example.memory.MemoryUserDatabase
    org.apache.struts.webapp.example.EditRegistrationAction
    org.apache.struts.webapp.example.EditSubscriptionAction
    org.apache.struts.webapp.example.LogoffAction
    org.apache.struts.webapp.example.LogonAction
    org.apache.struts.webapp.example.SaveRegistrationAction
    org.apache.struts.webapp.example.SaveSubscriptionAction
    
share:
    org.apache.struts.action.ActionServlet
    org.apache.struts.action.RequestProcessor
    org.apache.struts.actions.DispatchAction
    org.apache.struts.actions.SwitchAction
    org.apache.struts.config.ModuleConfigFactory
    org.apache.struts.taglib.html.BaseHandlerTag
    org.apache.struts.taglib.html.MessagesTag
    org.apache.struts.taglib.tiles.DefinitionTag
    org.apache.struts.taglib.tiles.InsertTag
    org.apache.struts.tiles.actions.DefinitionDispatcherAction
    org.apache.struts.tiles.xmlDefinition.I18nFactorySet
    org.apache.struts.tiles.xmlDefinition.XmlDefinition
    org.apache.struts.tiles.ComponentDefinition
    org.apache.struts.tiles.DefinitionsUtil
    org.apache.struts.tiles.TilesPlugin
    org.apache.struts.tiles.TilesRequestProcessor
    org.apache.struts.tiles.TilesServlet
    org.apache.struts.tiles.TilesUtil
    org.apache.struts.tiles.TilesUtilImpl
    org.apache.struts.upload.CommonsMultipartRequestHandler
    org.apache.struts.upload.DiskMultipartRequestHandler
    org.apache.struts.util.MessageResources
    org.apache.struts.util.MessageResourcesFactory
    org.apache.struts.util.PropertyMessageResources
    org.apache.struts.util.RequestUtils
    org.apache.struts.validator.DynaValidatorActionForm
    org.apache.struts.validator.DynaValidatorForm
    org.apache.struts.validator.FieldChecks
    org.apache.struts.validator.ValidatorActionForm
    org.apache.struts.validator.ValidatorForm
    org.apache.struts.validator.ValidatorPlugIn
    
test:
    test.org.apache.struts.mock.MockServletContext
    
tiles-documentation:
    tiles-documentation.org.apache.struts.webapp.tiles.rssChannel.Channels

Technical Tips