LogMiner http://logminer.sourceforge.net

What is it?

LogMiner is a powerful log analysis tool for Apache and IIS, or other servers using the Extended W3C format. It can create reports similar to those generated by Webalizer (http://www.mrunix.net/webalizer/), plus others like the navigation graph.

Differently than Webalizer, LogMiner doesn't generate static html files: I reckon this may be a problem for someone, but I didn't like ending up with hundreds of files which were mostly unused, due to the fact that Webalizer indexes only the last 12 months...

Also, using a PostgreSQL backend, there's a better flexibility and it's possible to create easily new reports and apply them to old data.

Who did it?

Simone Tellini, <tellini@users.sourceforge.net> http://tellini.info/

License

The GPL version 2 license applies to LogMiner.

Show your support

If you use this software in a production environment and/or you wish to show your support, you can get me something off one of the Amazon wish-lists of mine, located at

http://www.amazon.co.uk/exec/obidos/registry/1K4OWZ581SIRE/ref%3Dwl%5Fs%5F3/026-2575462-0900418

Requirements

GCC 3.4
PHP 4.3.x or better (it's been tested only on PHP 5.x anyway)
JPGraph (if not installed, you won't see graphs)
graphviz (needed to generate the navigation graphs)
PostgreSQL 8.x
libpqxx (tested with version 2.5.5)

plpgsql must be enabled in the database that will be used.

Installation

To setup the DB, create a database or a schema and use psql to load the file db/schema.sql. E.g.

psql -U logminer logminer < db/schema.sql

"./configure; make install" should compile and install the logminer exe in your bin directory.

The web directory contains the PHP part of the application: copy it wherever you like and point your Apache to it. Don't forget to edit the file common/config.php.sample and rename it to common/config.php

Parsing log files

Log files are parsed by logminer. You need to pass at least one parameter on the command line: the name of the config file to use. An example configuration file is shown below:

---8<---8<--- [Database]

DB          = logminer
Host        = localhost
User        = logminer

Password = logminer

[Logging]
; log to stderr?
UseStdErr = 0
; log to syslog?
UseSysLog = 1

[Logs]
; which extensions should be considered a "page"? PageTypes = html htm php php3 pl cgi phtml ; Should client IP's be resolved?
ResolveHosts = 1

[Site site1]
Log = /usr/local/apache/logs/site1_access.log ; type can be Apache or EW3C
; EW3C is the Extended W3C format, used by IIS for example Type = Apache
; aliases for this site, used to determine which are internal referrers

Hosts       = www.site1.com web.site1.com
                              ---8<---8<---

You can also pass a second parameter to logminer: the name of a site defined in the configuration. In this case, logminer will parse only that site's log, otherwise it will parse all the logs defined in the config.

About IIS or other W3C logs

LogMiner can parse IIS log files in the Extended W3C format, as long as you specify the "Type = EW3C" option in the Site definition.

The following are the fields that LogMiner understands:

        date
        time
        c-ip
        cs-username
        cs-method
        cs-uri-stem
        cs-uri-query
        cs-version
        sc-bytes
        sc-status
        cs(Referer)
        cs(User-Agent)

I suggest you enable all of them in IIS' configuration to get the max out of LogMiner.