Log Toolkit

This package provides a set of tools to manipulate and maintain webserver logfiles.

Tools

elfsort - Sort extended log format (ELF) log files by date.

        Logfiles from different servers may be combined and
        sorted using this tool. It's useful for brain dead
        log analysers that are incapable of handling logfiles
        whose entries out out of date order, which can result
        for example when logfiles from two separate servers
        are concatenated.

        An attempt is made to do the sorting of the logfiles
        in a memory efficient way, so that logfiles may be
        sorted that are larger than available memory.

w3ctoelf - Convert W3C format logfiles to ELF.

        Logfiles created using the W3C format of Microsoft
        IIS may be converted to extended log format (as used
        by Apache) using this tool. The tool parses the
        "Fields" entry at the beginning of the logfile to
        determine the log format, and lines are created in
        ELF format. W3C logfile formats may be changed mid-file
        without a problem.

        Lines without sufficient information to build a valid
        ELF log line will be ignored. The number of lines
        ignored or corrupt are reported to stderr once scanning
        is complete.

nstoelf - Convert Netscape format logfiles to ELF.

        Logfiles created using Netscape Enterprise or Netscape
        Proxy server can be converted to extended log format (as
        used by Apache) using this tool. The tool parses the
        first line of the logfile to determine the log format,
        and lines are created in ELF format. Netscape logfile
        formats may be changed mid-file without a problem.

        Lines without sufficient information to build a valid
        ELF log line will be ignored. The number of lines
        ignored or corrupt are reported to stderr once scanning
        is complete.

elffilter - Perform logfile filtering.

        Logfile lines can be included or excluded in the output
        depending on whether certain conditions are met or not
        met.

        Lines may be scanned for by one or more specific URIs
        and lines that match the URI prefix will be either
        included or excluded.

elfstrip - Strips non-ELF log lines from a logfile.

        Logfile lines that cannot be parsed as an ELF format
        log entry can be filtered from a log file. Optionally
        these corrupt lines can be redirected through stderr
        to analyse them.

urltouri - Strips the sitename from a URL.

        A URL can be converted to a URI using this tool. If
        a sitename like http://foo/bar is given it will be
        converted to /bar. URLs can be given on the command
        line or in a file.

        It is useful within scripts to determine the directory
        and filename portion of a URL without any Perl magic.

Installation

The package is built using GNU autoconf, so configuration and installation should be as simple as:

        ./configure
        make
        make install

For a more detailed explanation, see the file INSTALL.

Help

The only guarantee that comes with this package is that if it breaks you get to keep both pieces. However, if you need help send mail to minfrin@sharp.fm and I'll see if I can be of assistance.

The package was developed under Solaris, but should work under Linux. If you can make it work under other systems, send me patches :)

Legal Stuff:

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA