Back to files

A BibTeX system in Common Lisp

This project provides a replacement for the BibTeX program, written in Common Lisp. The aim is to enable the user to format bibliographic entries using Common Lisp programs, rather than using the stack language of BibTeX style files. It is released under the GNU General Public License (see the file COPYING).

The project pages are at

http://www.freesoftware.fsf.org/cl-bibtex/

There is a mailing list `cl-bibtex-users@mail.freesoftware.fsf.org'. You can subscribe to the list at

http://mail.freesoftware.fsf.org/mailman/listinfo/cl-bibtex-users

Reasons

A powerful tool is needed for dealing with citations in scientific documents. BibTeX is good for formatting bibliographies, but customizing the format is a pain because it requires writing/changing a program in BAFLL (BibTeX Anonymous Forth-Like Language -- Drew McDermott in comp.lang.lisp), which looks like this:

FUNCTION {tie.or.space.connect}
{ duplicate$ text.length$ #3 <

{ "~" }
{ " " }
if$
swap$ * *
}

Other citation-related processing like producing a list of authors and citation indices is usually done with a combination of BibTeX, TeX macros, and ad-hoc scripting with Perl or AWK.

The new cl-bibtex system aims to replace all this.

Status

Alpha. The system already has:

• A reader for BibTeX style files (.bst),
• a reader and a writer for bibliography databases (.bib),
• a reader for .aux files,
• an implementation of BibTeX's built-in functions (like for parsing and formatting names, etc.)
• an interpreter for the stack language of BibTeX style files,
• a compiler that transforms BibTeX style files into comprehensible Common Lisp programs (using type analysis)

Missing so far: see the TODO file

Where to start

Compile and load all Lisp files, using bibtex.system.

The function BIBTEX is like the bibtex program. Pass the file name STEM; BIBTEX will read STEM.aux, interpret a BST file, and write STEM.bbl:

(bibtex-compiler:bibtex "ibm-theory")

In fact, when the STEM.aux file requests a bibliography style, BIBTEX first looks whether a Lisp function that implements the style has been defined using DEFINE-BIBTEX-STYLE. Otherwise, it tries to find (using kpathsearch) and load a Lisp file named "STYLE.lbst"; it is supposed to define the style using DEFINE-BIBTEX-STYLE. Finally, BIBTEX tries to find "STYLE.bst" and interpret it.

A BibTeX style implemented in Lisp is supposed to read all bibliography files in BIB-FILES and to write the formatted bibliography to the stream BBL-OUTPUT. The package BIBTEX-RUNTIME contains useful routines for reading AUX files, formatting names, etc. An important function is READ-ALL-BIB-FILES-AND-COMPUTE-BIB-ENTRIES.

You can invoke the BST-to-Common-Lisp compiler with the function COMPILE-BST-FILE:

        (bibtex-compiler:compile-bst-file (kpathsea:find-file "amsalpha.bst")
                                          "amsalpha.lbst")

The resulting Lisp file is a readable Common Lisp version of the BST file, which can be run on the AUX file to produce a BBL file, without using the BST interpreter:

        (load "amsalpha.lbst" 
              :if-source-newer :compile)            ; calls DEFINE-BIBTEX-STYLE

        (bibtex-compiler:bibtex "ibm-theory")       ; now uses the Lisp style

There is also a little shell script "bibtex" for invoking CL-BIBTEX from the shell; it uses the "run-lisp" script from CLOCC.

Restrictions of the compiler

The BST->CL compiler works by analyzing the type of the wizard-defined functions (i.e., those defined by a FUNCTION command in the BST file). The compiler tries to find out how many "literals" (values) a function pops from the stack and how many it pushes and determines their types (BOOLEAN, INTEGER, STRING).

• If a wizard-defined function tries to pop a function literal that it hasn't pushed, or leaves a function literal on the stack, you lose. (In other words, you can't define higher-order BST functions.)
• The two branches of an IF$ function must deliver the same net number of values. (As a special exception, the type of IF$ within a WHILE$ body which occurs in the FORMAT.NAMES function of the standard styles is also supported. This is reported with a warning message.)

On the compilation of a language that no-one can read or write

The language of BibTeX style files (BAFLL) is expected to be read or written by "wizards" only. (This is how the original BibTeX calls the authors of BibTeX style files.) In fact, when casual users try to customize a BibTeX style file, they often come up with slightly broken styles that appear to work but which fail in corner cases. The reason is that programs written in BAFLL can hardly be read, written, or even debugged.

This causes an extra difficulty for the compiler, because broken corner cases tend to misbehave on the stack, rendering the function unanalyzable, hence uncompilable. The compiler now contains a heuristic that tries to fix these situations, but it may fail. In this case, you should try to understand the error messages, fix the BibTeX style file accordingly, and try compiling it again.

Implementation dependencies

It does not run on some 0.7.? version of SBCL because it does not like (COPY-READTABLE NIL). SBCL 0.7.9 seems to be fine.

I noticed that CMUCL 3.0.8 18c+ does not like (PEEK-CHAR T STREAM NIL #\Space); it won't skip over whitespace when the EOF character is whitespace? (I don't use this any more in CL-BibTeX.)

CLISP 2.30:

1. When I invoke COMPILE-BST-FILE, CLISP says "Lisp stack overflow. RESET"; this seems to be caused by the pprint-dispatch function for DEFINE-BIBTEX-STYLE.
2. The format directive ~<...~:> (logical block) does not seem to work properly. (I have used reader conditionalization to make CL-BibTeX work with CLISP.)

Have fun!

-- Matthias Koeppe <mkoeppe@mail.math.uni-magdeburg.de>