big documenttion update, bring docs in sync with reality

1 files changed, 192 insertions, 39 deletions

diff --git a/doc/ulogd.sgml b/doc/ulogd.sgml
index f8d95ec..e8bbeb6 100644
--- a/doc/ulogd.sgml
+++ b/doc/ulogd.sgml
@@ -1,17 +1,17 @@
 <!doctype linuxdoc system>
 
-<!-- $Id: ulogd.sgml,v 1.6 2002/06/13 12:56:16 laforge Exp $ -->
+<!-- $Id: ulogd.sgml,v 1.7 2003/02/08 12:21:36 laforge Exp $ -->
 
 <article>
 
 <title>ULOGD - the Userspace Logging Daemon</title>
 <author>Harald Welte &lt;laforge@gnumonks.org&gt</author>
-<date>Revision $Revision: 1.6 $, $Date: 2002/06/13 12:56:16 $</date>
+<date>Revision $Revision: 1.7 $, $Date: 2003/02/08 12:21:36 $</date>
 
 <abstract>
 This is the documentation for <tt>ulogd</tt>, the Userspace logging daemon.
-ulogd makes use of the Linux 2.4 firewalling subsystem (netfilter) and the
-ULOG target for netfilter.
+ulogd makes use of the Linux >= 2.4.x packet filter subsystem (iptables) and
+the ULOG target for iptables
 </abstract>
 
 <toc>
@@ -48,15 +48,12 @@ Nobody knows what strange network protocols are out there :) Flexibility
 depends on the communication between the output of the logging plugins 
 and input of the output plugins.
 <p>
-Rusty advised me to use some kind of type-key-value triples, but I think
-this is the total overkill and is too complicated for me to implement it 
-in a reasonable short period of time. (3 hours later) Hmm... Rusty finally 
-convinced me to use linked lists of type-key-value triples - and it wasn't 
-that difficult.
+Rusty advised me to use some kind of type-key-value triples, which is in fact
+what I implemented.
 <p>
-Another issue is, of course, performance. Up to ulogd 0.3, ulogd did several
+One issue is, of course, performance. Up to ulogd 0.3, ulogd did several
 linked list iterations and about 30 malloc() calls _per packet_. This 
-changed with the new 0.9 revision:
+changed with the new &gt;= 0.9 revisions:
 <itemize>
 <item>Not a single dynamic allocation in the core during runtime. 
 Everything is pre-allocated at start of ulogd to provide the highest
@@ -84,7 +81,7 @@ You only need to read this chapter if you have a 2.4.x kernel &lt;=
 In order to put the ipt_ULOG module into your kernel source,you need the latest
 iptables package, or even better: the latest CVS snapshot. A description how to
 obtain this is provided on the netfilter
-homepage <URL URL="http://netfilter.gnumonks.org">.
+homepage <URL URL="http://www.netfilter.org/">.
 <p>
 To run patch-o-matic, just type 
 <tscreen><verb>
@@ -95,7 +92,7 @@ in the userspace directory of netfilter CVS.
 <sect1>ulogd
 <sect2>Recompiling the source
 <p>
-Download the ulogd package from <URL URL="http://www.gnumonks.org/ftp/pub/netfilter/"> and
+Download the ulogd package from <URL URL="http://ftp.netfilter.org/pub/ulogd/"> and
 untar it. 
 <p>
 If you want to build ulogd with MySQL support, type './configure --with-mysql'. You may also have to specify the path of the mysql libraries using '--with-mysql=path'. To build ulogd without MySQL support, just use './configure'.
@@ -104,17 +101,18 @@ To compile and install the program, call 'make install'.
 
 <sect2>Using a precompiled package
 <p>
-I also provide a SRPM, which should compile on almost any rpm-based distribution. It is available at <URL URL="http://www.gnumonks.org/ftp/pub/netfilter/">
+I also provide a SRPM, which should compile on almost any rpm-based distribution. It is available at <URL URL="http://ftp.netfilter.org/pub/ulogd/">
 <p>
 Just download the package and do the usual 'rpm --rebuild &lt;file&gt;'.
 
 <sect>Configuration
-<sect1>netfilter
+<sect1>iptables ULOG target
+<sect2>Quick Setup
 <p>
 Just add rules using the ULOG target to your firewalling chain. A very basic
 example:
 <tscreen><verb>
-iptables -A FORWARD -j ULOG --ulog-nlgroup 32 --prefix foo 
+iptables -A FORWARD -j ULOG --ulog-nlgroup 32 --ulog-prefix foo 
 </verb></tscreen>
 <p>
 To increase logging performance, try to use the
@@ -127,36 +125,133 @@ kernel schedules ulogd only once every 20 packets. All 20 packets are then
 processed by ulogd. This reduces the number of context switches between kernel
 and userspace.
 <p>
-Of course you can combine the ULOG target with the different netfilter match modules.
-For a more detailed description, have a look at the netfilter HOWTO's, available on 
-the netfilter homepage.
+Of course you can combine the ULOG target with the different netfilter match
+modules.  For a more detailed description, have a look at the netfilter
+HOWTO's, available on the netfilter homepage.
+<sect2>ULOG target reference
+<p>
+<descrip>
+<tag>--ulog-nlgroup N</tag>
+The number of the netlink multicast group to which ULOG'ed packets are sent.
+You will have to use the same group number in the ULOG target and ulogd in
+order to make logging work.
+<tag>--ulog-cprange N</tag>
+Copyrange.  This works like the 'snaplen' paramter of tcpdump.  You can specify
+a number of bytes up to which the packet is copied.  If you say '40', you will
+receive the first fourty bytes of every packet. Leave it to '0'
+<tag>--ulog-qthreshold N</tag>
+Queue threshold.  If a packet is matched by the iptables rule, and already N
+packets are in the queue, the queue is flushed to userspace.  You can use this
+to implement a policy like: Use a big queue in order to gain high performance,
+but still have certain packets logged immediately to userspace.
+<tag>--ulog-prefix STRING</tag>
+A string that is associated with every packet logged by this rule.  You can use
+this option to later tell from which rule the packet was logged.
+</descrip>
+
+<sect2>ipt_ULOG module parameters
+<p>
+The ipt_ULOG kernel module has a couple of module loadtime parameters which can
+(and should) be tuned to accomodate the needs of the application:
+<descrip>
+<tag>nlbufsiz N</tag>
+Netlink buffer size. A buffer of the specified size N is allocated for every
+netlink group that is used.  Please note that due to restrictions of the kernel
+memory allocator, we cannot have a buffer size &gt; 128kBytes.  Larger buffer
+sizes increase the performance, since less kernel/userspace context switches
+are needed for the same amount of packets.  The backside of this performance
+gain is a potentially larger delay. The default value is 4096 bytes, which is
+quite small.
+<tag>flushtimeout N</tag> 
+The flushtimeout determines, after how many clock ticks (on alpha: 1ms, on
+x86 and most other platforms: 10ms time units) the buffer/queue is to be
+flushed, even if it is not full.  This can be used to have the advantage of a
+large buffer, but still a finite maximum delay introduced.  The default value
+is set to 10 seconds.
+</descrip>
+Example:
+<tscreen><verb>
+modprobe ipt_ULOG nlbufsiz=65535 flushtimeout=100
+</verb></tscreen>
+This would use a buffer size of 64k and a flushtimeout of 100 clockticks (1 second on x86).
 
 <sect1>ulogd
 <p>
-All configurable parameters of ulogd are in the configfile '/etc/ulogd.conf'
+ulogd is what this is all about, so let's describe it's configuration...
+<sect2>ulogd configfile syntax reference
+<p>
+All configurable parameters of ulogd are in the configfile, typically located
+at '/etc/ulogd.conf'.
 <p>
 The following configuration parameters are available:
 <descrip>
 <tag>nlgroup</tag>
-The netlink multicast group, which ulgogd should bind to. This is the same as given with the '--ulog-nlgroup' option to iptables.
+The netlink multicast group, which ulgogd should bind to. This is the same as
+given with the '--ulog-nlgroup' option to iptables.
 <tag>logfile</tag>
 The main logfile, where ulogd reports any errors, warnings and other unexpected 
 conditions.
 <tag>loglevel</tag>
-This specifies, how verbose the logging to logfile is. Currently defined loglevels are: 1=debug information, 3=informational messages, 5=noticable exceptional conditions, 7=error conditions, 8=fatal errors, program abort.
+This specifies, how verbose the logging to logfile is. Currently defined
+loglevels are: 1=debug information, 3=informational messages, 5=noticable
+exceptional conditions, 7=error conditions, 8=fatal errors, program abort.
 <tag>plugin</tag>
-This option is followed by a filename of a ulogd plugin, which ulogd shold load upon initialization. This option may appear more than once.
+This option is followed by a filename of a ulogd plugin, which ulogd shold load
+upon initialization. This option may appear more than once.
+<tag>bufsize</tag>
+Size of the receive buffer. You should set this to at least the size of the
+kernel buffer (nlbufsiz parameter of the ipt_ULOG module).
+</descrip>
+<sect2>ulogd commandline option reference
+<p>
+Apart from the configfile, there are a couple of commandline options to ulogd:
+<descrip>
+<tag>-h --help</tag>
+Print a help message about the commandline options.
+<tag>-V --version</tag>
+Print version information about ulogd.
+<tag>-d --daemon</tag> 
+For off into daemon mode.  Unless you are debugging, you will want to use this
+most of the time.
+<tag>-c --configfile</tag>
+Using this commandline option, an alternate config file can be used.  This is
+important if multiple instances of ulogd are to be run on a single machine.
 </descrip>
 
 <sect>Available plugins
 <p>
-ulogd comes with the following plugins:
-<descrip>
-<tag>ulogd_BASE.so</tag>
-Basic interpreter plugin for nfmark, timestamp, mac address, ip header, tcp header, udp header, icmp header, ah/esp header.
-<tag>ulogd_PWSNIFF.so</tag>
-Example interpreter plugin to log plaintext passwords as used with FTP and POP3. Don't blame me for writing this plugin! The protocols are inherently insecure, and there are a lot of other tools for sniffing passwords... it's just an example.
-<tag>ulogd_OPRINT.so</tag>
+It is important to understand that ulogd without plugins does nothing.  It will receive packets, and do nothing with them.
+<p>
+There are two kinds of plugins, interpreter and output plugins.  Interpreter
+plugins parse the packet, output plugin write the interpreted information to
+some logfile/database/...
+
+<sect1>Interpreter plugins
+<p>
+ulogd comes with the following interpreter plugins:
+<sect2>ulogd_BASE.so
+<p>
+Basic interpreter plugin for nfmark, timestamp, mac address, ip header, tcp
+header, udp header, icmp header, ah/esp header... Most people will want to load
+this very important plugin.
+<sect2>ulogd_PWSNIFF.so
+<p>
+Example interpreter plugin to log plaintext passwords as used with FTP and
+POP3. Don't blame me for writing this plugin! The protocols are inherently
+insecure, and there are a lot of other tools for sniffing passwords... it's
+just an example.
+<sect2>ulogd_LOCAL.so
+<p>
+This is a 'virtual interpreter'.  It doesn't really return any information on
+the packet itself, rather the local system time and hostname.  Please note that
+the time is the time at the time of logging, not the packets receive time.
+
+<sect1>Output plugins
+<p>
+ulogd comes with the following output plugins:
+
+<sect2>ulogd_OPRINT.so
+<p>
 A very simple output module, dumping all packets in the format
 <tscreen><verb>
 ===>PACKET BOUNDARY
@@ -166,25 +261,30 @@ key=value
 ===>PACKET BOUNDARY
 ...
 </verb></tscreen>
-to a file.
+to a file.  The only useful application is debugging.
 <p>The module defines the following configuration directives:
 <descrip>
 <tag>dumpfile</tag>
-The filename where it should log to. The default is <tt>/var/log/ulogd.pktlog</tt>
+The filename where it should log to. The default is
+<tt>/var/log/ulogd.pktlog</tt>
 </descrip>
 
-<tag>ulogd_LOGEMU.so</tag>
-An output module which tries to emulate the old syslog-based LOG targed as far as possible. Logging is done to a textfile instead of syslog, though.
+<sect2>ulogd_LOGEMU.so
+<p>
+An output module which tries to emulate the old syslog-based LOG targed as far
+as possible. Logging is done to a seperate textfile instead of syslog, though.
 <p>
 The module defines the following configuration directives:
 <descrip>
-<tag>syslogfile</tag>The filename where it should log to. The default is <tt>/var/log/ulogd.syslogemu</tt>
-</descrip>
-<descrip>
-<tag>syslogsync</tag>Set this to 1 if you want to have your logfile written synchronously. This may reduce performance, but makes your log-lines appear immediately.  The default is <tt>0</tt>
+<tag>syslogfile</tag>The filename where it should log to. The default is
+<tt>/var/log/ulogd.syslogemu</tt>
+<tag>syslogsync</tag>Set this to 1 if you want to have your logfile written
+synchronously. This may reduce performance, but makes your log-lines appear
+immediately.  The default is <tt>0</tt>
 </descrip>
 
-<tag>ulogd_MYSQL.so</tag>
+<sect2>ulogd_MYSQL.so
+<p>
 An output plugin for logging into a mysql database. This is only compiled if
 you have the mysql libraries installed, and the configure script was able to
 detect them. (that is: --with-mysql was specified for ./configure) <p>
@@ -217,7 +317,57 @@ Name of the mysql user
 Password for mysql
 </descrip>
 
+<sect2>ulogd_PGSQL.so
+<p>
+An output plugin for logging into a postgresql database. This is only compiled
+if you have the mysql libraries installed, and the configure script was able to
+detect them. (that is: --with-pgsql was specified for ./configure) <p>
+
+The plugin automagically inserts the data into the configured table; It
+connects to mysql during the startup phase of ulogd and obtains a list of the
+columns in the table. Then it tries to resolve the column names against keys of
+interpreter plugins. This way you can easly select which information you want
+to log - just by the layout of the table.  <p>
+
+If, for example, your table contains a field called 'ip_saddr', ulogd will
+resolve this against the key 'ip.saddr' and put the ip address as 32bit
+unsigned integer into the table.  <p>
+
+You may want to have a look at the file '<tt>doc/mysql.table</tt>' as an
+example table including fields to log all keys from ulogd_BASE.so. Just delete
+the fields you are not interested in, and create the table.  <p>
+
+The module defines the following configuration directives:
+<descrip>
+<tag>pgsqltable</tag>
+Name of the table to which ulogd should log
+<tag>pgsqldb</tag>
+Name of the mysql database
+<tag>pgsqlhost</tag>
+Name of the mysql database host
+<tag>pgsqluser</tag>
+Name of the mysql user
+<tag>pgsqlpass</tag>
+Password for mysql
+</descrip>
+
+<sect2>ulogd_PCAP.so
+<p>
+An output plugin that can be used to generate libpcap-style packet logfiles.
+This can be useful for later analysing the packet log with tools like tcpdump
+or ethereal.
+
+The module defines the following configuration directives:
+<descrip>
+<tag>pcapfile</tag>
+The filename where it should log to.  The default is:
+<tt>/var/log/ulogd.pcap</tt>
+<tag>pcapsync</tag>
+Set this to <tt>1</tt> if you want to have your pcap logfile written
+synchronously.  This may reduce performance, but makes your packets appear
+immediately in the file on disk.  The default is <tt>0</tt>
 </descrip>
+
 <sect> QUESTIONS / COMMENTS
 <p>
 All comments / questions / ... are appreciated.
@@ -227,5 +377,8 @@ Just drop me a note to laforge@gnumonks.org
 Please note also that there is now a mailinglist, ulogd@lists.gnumonks.org. 
 You can subscribe at
 <URL URL="http://lists.gnumonks.org/mailman/listinfo/ulogd/">
+<p> 
+The preferred method for reporting bugs is the netfilter bugzilla system,
+available at <URL URL="http://bugzilla.netfilter.org/">.
 
 </article>