summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* IPFIX: Introduce template record supportHEADmasterAnder Juaristi2019-04-304-37/+139
| | | | | | | | | | | | | | | | | | | | This commit adds the ability to send template records to the remote collector. In addition, it also introduces a new configuration parameter 'send_template', which tells when template records should be sent. It accepts the following string values: - "once": Send the template record only the first time (might be coalesced with data records). - "always": Send the template record always, with every data record that is sent to the collector (multiple data records might be sent together). - "never": Assume the collector knows the schema already. Do not send template records. If omitted, the default value for 'send_template' is "once". Signed-off-by: Ander Juaristi <a@juaristi.eus> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* IPFIX: Add IPFIX output pluginAnder Juaristi2019-04-309-550/+747
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds an IPFIX output plugin to ulogd2. It generates NetFlow/IPFIX traces and sends them to a remote server (collector) via TCP or UDP. Based on original work by Holger Eitzenberger <holger@eitzenberger.org>. How to test this ---------------- I am currently testing this with the NFCT input and Wireshark. Place the following in ulogd.conf: # this will print all flows on screen loglevel=1 # load NFCT and IPFIX plugins plugin="/lib/ulogd/ulogd_inpflow_NFCT.so" plugin="/lib/ulogd/ulogd_output_IPFIX.so" stack=ct1:NFCT,ipfix1:IPFIX [ct1] netlink_socket_buffer_size=217088 netlink_socket_buffer_maxsize=1085440 accept_proto_filter=tcp,sctp [ipfix1] oid=1 host="127.0.0.1" #port=4739 #send_template="once" I am currently testing it by launching a plain NetCat listener on port 4739 (the default for IPFIX) and then running Wireshark and see that it dissects the IPFIX/NetFlow traffic correctly (obviously this relies on the Wireshark NetFlow dissector being correct). First: nc -vvvv -l 127.0.0.1 4739 Then: sudo ulogd -vc ulogd.conf Signed-off-by: Ander Juaristi <a@juaristi.eus> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* ulogd: fix build with musl libcCameron Norman2018-11-011-0/+1
| | | | | | | | The attached patch fixes building ulogd2 with musl libc. It is being used on Void Linux right now. Closes: https://bugzilla.netfilter.org/show_bug.cgi?id=1278 Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* ulogd: json: send messages to a remote host / unix socketAndreas Jaggi2018-06-012-21/+281
| | | | | | | | Extend the JSON output plugin so that the generated JSON stream can be sent to a remote host via TCP/UDP or to a local unix socket. Signed-off-by: Andreas Jaggi <andreas.jaggi@waterwave.ch> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* remove ulogd2.rotate and ulogd2.spec from Makefile.amulogd-2.0.7Arturo Borrero Gonzalez2018-04-271-1/+1
| | | | | Fixes: 42b384044dab ("ulogd2: cleanup downstream files") Signed-off-by: Arturo Borrero Gonzalez <arturo@netfilter.org>
* Set release number to 2.0.7.Arturo Borrero Gonzalez2018-04-271-1/+1
| | | | Signed-off-by: Arturo Borrero Gonzalez <arturo@netfilter.org>
* ulogd2: cleanup downstream filesArturo Borrero Gonzalez2018-04-113-187/+0
| | | | | | | | | | | | These files are outdated and they belong to downstream users (distributions). Providing outdated and unmaintained files here serves no purpose other than confusing users and annoy packagers. If an user is using ulogd2 directly from the source tarball, I would expect it to be proficient enough to generate these files by itself. Signed-off-by: Arturo Borrero Gonzalez <arturo@netfilter.org> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* ulogd: load all plugins by defaultArturo Borrero Gonzalez2018-01-083-20/+83
| | | | | | | | | | | | | | | | | | | | | | | | | | | This new configuration behaviour option eases a bit the configuration of ulogd2 by allowing to load all plugins in one go, without having to know their full path. Choosing concrete plugins and using full path for them is great for some environmnets, but I don't think it's a common case. The common case is to load all plugins, even ignoring where do they live in the filesystem. Even worse, the full path may be architecture-dependant, which makes copying the ulogd.conf file between machines unnecesarily complex. To experiment this new behaviour, don't put any 'plugin=' directive in the config file. Plugins will be loaded from a default directory, choosen at build/configure time (--with-ulogd2libdir). If no specified, this is something like '/usr/local/lib/ulogd/'. This new configuration option doesn't implement any special logic. We simply open the dir and try to load all files ending with '.so'. The log message level for plugins loading is increased so users can see by default which plugins are loaded. Signed-off-by: Arturo Borrero Gonzalez <arturo@netfilter.org> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* ulogd: use a RT scheduler by defaultArturo Borrero Gonzalez2017-09-201-0/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Is common that ulogd runs in scenarios where a lot of packets are to be logged. If there are more packets than ulogd can handle, users can start seing log messages like this: ulogd[556]: We are losing events. Please, consider using the clauses \ `netlink_socket_buffer_size' and `netlink_socket_buffer_maxsize' Which means that Netlink buffer overrun have happened. There are several approaches to prevent this situation: * in the ruleset, limit the amount of packet queued for log * in the ruleset, instruct the kernel to use a queue-threshold * from userspace, increment Netlink buffer sizes * from userspace, configure ulogd to run as high priority process The first 3 method can be configured by users at runtime. This patch deals with the last method. SCHED_RR is configured by default, with no associated configuration parameter for users, since I believe this is common enough, and should produce no harm. A similar approach is used in the conntrackd daemon. Signed-off-by: Arturo Borrero Gonzalez <arturo@netfilter.org> Acked-by: Eric Leblond <eric@regit.org> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* ip2bin: fix plugin link for some compilerEric Leblond2017-07-021-1/+1
| | | | | | | Declaring a function inline and building with -O0 was causing the following message: undefined symbol: uint32_to_ipv6 By declaring the function as static we fix the problem.
* ulogd: fix crash when plugin version are incorrectEric Leblond2017-07-021-0/+1
| | | | | Format string in error message had more arguments than given and it was resulting in a crash at start.
* Set release number to 2.0.6.Eric Leblond2017-05-161-1/+1
|
* rotate all default output filesKaarle Ritvanen2017-05-161-1/+1
| | | | Signed-off-by: Kaarle Ritvanen <kaarle.ritvanen@datakunkku.fi>
* harmonize log file defaults with ulogd.confKaarle Ritvanen2017-05-165-6/+6
| | | | Signed-off-by: Kaarle Ritvanen <kaarle.ritvanen@datakunkku.fi>
* ulogd.conf: fix incorrect stackEric Leblond2017-03-211-1/+1
| | | | | | | The stack was not correctly defined triggering an error on type conflict. Signed-off-by: Eric Leblond <eric@regit.org>
* ulogd: use strncpy instead of memcpyEric Leblond2017-03-211-1/+1
| | | | | | | | | On some architecture, ulogd is not starting due to a crash in memcpy. This patch switches to strncpy to avoid the problem. Reported-by: Alexandru Ardelean <ardeleanalex@gmail.com> Signed-off-by: Eric Leblond <eric@regit.org>
* ulogd: add automake optionEric Leblond2017-03-211-1/+1
| | | | | | This option will be needed for future version of automake. Signed-off-by: Eric Leblond <eric@regit.org>
* ulogd: fix crash when ipv4 packet is truncatedLiping Zhang2016-10-171-1/+2
| | | | | | | | | | | If ipv4 packet is truncated, we should not try to dereference the iph pointer. Otherwise, if the user add such iptables rules "-j NFLOG --nflog-size 0", we will dereference the NULL pointer and crash may happen. Reported-by: Chris Caputo <ccaputo@alt.net> Signed-off-by: Liping Zhang <liping.zhang@spreadtrum.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* ulogd: fix indentation in acinclude.m4Eric Leblond2016-03-311-70/+66
| | | | | | Some imbricated tests were not indented. Signed-off-by: Eric Leblond <eric@regit.org>
* ulogd: fix cross compilation errors with mysql_configHelmut Schaa2016-03-311-2/+2
| | | | | | | | | | | When cross-compiling ulogd, mysql_config and pg_config will return build host configuration not build target configuration. This leads to build failures if mysql_config is installed on the host system but mysql is not available on the build target. Fix this by not using mysql_config and pg_config for cross-compilation. Signed-off-by: Helmut Schaa <helmut.schaa@googlemail.com>
* ulogd: add missing newline in log messageEric Leblond2016-02-051-1/+1
|
* ulogd: restructures signal handling by self-pipe trickHironobu Ishii2016-02-051-7/+125
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ulogd had a critical bug that is calling Async-Signal-Unsafe functions in signal hander context. - Most of libc functions like fopen(), malloc() are Async-Signal-Unsafe. So you should not call these functions in signal handler context. - Calling pluginstances in signal handler context is danger. For implementer of pluginstances, it is very hard to recognize their functions are called in signal handler context. To solve the issue, I restructured signal handling by self-pipe trick. For more detail on self-pipe trick, please see the following. https://lwn.net/Articles/177897/ This patch will solve various symptoms like following. - Deadlock - Segmentation fault caused by libc management data corruption, - Other unpredictable behavior. Deadlock example ================ This bug was already filed at: https://bugzilla.netfilter.org/show_bug.cgi?id=1030 I also hit this bug. The backtrace of this issue is following. In this case, main thread was calling ctime(), and signal handler called localtime_r(). That caused the dead lock while getting tzset_lock in __tz_convert(). Because vsyslog() is Async-Signal-Unsafe function, we cannot call this function in signal handler context. (gdb) bt #0 __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 #1 0x00007f3c3fc7e4ac in _L_lock_2462 () at tzset.c:621 #2 0x00007f3c3fc7e2e7 in __tz_convert (timer=0x7f3c3ff8bf00 <tzset_lock>, timer@entry=0x7fffcfa923b8, use_localtime=use_localtime@entry=1, tp=tp@entry=0x7fffcfa92400) at tzset.c:624 #3 0x00007f3c3fc7c28d in __localtime_r (t=t@entry=0x7fffcfa923b8, tp=tp@entry=0x7fffcfa92400) at localtime.c:32 #4 0x00007f3c3fcbf1ba in __GI___vsyslog_chk (pri=<optimized out>, flag=1, fmt=0x406fa8 "signal received, calling pluginstances\n", ap=0x7fffcfa924a0) at ../misc/syslog.c:199 #5 0x00000000004037b5 in __ulogd_log () #6 0x00000000004047be in signal_handler () #7 <signal handler called> #8 0x00007f3c3fcb62f5 in __GI___xstat (vers=<optimized out>, name=0x7f3c3fd4b2c3 "/etc/localtime", buf=0x7fffcfa92c10) at ../sysdeps/unix/sysv/linux/wordsize-64/xstat.c:37 #9 0x00007f3c3fc7e5f6 in __tzfile_read (file=file@entry=0x7f3c3fd4b2c3 "/etc/localtime", extra=extra@entry=0, extrap=extrap@entry=0x0) at tzfile.c:170 #10 0x00007f3c3fc7d954 in tzset_internal (always=<optimized out>, explicit=explicit@entry=1) at tzset.c:444 #11 0x00007f3c3fc7e303 in __tz_convert (timer=0x7fffcfa92d50, use_localtime=use_localtime@entry=1, tp=tp@entry=0x7f3c3ff8ed80 <_tmbuf>) at tzset.c:629 #12 0x00007f3c3fc7c2a1 in __GI_localtime (t=<optimized out>) at localtime.c:42 #13 0x00007f3c3fc7c1f9 in ctime (t=<optimized out>) at ctime.c:27 #14 0x00007f3c3e180ec2 in ?? () #15 0x0000000056a100c2 in ?? () #16 0xf8570f79d4fc4200 in ?? () #17 0x000000000209bec0 in ?? () #18 0x00007f3c4059f1f8 in ?? () #19 0x000000000000003c in ?? () #20 0x0000000000404952 in ulogd_propagate_results () #21 0x00007f3c3f9cc203 in ?? () #22 0x0000000000000000 in ?? () Segmentation fault in free() ============================ >From my experience, I think this was caused by some routine called malloc()/free() in signal handler context. By that, malloc() management data became inconsistent. As a result, free() made a wrong dereference. Program terminated with signal SIGSEGV, Segmentation fault. #0 __GI___libc_free (mem=0x7f430f011000) at malloc.c:2903 2903 if (chunk_is_mmapped(p)) /* release mmapped memory. */ (gdb) bt #0 __GI___libc_free (mem=0x7f430f011000) at malloc.c:2903 #1 0x00007f430e68affa in __GI__IO_free_backup_area (fp=fp@entry=0x742500) at genops.c:210 #2 0x00007f430e68a795 in _IO_new_file_overflow (f=0x742500, ch=-1) at fileops.c:849 #3 0x00007f430e689511 in _IO_new_file_xsputn (f=0x742500, data=<optimized out>, n=15) at fileops.c:1372 #4 0x00007f430e65aa4d in _IO_vfprintf_internal (s=s@entry=0x742500, format=<optimized out>, format@entry=0x7f430cbc4008 "%.15s %s %s", ap=ap@entry=0x7fff456ece38) at vfprintf.c:1635 #5 0x00007f430e71d615 in ___fprintf_chk (fp=0x742500, flag=flag@entry=1, format=format@entry=0x7f430cbc4008 "%.15s %s %s") at fprintf_chk.c:36 #6 0x00007f430cbc3f04 in fprintf (__fmt=0x7f430cbc4008 "%.15s %s %s", __stream=<optimized out>) at /usr/include/bits/stdio2.h:97 #7 _output_logemu (upi=0x74e5a0) at ulogd_output_LOGEMU.c:102 #8 0x0000000000404952 in ulogd_propagate_results () #9 0x00007f430e40f203 in interp_packet (ldata=0x7fff456ed060, pf_family=2 '\002', upi=0x74a6b0) at ulogd_inppkt_NFLOG.c:400 #10 msg_cb (gh=<optimized out>, nfmsg=0x7f430efe2020, nfa=0x7fff456ed060, data=0x74a6b0) at ulogd_inppkt_NFLOG.c:483 #11 0x00007f430e20a307 in __nflog_rcv_pkt (nlh=<optimized out>, nfa=<optimized out>, data=<optimized out>) at libnetfilter_log.c:160 #12 0x00007f430e0056b7 in __nfnl_handle_msg (len=268, nlh=0x7f430efe2010, h=0x74e8e0) at libnfnetlink.c:1236 #13 nfnl_handle_packet (h=0x74e8e0, buf=0x7f430efe2010 "\f\001", len=<optimized out>) at libnfnetlink.c:1256 #14 0x00007f430e20a508 in nflog_handle_packet (h=<optimized out>, buf=<optimized out>, len=<optimized out>) at libnetfilter_log.c:323 #15 0x00007f430e40eaed in nful_read_cb (fd=<optimized out>, what=<optimized out>, param=0x74a6b0) at ulogd_inppkt_NFLOG.c:463 #16 0x0000000000404ee0 in ulogd_select_main () #17 0x0000000000402b17 in main () Signed-off-by: Hironobu Ishii <ishii.hironobu@jp.fujitsu.com>
* sqlite3: Remove unused "buffer" option.Alex Xu2016-01-193-20/+1
| | | | | | | This option was left behind when the code was rewritten and is no longer functional or useful. Remove it entirely. Signed-off-by: Alex Xu (Hello71) <alex_y_xu@yahoo.ca>
* json: append timezone information to ISO 8601 dateVincent Bernat2015-10-021-5/+20
| | | | | | | | | | | | | | | | | | While this is not strictly needed for ISO 8601, this is helpful since otherwise, the receiver can't assume anything about the timezone. This uses a GNU extension but as ulogd is quite Linux-specific, this shouldn't be a problem. The POSIX variables (tzname and daylight) are quite difficult to use because daylight handling is incomplete (daylight don't say if DST is now in effect, it just says it is sometimes in effect). A timezone offset is used instead of a timezone since it is usually easier to parse (strptime in glibc is not able to parse a timezone name) and don't require an up-to-date TZ database. Signed-off-by: Vincent Bernat <Vincent.Bernat@exoscale.ch>
* json: output messages in JSONv1 formatVincent Bernat2015-10-022-1/+17
| | | | | | | | | | | | | | | | | | While Logstash is quite flexible in the JSON messages received, the canonical format it "expects" is the JSON Event v1 format. The timestamp should be keyed by `@timestamp` and there should be a `@version` key whose value is 1. All other keys are free. There is no formal specification of this format. It is however described here: https://github.com/elastic/logstash/blob/1.5/lib/logstash/event.rb#L26-L47 It's useful to respect this format as it allows a user to use a less capable receiver. The new format is enabled only when `eventv1=1` is set in plugin configuration. Signed-off-by: Vincent Bernat <Vincent.Bernat@exoscale.ch>
* Use stdint types everywhereFelix Janda2015-06-2619-122/+126
| | | | Signed-off-by: Felix Janda <felix.janda@posteo.de>
* ulogd: Use /dev/null as dummy logfile when logging to syslogFelix Janda2015-06-231-5/+5
| | | | | | | | | Fixes compilation error with musl libc: ulogd.c:86:13: error: storage size of 'syslog_dummy' isn't known static FILE syslog_dummy; Signed-off-by: Felix Janda <felix.janda@posteo.de>
* Define _GNU_SOURCE to get members of tcphdrFelix Janda2015-06-232-0/+2
| | | | | | | | The source uses linux names for members of tcphdr. For example "source" instead of "th_sport", ... musl libc's headers need _GNU_SOURCE defined in order to expose these. Signed-off-by: Felix Janda <felix.janda@posteo.de>
* Sync with kernel headersFelix Janda2015-06-231-10/+18
| | | | Signed-off-by: Felix Janda <felix.janda@posteo.de>
* configure.ac: Add --without-{mysql,pgsql}Harald Welte2015-05-031-2/+9
| | | | | In some cases you may not want to build a certain output plugin, even if the headers/libraries actually exist on the build host.
* Add missing enableval to configureulogd-2.0.5Eric Leblond2015-04-241-3/+14
| | | | Without that --disable-FEATURE is not working correctly.
* Add flag to disable ULOG input pluginEric Leblond2015-04-242-1/+14
| | | | | ULOG target is removed from kernel so we can prepare the removal of the plugin for ulogd. For now, we just add a configure flag.
* Set release number to 2.0.5.Eric Leblond2015-04-031-1/+1
|
* include: keep a copy of linux/netfilter_ipv4/ipt_ULOG.hPablo Neira Ayuso2014-11-144-3/+53
| | | | | | | | | | This fixes compilation if you use a Linux kernel >= 3.17. This problem occurs since ULOG was removed from mainstream: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=7200135bc1e61f1437dc326ae2ef2f310c50b4eb Closes: https://bugzilla.netfilter.org/show_bug.cgi?id=986 Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* Fix JSON output on big endian systemsJimmy Jones2014-08-111-0/+11
| | | | Signed-off-by: Jimmy Jones <jimmyjones2@gmx.co.uk>
* gitignore: add manpageEric Leblond2014-07-011-0/+1
| | | | Signed-off-by: Eric Leblond <eric@regit.org>
* gitignore: ignore generated documentationVincent Bernat2014-07-011-0/+3
| | | | Signed-off-by: Vincent Bernat <Vincent.Bernat@exoscale.ch>
* Set release number to 2.0.4.ulogd-2.0.4Eric Leblond2014-03-071-1/+1
| | | | Signed-off-by: Eric Leblond <eric@regit.org>
* ulogd: fix loglevel handlingKen-ichirou MATSUZAWA2014-03-071-1/+2
| | | | | | It was always default if not specified by command parameter. Signed-off-by: Ken-ichirou MATSUZAWA <chamas@h4.dion.ne.jp>
* printpkt: oob.time.sec was not usedEric Leblond2014-03-072-2/+0
| | | | | | | This patch supresses the oob.time.sec from the input keys as this is not used when creating the string corresponding to the packet. Signed-off-by: Eric Leblond <eric@regit.org>
* json: use packet timestamp if availableEric Leblond2014-03-071-5/+43
| | | | | | | | | This patch updates the JSON output plugin to have it use the timestamp of the packet if available. The date format used for the timestamp is now using ISO 8601 to have an easy import in most software (tested with logstash and splunk). Signed-off-by: Eric Leblond <eric@regit.org>
* nfct: use start timestamp if providedEric Leblond2014-02-221-4/+14
| | | | | | | | When hash table is not used, the start timestamp was not used even if the kernel is sending it. This patch modifies the code to use it when available. This allows to log connection with start and end with a single message per connection and without the cost of maintaining the hash table.
* ulogd: add carriage return as separatorEric Leblond2014-02-221-3/+3
| | | | | | | If the file is in DOS mode, the string coming from config file parsing are containing the carriage return. The result is that string are not correct and the parsing of confuguration file is failling.
* ulogd: avoid potential double print of messageEric Leblond2014-02-041-2/+2
| | | | | In case there is no logfile, ulogd could possibly display each log message twice to stderr.
* json: introduce new JSON output pluginEric Leblond2014-01-284-0/+291
| | | | | | | | | | | | | | This patch introduces a new JSON output plugin. This patch displays CIM field name instead of ulogd key valu if this CIM field is available. The module does not display binary address but uses the string version of them. So a complete stack is for example: stack=log2:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,mac2str1:HWHDR,json1:JSON If boolean_label is set to 1, then the numeric_label put on packet by the input plugin is coding the decision on packet. If 0, then packet has been blocked and if non null it has been accepted.
* store Common Information Model name in ulogd keyEric Leblond2014-01-283-2/+15
| | | | | | | | | | | | | | | | | | | This patch adds storage for CIM field name in ulogd key. This will be used by JSON output to interoperate with logging collector such as logstash or splunk. Common Information Model is an open standard that defines how managed elements in an IT environment are represented as a common set of objects and relationships between them: http://www.dmtf.org/standards/cim This seems to be mainly XML based but there is a JSON version of some aspects of the model. One of the main documentation on CIM in JSON format seems to be: http://docs.splunk.com/Documentation/PCI/2.0/DataSource/CommonInformationModelFieldReference Using the correct CIM field name allow events coming from ulogd to be correlated with events coming from other sources.
* db: don't loose one packet on reconnectEric Leblond2014-01-261-1/+2
| | | | | | This patch improves database reconnection handling in ring buffer mode. Ulogd now redo the failed query and do not loose anymore one packet.
* db: cancel injection thread when terminatingEric Leblond2014-01-251-0/+17
| | | | | | Injection thread was not cancelled when a termination signal was sent. This was causing a crash in some cases. This patch fixes this by canceling the thread when a SIGTERM or SIGINT signal is received.
* db: set ring default size to 0Eric Leblond2014-01-251-1/+1
| | | | | | | | | As default size was non null, the ring system was activated by default. It was only possible to desactivate the ring system by setting it to ring_buffer_size to 0 in the configuration. This was not the attended behavior. This patch set it to 0 to have only explicit activation of the ring feature.
* util: fix warning on format stringEric Leblond2014-01-251-20/+21
| | | | | | | pp_print macro was not correctly formatting u64. This patch renames it to pp_print_u as it is only used to print integer. It also use the PRIu* macros to have a correct format string for all integers type.