summaryrefslogtreecommitdiffstats
path: root/output
Commit message (Collapse)AuthorAgeFilesLines
* IPFIX: Introduce template record supportAnder Juaristi2019-04-303-37/+138
| | | | | | | | | | | | | | | | | | | | This commit adds the ability to send template records to the remote collector. In addition, it also introduces a new configuration parameter 'send_template', which tells when template records should be sent. It accepts the following string values: - "once": Send the template record only the first time (might be coalesced with data records). - "always": Send the template record always, with every data record that is sent to the collector (multiple data records might be sent together). - "never": Assume the collector knows the schema already. Do not send template records. If omitted, the default value for 'send_template' is "once". Signed-off-by: Ander Juaristi <a@juaristi.eus> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* IPFIX: Add IPFIX output pluginAnder Juaristi2019-04-306-547/+741
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds an IPFIX output plugin to ulogd2. It generates NetFlow/IPFIX traces and sends them to a remote server (collector) via TCP or UDP. Based on original work by Holger Eitzenberger <holger@eitzenberger.org>. How to test this ---------------- I am currently testing this with the NFCT input and Wireshark. Place the following in ulogd.conf: # this will print all flows on screen loglevel=1 # load NFCT and IPFIX plugins plugin="/lib/ulogd/ulogd_inpflow_NFCT.so" plugin="/lib/ulogd/ulogd_output_IPFIX.so" stack=ct1:NFCT,ipfix1:IPFIX [ct1] netlink_socket_buffer_size=217088 netlink_socket_buffer_maxsize=1085440 accept_proto_filter=tcp,sctp [ipfix1] oid=1 host="127.0.0.1" #port=4739 #send_template="once" I am currently testing it by launching a plain NetCat listener on port 4739 (the default for IPFIX) and then running Wireshark and see that it dissects the IPFIX/NetFlow traffic correctly (obviously this relies on the Wireshark NetFlow dissector being correct). First: nc -vvvv -l 127.0.0.1 4739 Then: sudo ulogd -vc ulogd.conf Signed-off-by: Ander Juaristi <a@juaristi.eus> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* ulogd: json: send messages to a remote host / unix socketAndreas Jaggi2018-06-011-21/+270
| | | | | | | | Extend the JSON output plugin so that the generated JSON stream can be sent to a remote host via TCP/UDP or to a local unix socket. Signed-off-by: Andreas Jaggi <andreas.jaggi@waterwave.ch> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* harmonize log file defaults with ulogd.confKaarle Ritvanen2017-05-164-4/+4
| | | | Signed-off-by: Kaarle Ritvanen <kaarle.ritvanen@datakunkku.fi>
* sqlite3: Remove unused "buffer" option.Alex Xu2016-01-191-16/+1
| | | | | | | This option was left behind when the code was rewritten and is no longer functional or useful. Remove it entirely. Signed-off-by: Alex Xu (Hello71) <alex_y_xu@yahoo.ca>
* json: append timezone information to ISO 8601 dateVincent Bernat2015-10-021-5/+20
| | | | | | | | | | | | | | | | | | While this is not strictly needed for ISO 8601, this is helpful since otherwise, the receiver can't assume anything about the timezone. This uses a GNU extension but as ulogd is quite Linux-specific, this shouldn't be a problem. The POSIX variables (tzname and daylight) are quite difficult to use because daylight handling is incomplete (daylight don't say if DST is now in effect, it just says it is sometimes in effect). A timezone offset is used instead of a timezone since it is usually easier to parse (strptime in glibc is not able to parse a timezone name) and don't require an up-to-date TZ database. Signed-off-by: Vincent Bernat <Vincent.Bernat@exoscale.ch>
* json: output messages in JSONv1 formatVincent Bernat2015-10-021-1/+14
| | | | | | | | | | | | | | | | | | While Logstash is quite flexible in the JSON messages received, the canonical format it "expects" is the JSON Event v1 format. The timestamp should be keyed by `@timestamp` and there should be a `@version` key whose value is 1. All other keys are free. There is no formal specification of this format. It is however described here: https://github.com/elastic/logstash/blob/1.5/lib/logstash/event.rb#L26-L47 It's useful to respect this format as it allows a user to use a less capable receiver. The new format is enabled only when `eventv1=1` is set in plugin configuration. Signed-off-by: Vincent Bernat <Vincent.Bernat@exoscale.ch>
* Use stdint types everywhereFelix Janda2015-06-262-11/+11
| | | | Signed-off-by: Felix Janda <felix.janda@posteo.de>
* Fix JSON output on big endian systemsJimmy Jones2014-08-111-0/+11
| | | | Signed-off-by: Jimmy Jones <jimmyjones2@gmx.co.uk>
* json: use packet timestamp if availableEric Leblond2014-03-071-5/+43
| | | | | | | | | This patch updates the JSON output plugin to have it use the timestamp of the packet if available. The date format used for the timestamp is now using ISO 8601 to have an easy import in most software (tested with logstash and splunk). Signed-off-by: Eric Leblond <eric@regit.org>
* json: introduce new JSON output pluginEric Leblond2014-01-282-0/+264
| | | | | | | | | | | | | | This patch introduces a new JSON output plugin. This patch displays CIM field name instead of ulogd key valu if this CIM field is available. The module does not display binary address but uses the string version of them. So a complete stack is for example: stack=log2:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,mac2str1:HWHDR,json1:JSON If boolean_label is set to 1, then the numeric_label put on packet by the input plugin is coding the decision on packet. If 0, then packet has been blocked and if non null it has been accepted.
* nacct: fix format warningEric Leblond2014-01-251-2/+2
| | | | | | Some counter have been recently switch to u64. This has caused warnings relative to format string. This patch uses PRIu64 macro to fix these warnings.
* nfct: make NFCT packet counter/length 64 bitUlrich Weber2013-10-101-8/+8
| | | | | | | | | | | | | | Kernel and libnetfilter_conntrack counters are 64bit, so use 64bit too in ulogd instead of 32bit. Worked fine on little endian systems but big endian systems had zero counter... Didn't test ipfix output, but RFC allows template with either 32 or 64 counters, so should be safe. Signed-off-by: Ulrich Weber <uw@xyne.com> Signed-off-by: Eric Leblond <eric@regit.org>
* pgsql: add var to specify arbitrary conn paramsEric Leblond2013-06-081-40/+47
| | | | | | | | | This patch adds a configuration variable for PostgreSQL output. Named connstring it stores the character string that will be used to connect to the PostgreSQL server. This allows the user to use all options available like TLS parameters for example. Signed-off-by: Eric Leblond <eric@regit.org>
* sqlite3: add sanity checkingEric Leblond2013-04-201-0/+2
| | | | Nullify sqlite3 handler at deinit.
* mysql: add sanity checkingEric Leblond2013-04-201-1/+3
| | | | Nullify mysql handler at deinit.
* postgresql: add sanity checkingEric Leblond2013-04-201-1/+3
| | | | Clean postgresql handler at deinit.
* logemu: return error if configuration is invalid.Eric Leblond2013-02-181-4/+1
|
* graphite: fix crash on i386Eric Leblond2013-02-171-2/+2
| | | | | | | | It seems a cast of time_t is needed for i386 system to avoid a crash. I've added a cast to uint64_t that should be ok on all Linux system. Reported-by: netfilter@openenterprise.co.uk
* graphite: fix warning about gnu extension usageEric Leblond2013-01-181-4/+4
| | | | | clang is complaining about missing = being a gnu extension. This patch adds equal sign to fix the warning.
* Get rid of SVN tag in comment.Eric Leblond2013-01-186-15/+3
| | | | This patch also update some copyright and licence declaration.
* Add GRAPHITE output module.Eric Leblond2012-12-272-1/+251
| | | | | | | Graphite is a web application which provide real-time visualization and storage of numeric time-series data. This patch adds a module named GRAPHITE which sends NFACCT accounting data to a graphite server.
* Add additional ip6 header fields to database scriptsBob Hockney2012-12-191-1/+1
| | | | | Rename internal keyname ip6.payload_len to remove "_" to facilitate this.
* Handle postgresql schemas correctlyBob Hockney2012-12-191-4/+31
| | | | Add 'schema' variable to look into corresponding schema.
* build: move remaining preprocessor flags into CPPFLAGSJan Engelhardt2012-11-272-5/+5
| | | | | | | | The flags retrieved from `pkg-config --cflags ...` are generally only preprocessor flags (mostly -I to point to the directories), since anything else would inconvenience downstream users. Signed-off-by: Jan Engelhardt <jengelh@inai.de>
* build: resolve compile error due to missing nfacct CPPFLAGSJan Engelhardt2012-11-271-1/+1
| | | | | | | | | | | | | | Fixes this error: make[3]: Entering directory "/home/jengelh/code/ulogd2/output" CC ulogd_output_XML.lo ulogd_output_XML.c:31:49: fatal error: libnetfilter_acct/libnetfilter_acct.h: No such file or directory (Note that pkgconfig-provided cflags are actually cppflags, so I add ${LIBNETFILTER_ACCT_CFLAGS} to AM_CPPFLAGS.) Signed-off-by: Jan Engelhardt <jengelh@inai.de>
* XML: add missing includeEric Leblond2012-11-041-0/+2
| | | | | | Inclusion of libnetfilter_acct.h is not enough as the integer type definition are not included in the header. So if NFCT is disable, the compilation fails.
* Use PRIu64 to print unsigned 64bit intEric Leblond2012-11-042-2/+4
|
* configure: add flag to disable NFLOG buildEric Leblond2012-11-041-1/+7
|
* configure: add flag to disable NFCT buildEric Leblond2012-11-041-1/+7
|
* configure: add flag to disable NFACCT buildEric Leblond2012-11-041-1/+8
| | | | | | It is now possible to pass the --disable-nfacct flag to disable compilation of NFACCT input plugin. Doing this the build of ulogd is possible on system where nfacct is not available.
* ipfix: fix bracket imbalance if IPPROTO_SCTP is definedThomas Jarosch2012-08-061-0/+1
| | | | | | | Detected by cppcheck. Signed-off-by: Thomas Jarosch <thomas.jarosch@intra2net.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* src: fix version that -V displaysPablo Neira Ayuso2012-08-0312-12/+12
| | | | | | | It was wrong, use VERSION constant which uses the version information available in configure.ac. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* pgsql: only disable key if it starts with underscoreEric Leblond2012-08-031-2/+3
| | | | | | | | | | ulogd2 was magically making inactive the first key of description table. This patch improves this system by only doing so when the key start with an undescore. This way, system like nfacct which do not have a primary key can be implemented easily. Signed-off-by: Eric Leblond <eric@regit.org> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* sqlite3: Add more logging if database initialization fails.Michael Tremer2012-07-231-2/+6
| | | | | Signed-off-by: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: XML: display time for NFACCTPablo Neira Ayuso2012-06-221-1/+2
| | | | | | <obj><name>test</name><pkts>00000000000000000000</pkts><bytes>00000000000000000000</bytes><hour>17</hour><min>46</min><sec>47</sec><wday>6</wday><day>22</day><month>6</month><year>2012</year></obj> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: XML: support for NFACCT input pluginPablo Neira Ayuso2012-06-222-2/+31
| | | | | | | | | This patch extends XML plugin to support NFACCT. You can use the following line in ulogd.conf to test it: stack=acct1:NFACCT,xml1:XML Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* src: add ULOGD_DTYPE_SUM for nfacct-based accountingPablo Neira Ayuso2012-06-221-1/+1
| | | | | | This new type will be used in flow-up patch to support XML output. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* pcap: resolve unreliable detectionJan Engelhardt2012-06-051-2/+2
| | | | | | | | | | | | pcap is not found reliably by either --with-pcap=%_prefix or --with-pcap-lib=%_libdir --with-pcap-inc=%_includedir. If you have any special paths, just use ./configure CPPFLAGS="-I/my/pcap" LDFLAGS="-L/my/pcap" (And -lpcap is already known so no need to specify that.) Signed-off-by: Jan Engelhardt <jengelh@inai.de>
* sqlite: resolve compiler warningsJan Engelhardt2012-06-051-1/+1
| | | | | | | | | | | | | | | | In file included from /usr/include/string.h:642:0, from ulogd_output_SQLITE3.c:34: In function 'strncat', inlined from 'db_count_cols' at ulogd_output_SQLITE3.c:306:9, inlined from 'sqlite3_init_db' at ulogd_output_SQLITE3.c:328:11: /usr/include/bits/string3.h:152:3: warning: call to __builtin___strncat_chk might overflow destination buffer [enabled by default] I: Statement might be overflowing a buffer in strncat. Common mistake: BAD: strncat(buffer,charptr,sizeof(buffer)) is wrong, it takes the left over size as 3rd argument GOOD: strncat(buffer,charptr,sizeof(buffer)-strlen(buffer)-1) E: ulogd2 bufferoverflowstrncat ulogd_output_SQLITE3.c:328:11 Signed-off-by: Jan Engelhardt <jengelh@inai.de>
* build: use pkglibdir instead of pkglibexecdir for automakeBjörn Lässig2012-05-186-6/+6
| | | | | | | | | | | This fixes the following problem while running `autoreconf -fi` `pkglibexecdir' is not a legitimate directory for `LTLIBRARIES' variable `ulogd_filter_PRINTPKT_la_SOURCES' is defined but no program or library has `ulogd_filter_PRINTPKT_la' as canonical name (possible typo) Signed-off-by: Björn Lässig <laessig@bitformer.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* ulogd2 / DBI / table nameHarald Welte2012-02-241-1/+4
| | | | | | | | | | | | | DBI: allow to define table name via table config option While using the DBI plugin of ulogd2 for NFCT based accounting, despite using table="conntrack", it always insisted in using the table "ulog" for deriving the keys/columns to be stored. I've hacked up a quick fix, and it seems to work as expected (though no proper null termination after strncpy). Signed-off-by: Harald Welte <laforge@netfilter.org>
* output: add GPRINT pluginPablo Neira Ayuso2012-02-222-1/+274
| | | | | | | | | | This patch adds GPRINT which is a generalization of OPRINT. It display the set of key-values separated by commas. This is the generic print that you can attach to whatever kind of input plugin. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* OPRINT: display u64 fieldsPablo Neira Ayuso2012-02-221-0/+3
| | | | Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* sqlite3: remove automatic creation of table `daily'Pablo Neira Ayuso2011-02-251-39/+7
| | | | | | | | | | | | | | | | | | | This patch removes the creation of the `daily' table. Now, we assume that the table that we use are created before launching ulogd2. This code is broken because you have to specify in the configuration file that the table used is `daily', otherwise this `daily' table is created and dropped during the daemon starting, but not used. Moreover, the code explicit shows a message that it says: /* FIXME make this configurable */ So, I think that this patch is the way to go :-). This patch also documents the table creation in ulogd.sgml Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* sqlite3: fix warning during compilationPablo Neira Ayuso2011-02-251-1/+1
| | | | | | | | | This patch fixes the following warning during the compilation: ulogd_output_SQLITE3.c: In function ‘ulogd_find_key’: ulogd_output_SQLITE3.c:292: warning: comparison between signed and unsigned integer expressions Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* sqlite3: cleanup error handling of sqlite3_bind_int() in sqlite3_interp()Pablo Neira Ayuso2011-02-251-20/+4
| | | | | | | Move error handling after the switch statement since it's the same for all cases, we save several lines of code. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* sqlite3: use continue instead of goto in sqlite3_interp()Pablo Neira Ayuso2011-02-251-3/+2
| | | | | | | | | Use continue instead of goto inside loop. I don't need to scroll up and down in the code to know what the jump is performing. I think this improve code readability. It's a comestic cleanup, of course. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* build: simplify sqlite3 checkJan Engelhardt2011-02-251-3/+3
| | | | | | | | With pkg-config variables, even a non-installed sqlite3 can be configured easily. Signed-off-by: Jan Engelhardt <jengelh@medozas.de> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* sqlite3: initial support extracted from Holger Eitzenberger's workPablo Neira Ayuso2011-02-241-259/+350
| | | | | | | | | | | | | | | | | | | | | | | | | | This patch imports the sqlite3 support from Holger. I applied folded the following patches, that Harald passed to me during the last Netfilter workshop, into one: 0001-ulog2-sqlite-port-to-v2.diff.patch 0002-ulog2-sqlite-prepare-fix.diff.patch 0003-ulogd-sqlite-new-logging.diff.patch 0004-ulogd-accounting-add-flow-start-day.patch 0005-ulogd-sqlite3-err-codes-fix.dif.patch 0006-ulogd-sqlite3-tbl-corrupt-fix.diff.patch 0007-ulogd-sqlite3-handle-schema-change.diff.patch 0008-SQLITE3-count-instead-of-log-table-busy-messags.patch 0009-ulogd-SQLITE-Added-flowstartsec.diff.patch Harald passed them to me with no description, so applying them separately does not provide more information. I'll start adding patches on top on these so Holger can get in sync with my work. This also can help him to take my patches and to integrate them to his tree. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>