summaryrefslogtreecommitdiffstats
path: root/output
Commit message (Collapse)AuthorAgeFilesLines
* output: JSON: increase time-stamp buffer sizeJeremy Sowden2022-01-031-1/+1
| | | | | | | | | | | | | | | The output buffer for date-times is of sufficient size provided that we don't get oversized integer values for any of the fields, which is a reasonable assumption. However, the compiler complains about possible truncation, e.g.: ulogd_output_JSON.c:314:65: warning: `%06u` directive output may be truncated writing between 6 and 10 bytes into a region of size between 0 and 18 ulogd_output_JSON.c:313:25: note: `snprintf` output between 27 and 88 bytes into a destination of size 38 Fix the warnings by increasing the buffer size. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: JSON: fix output of GMT offsetJeremy Sowden2022-01-031-4/+5
| | | | | | | | | | | | | | | | | | | | | | | | The compiler has two sets of complaints. Firstly, `t->tm_gmtoffset` is a `long int`, but it is being passed to `abs`, which leads to warnings such as: ulogd_output_JSON.c:308:34: warning: absolute value function `abs` given an argument of type `long int` but has parameter of type `int` which may cause truncation of value Secondly, it can't verify that the hour value derived from the offset will in fact fit into `%02d`, thus: ulogd_output_JSON.c:306:37: warning: `%02d` directive output may be truncated writing between 2 and 6 bytes into a region of size 5 To remedy these, we now mod the offset by 86,400 and assign it to an `int` before deriving the hour and minute values. We also change the format-specifier for the hour value to `%+03d` which causes a sign to be printed even if the value is positive, thus allowing us not to specify the sign explicitly and to drop the `abs` call for the hour value. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: SQLITE3: catch errors creating SQL statementJeremy Sowden2022-01-031-7/+13
| | | | | | | | | `sqlite3_createstmt` returns non-zero on error, but the return-value was being ignored. Change the calling code to check the return-value, log an error message and propagate the error. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: SQLITE3: improve mapping of fields to DB columnsPablo Neira Ayuso2022-01-031-16/+8
| | | | | | | | | | | | | | | | | | | | | | Currently, we derive a field-name by replacing all the underscores in a DB column-name with full-stops and use the field-name to find the matching input-key. However, every time we create a new insert SQL statement, we derive the column-names by copying the field-names to a buffer, replacing all the full-stops with underscores, and then appending the buffer containing the column-name to the one containing the statments. Apart from the inefficiency, `strncpy` is used to do the copies, which leads gcc to complain: ulogd_output_SQLITE3.c:234:17: warning: `strncpy` output may be truncated copying 31 bytes from a string of length 31 Instead, leave the underscores in the field-name, but copy it once to a buffer in which the underscores are replaced and use this to find the input-key. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: SQLITE3: improve mapping of DB columns to fieldsJeremy Sowden2022-01-031-13/+13
| | | | | | | | | | | | | | | | | | | | | | Currently, we copy the column-name to a buffer, iterate over it to replace the underscores with full-stops, using `strchr` from the start of the buffer on each iteration, then copy the buffer to the field's `name` member. Apart from the inefficiency, `strncpy` is used to do the copies, which leads gcc to complain: ulogd_output_SQLITE3.c:341:17: warning: `strncpy` output may be truncated copying 31 bytes from a string of length 31 Furthermore, the buffer is not initialized, which means that there is also a possible buffer overrun if the column-name is too long, since `strncpy` will not append a NUL. Instead, copy the column-name directly to the field using `snprintf`, and run `strchr` from the last underscore on each iteration. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: SQLITE3: improve formatting of insert statementJeremy Sowden2022-01-031-10/+6
| | | | | | | | | | | | | | | | | | | | | | `sqlite3_createstmt` contains a variable `stmt_pos` which points to the end of the SQL already written, where the next chunk should be appended. Currently, this is assigned after every write: sprintf(stmt_pos, ...); stmt_pos = priv->stmt + strlen(priv->stmt); However, since `sprintf` returns the number of bytes written, increment `stmt_pos` by the return-value of `sprintf` in order to avoid the repeated `strlen` calls. Pablo mangled this original patch to add this chunk at the end of this patch (originally submitted as a conversion to use strcpy). + for (i = 0; i < cols - 1; i++) + stmt_pos += sprintf(stmt_pos, "?,"); Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: SQLITE3: fix memory-leak in error-handlingJeremy Sowden2021-12-061-1/+5
| | | | | | | | | When mapping DB column names to input-keys, if we cannot find a key to match a column, the newly allocated `struct field` is leaked. Free it, and log an error message. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: SQLITE3: fix possible buffer overrunsJeremy Sowden2021-12-061-3/+3
| | | | | | | | | There is a an off-by-one error in the size of some of the buffers used to hold key-names. The maximum length of a name is `ULOGD_MAX_KEYLEN`, and so declare the buffers with size `ULOGD_MAX_KEYLEN + 1`. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: PGSQL: fix non-`connstring` configuration of DB connectionJeremy Sowden2021-12-061-27/+17
| | | | | | | | | | | | | In `open_db_pgsql`, we test whether various config-settings are defined by comparing their string values to `NULL`. However, the `u.string` member of `struct config_entry` is an array, not a pointer, so it is never `NULL`. Instead, check whether the string is empty. Use a pointer to the end of the `connstr` buffer and `sprintf`, rather than repeated `strcat`s. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: PGSQL: improve mapping of DB columns to input-keysJeremy Sowden2021-12-061-7/+7
| | | | | | | | | | | | | | | | | | | | | | Currently, we copy the column-name to a buffer, iterate over it to replace the underscores with full-stops, using `strchr` from the start of the buffer on each iteration, then copy the buffer to the input-key's `name` member. Apart from the inefficiency, `strncpy` is used to do the copies, which leads gcc to complain: ulogd_output_PGSQL.c:204:17: warning: `strncpy` output may be truncated copying 31 bytes from a string of length 31 Furthermore, the buffer is not initialized, which means that there is also a possible buffer overrun if the column-name is too long, since `strncpy` will not append a NUL. Instead, copy the column-name directly to the input-key using `snprintf`, and run `strchr` from the last underscore on each iteration. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: MYSQL: improve mapping of DB columns to input-keysJeremy Sowden2021-12-061-7/+7
| | | | | | | | | | | | | | | | | | | | | | Currently, we copy the column-name to a buffer, iterate over it to replace the underscores with full-stops, using `strchr` from the start of the buffer on each iteration, then copy the buffer to the input-key's `name` member. Apart from the inefficiency, `strncpy` is used to do the copies, which leads gcc to complain: ulogd_output_MYSQL.c:149:17: warning: `strncpy` output may be truncated copying 31 bytes from a string of length 31 Furthermore, the buffer is not initialized, which means that there is also a possible buffer overrun if the column-name is too long, since `strncpy` will not append a NUL. Instead, copy the column-name directly to the input-key using `snprintf`, and run `strchr` from the last underscore on each iteration. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: DBI: fix configuration of DB connectionJeremy Sowden2021-12-061-7/+7
| | | | | | | | | | In `open_db_dbi`, we test whether various config-settings are defined by comparing their string values to `NULL`. However, the `u.string` member of `struct config_entry` is an array, not a pointer, so it is never `NULL`. Instead, check whether the string is empty. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: DBI: fix NUL-termination of escaped SQL stringJeremy Sowden2021-12-061-4/+6
| | | | | | | | | | | | On error, `dbi_conn_quote_string_copy` returns zero. In this case, we need to set `*dst` to NUL. Handle a return-value of `2` as normal below. `1` is never returned. Replace `strncpy` with `memcpy`: using `strncpy` is nearly always a mistake, and we don't need its special behaviour here. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: DBI: improve mapping of DB columns to input-keysJeremy Sowden2021-11-301-21/+13
| | | | | | | | | | | | | | | | | | | | | | | | Currently, we copy the column-name to a buffer, iterate over it to replace the underscores with full-stops, using `strchr` from the start of the buffer on each iteration, iterate over it a second time to lower-case all letters, and finally copy the buffer to the input-key's `name` member. In addition to being inefficient, `strncpy` is used to do the copies, which leads gcc to complain: ulogd_output_DBI.c:160:17: warning: `strncpy` output may be truncated copying 31 bytes from a string of length 31 Furthermore, the buffer is not initialized, which means that there is also a possible buffer overrun if the column-name is too long, since `strncpy` will not append a NUL. Instead, copy the column-name directly to the input-key using `snprintf`, and then iterate over it once to replace underscores and lower-case letters. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: DBI: fix deprecation warningsJeremy Sowden2021-11-301-7/+11
| | | | | | | | | | | | | The DBI output plugin uses some libdbi functions which have been deprecated in favour of re-entrant equivalents. Switch to the re-entrant functions. Remove superfluous `init` declaration. Add destructor to clean up DBI instance on exit. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* Replace malloc+memset with callocJeremy Sowden2021-11-304-18/+4
| | | | | | | | There are a number of places where we `malloc` some memory and then `memset` it to zero. Use `calloc` instead. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* XML: show both nflog packet and conntrackKen-ichirou MATSUZAWA2021-11-231-11/+22
| | | | | | | | This patch enables to show "ct" as well as "raw" if output type is ULOGD_DTYPE_RAW and "ct" input exists. Signed-off-by: Ken-ichirou MATSUZAWA <chamas@h4.dion.ne.jp> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* output: IPFIX: correct format specifiersJeremy Sowden2021-11-231-4/+5
| | | | | | | | | | | There are a couple of logging calls which use the wrong specifiers for their integer arguments. Change the specifiers to match the arguments. Use the correct type for the variable holding the return-value of `send(2)`. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* build: missing ipfix.h header when running make distcheckPablo Neira Ayuso2021-11-161-0/+2
| | | | | | make distcheck reports ipfix.h is not included in the tarball file. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* build: skip sub-directories containing disabled pluginsJeremy Sowden2021-11-156-22/+27
| | | | | | | | | | | | | | | Currently, make enters all sub-directories containing source-code, even if they only contain optional targets which are not configured to be built. Instead, change the Makefiles so that the sub-directories are optional, rather than the targets. Group sub-directory definitions consistently at the top of the Makefiles that contain them. Trim a few leading and trailing blank lines. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* build: add Make_global.am for common flagsJeremy Sowden2021-11-157-16/+17
| | | | | | | | | | Move `${regular_CFLAGS}` from configure.ac to Make_global.am, renaming it to `AM_CFLAGS`. Add `AM_CPPFGLAGS` to include `$(top_srcdir)/include`. Include the new file in the Makefiles that require it. Signed-off-by: Jeremy Sowden <jeremy@azazel.net> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* XML: support nflog pkt outputKen-ichirou MATSUZAWA2021-10-121-1/+1
| | | | | | | plugin input type ULOGD_DTYPE_RAW was missing Signed-off-by: Ken-ichirou MATSUZAWA <chamas@h4.dion.ne.jp> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* IPFIX: Introduce template record supportAnder Juaristi2019-04-303-37/+138
| | | | | | | | | | | | | | | | | | | | This commit adds the ability to send template records to the remote collector. In addition, it also introduces a new configuration parameter 'send_template', which tells when template records should be sent. It accepts the following string values: - "once": Send the template record only the first time (might be coalesced with data records). - "always": Send the template record always, with every data record that is sent to the collector (multiple data records might be sent together). - "never": Assume the collector knows the schema already. Do not send template records. If omitted, the default value for 'send_template' is "once". Signed-off-by: Ander Juaristi <a@juaristi.eus> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* IPFIX: Add IPFIX output pluginAnder Juaristi2019-04-306-547/+741
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds an IPFIX output plugin to ulogd2. It generates NetFlow/IPFIX traces and sends them to a remote server (collector) via TCP or UDP. Based on original work by Holger Eitzenberger <holger@eitzenberger.org>. How to test this ---------------- I am currently testing this with the NFCT input and Wireshark. Place the following in ulogd.conf: # this will print all flows on screen loglevel=1 # load NFCT and IPFIX plugins plugin="/lib/ulogd/ulogd_inpflow_NFCT.so" plugin="/lib/ulogd/ulogd_output_IPFIX.so" stack=ct1:NFCT,ipfix1:IPFIX [ct1] netlink_socket_buffer_size=217088 netlink_socket_buffer_maxsize=1085440 accept_proto_filter=tcp,sctp [ipfix1] oid=1 host="127.0.0.1" #port=4739 #send_template="once" I am currently testing it by launching a plain NetCat listener on port 4739 (the default for IPFIX) and then running Wireshark and see that it dissects the IPFIX/NetFlow traffic correctly (obviously this relies on the Wireshark NetFlow dissector being correct). First: nc -vvvv -l 127.0.0.1 4739 Then: sudo ulogd -vc ulogd.conf Signed-off-by: Ander Juaristi <a@juaristi.eus> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* ulogd: json: send messages to a remote host / unix socketAndreas Jaggi2018-06-011-21/+270
| | | | | | | | Extend the JSON output plugin so that the generated JSON stream can be sent to a remote host via TCP/UDP or to a local unix socket. Signed-off-by: Andreas Jaggi <andreas.jaggi@waterwave.ch> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* harmonize log file defaults with ulogd.confKaarle Ritvanen2017-05-164-4/+4
| | | | Signed-off-by: Kaarle Ritvanen <kaarle.ritvanen@datakunkku.fi>
* sqlite3: Remove unused "buffer" option.Alex Xu2016-01-191-16/+1
| | | | | | | This option was left behind when the code was rewritten and is no longer functional or useful. Remove it entirely. Signed-off-by: Alex Xu (Hello71) <alex_y_xu@yahoo.ca>
* json: append timezone information to ISO 8601 dateVincent Bernat2015-10-021-5/+20
| | | | | | | | | | | | | | | | | | While this is not strictly needed for ISO 8601, this is helpful since otherwise, the receiver can't assume anything about the timezone. This uses a GNU extension but as ulogd is quite Linux-specific, this shouldn't be a problem. The POSIX variables (tzname and daylight) are quite difficult to use because daylight handling is incomplete (daylight don't say if DST is now in effect, it just says it is sometimes in effect). A timezone offset is used instead of a timezone since it is usually easier to parse (strptime in glibc is not able to parse a timezone name) and don't require an up-to-date TZ database. Signed-off-by: Vincent Bernat <Vincent.Bernat@exoscale.ch>
* json: output messages in JSONv1 formatVincent Bernat2015-10-021-1/+14
| | | | | | | | | | | | | | | | | | While Logstash is quite flexible in the JSON messages received, the canonical format it "expects" is the JSON Event v1 format. The timestamp should be keyed by `@timestamp` and there should be a `@version` key whose value is 1. All other keys are free. There is no formal specification of this format. It is however described here: https://github.com/elastic/logstash/blob/1.5/lib/logstash/event.rb#L26-L47 It's useful to respect this format as it allows a user to use a less capable receiver. The new format is enabled only when `eventv1=1` is set in plugin configuration. Signed-off-by: Vincent Bernat <Vincent.Bernat@exoscale.ch>
* Use stdint types everywhereFelix Janda2015-06-262-11/+11
| | | | Signed-off-by: Felix Janda <felix.janda@posteo.de>
* Fix JSON output on big endian systemsJimmy Jones2014-08-111-0/+11
| | | | Signed-off-by: Jimmy Jones <jimmyjones2@gmx.co.uk>
* json: use packet timestamp if availableEric Leblond2014-03-071-5/+43
| | | | | | | | | This patch updates the JSON output plugin to have it use the timestamp of the packet if available. The date format used for the timestamp is now using ISO 8601 to have an easy import in most software (tested with logstash and splunk). Signed-off-by: Eric Leblond <eric@regit.org>
* json: introduce new JSON output pluginEric Leblond2014-01-282-0/+264
| | | | | | | | | | | | | | This patch introduces a new JSON output plugin. This patch displays CIM field name instead of ulogd key valu if this CIM field is available. The module does not display binary address but uses the string version of them. So a complete stack is for example: stack=log2:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,mac2str1:HWHDR,json1:JSON If boolean_label is set to 1, then the numeric_label put on packet by the input plugin is coding the decision on packet. If 0, then packet has been blocked and if non null it has been accepted.
* nacct: fix format warningEric Leblond2014-01-251-2/+2
| | | | | | Some counter have been recently switch to u64. This has caused warnings relative to format string. This patch uses PRIu64 macro to fix these warnings.
* nfct: make NFCT packet counter/length 64 bitUlrich Weber2013-10-101-8/+8
| | | | | | | | | | | | | | Kernel and libnetfilter_conntrack counters are 64bit, so use 64bit too in ulogd instead of 32bit. Worked fine on little endian systems but big endian systems had zero counter... Didn't test ipfix output, but RFC allows template with either 32 or 64 counters, so should be safe. Signed-off-by: Ulrich Weber <uw@xyne.com> Signed-off-by: Eric Leblond <eric@regit.org>
* pgsql: add var to specify arbitrary conn paramsEric Leblond2013-06-081-40/+47
| | | | | | | | | This patch adds a configuration variable for PostgreSQL output. Named connstring it stores the character string that will be used to connect to the PostgreSQL server. This allows the user to use all options available like TLS parameters for example. Signed-off-by: Eric Leblond <eric@regit.org>
* sqlite3: add sanity checkingEric Leblond2013-04-201-0/+2
| | | | Nullify sqlite3 handler at deinit.
* mysql: add sanity checkingEric Leblond2013-04-201-1/+3
| | | | Nullify mysql handler at deinit.
* postgresql: add sanity checkingEric Leblond2013-04-201-1/+3
| | | | Clean postgresql handler at deinit.
* logemu: return error if configuration is invalid.Eric Leblond2013-02-181-4/+1
|
* graphite: fix crash on i386Eric Leblond2013-02-171-2/+2
| | | | | | | | It seems a cast of time_t is needed for i386 system to avoid a crash. I've added a cast to uint64_t that should be ok on all Linux system. Reported-by: netfilter@openenterprise.co.uk
* graphite: fix warning about gnu extension usageEric Leblond2013-01-181-4/+4
| | | | | clang is complaining about missing = being a gnu extension. This patch adds equal sign to fix the warning.
* Get rid of SVN tag in comment.Eric Leblond2013-01-186-15/+3
| | | | This patch also update some copyright and licence declaration.
* Add GRAPHITE output module.Eric Leblond2012-12-272-1/+251
| | | | | | | Graphite is a web application which provide real-time visualization and storage of numeric time-series data. This patch adds a module named GRAPHITE which sends NFACCT accounting data to a graphite server.
* Add additional ip6 header fields to database scriptsBob Hockney2012-12-191-1/+1
| | | | | Rename internal keyname ip6.payload_len to remove "_" to facilitate this.
* Handle postgresql schemas correctlyBob Hockney2012-12-191-4/+31
| | | | Add 'schema' variable to look into corresponding schema.
* build: move remaining preprocessor flags into CPPFLAGSJan Engelhardt2012-11-272-5/+5
| | | | | | | | The flags retrieved from `pkg-config --cflags ...` are generally only preprocessor flags (mostly -I to point to the directories), since anything else would inconvenience downstream users. Signed-off-by: Jan Engelhardt <jengelh@inai.de>
* build: resolve compile error due to missing nfacct CPPFLAGSJan Engelhardt2012-11-271-1/+1
| | | | | | | | | | | | | | Fixes this error: make[3]: Entering directory "/home/jengelh/code/ulogd2/output" CC ulogd_output_XML.lo ulogd_output_XML.c:31:49: fatal error: libnetfilter_acct/libnetfilter_acct.h: No such file or directory (Note that pkgconfig-provided cflags are actually cppflags, so I add ${LIBNETFILTER_ACCT_CFLAGS} to AM_CPPFLAGS.) Signed-off-by: Jan Engelhardt <jengelh@inai.de>
* XML: add missing includeEric Leblond2012-11-041-0/+2
| | | | | | Inclusion of libnetfilter_acct.h is not enough as the integer type definition are not included in the header. So if NFCT is disable, the compilation fails.
* Use PRIu64 to print unsigned 64bit intEric Leblond2012-11-042-2/+4
|