summaryrefslogtreecommitdiffstats
path: root/tests/shell/testcases/optimizations/dumps
Commit message (Collapse)AuthorAgeFilesLines
* tests: shell: Regenerate all json-nft dumpsPhil Sutter2024-03-193-37/+37
| | | | | | | | Ordering of 'nft -j list ruleset' output has changed, Regenerate existing json-nft dumps. No functional change intended, merely the position of chain objects should have moved up in the "nftables" array. Signed-off-by: Phil Sutter <phil@nwl.cc>
* tests: shell: add more json dumpsFlorian Westphal2024-02-262-0/+568
| | | | | | | | | | | | Those are expected to be stable, so add them. Some are not 100% correct, as "typeof" is misprinted as "type" (json output and input parser lack support for this), but for these files the "type" is valid too. This will allow better validation once proper "typeof" support is added to json.c and json-parser.c. Signed-off-by: Florian Westphal <fw@strlen.de>
* tests: shell: Pretty-print all *.json-nft dumpsPhil Sutter2024-02-0817-17/+3924
| | | | | | | | | | | | | The problem with single line output as produced by 'nft -j list ruleset' is its incompatibility to unified diff format as any change in this single line will produce a diff which contains the old and new lines in total. This is not just unreadable but will blow up patches which may exceed mailinglists' mail size limits. Convert them all at once by feeding their contents to tests/shell/helpers/json-pretty.sh. Signed-off-by: Phil Sutter <phil@nwl.cc>
* tests/shell: sanitize "handle" in JSON outputThomas Haller2023-11-2215-15/+15
| | | | | | | | | The "handle" in JSON output is not stable. Sanitize/normalize to zero. Adjust the sanitize code, and regenerate the .json-nft files. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* tests: shell: split nat inet testsPablo Neira Ayuso2023-11-222-11/+11
| | | | | | Detach nat inet from existing tests not to reduce test coverage. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* tests/shell: add JSON dump filesThomas Haller2023-11-1517-0/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Generate and add ".json-nft" files. These files contain the output of `nft -j list ruleset` after the test. Also, "test-wrapper.sh" will compare the current ruleset against the ".json-nft" files and test them with "nft -j --check -f $FILE`. These are useful extra tests, that we almost get for free. Note that for some JSON dumps, `nft -f --check` fails (or prints something). For those tests no *.json-nft file is added. The bugs needs to be fixed first. An example of such an issue is: $ DUMPGEN=all ./tests/shell/run-tests.sh tests/shell/testcases/maps/nat_addr_port which gives a file "rc-failed-chkdump" with Command `./tests/shell/../../src/nft -j --check -f "tests/shell/testcases/maps/dumps/nat_addr_port.json-nft"` failed >>>> internal:0:0-0: Error: Invalid map type 'ipv4_addr . inet_service'. internal:0:0-0: Error: Parsing command array at index 3 failed. internal:0:0-0: Error: unqualified type integer specified in map definition. Try "typeof expression" instead of "type datatype". <<<< Tests like "tests/shell/testcases/nft-f/0012different_defines_0" and "tests/shell/testcases/nft-f/0024priority_0" also don't get a .json-nft dump yet, because their output is not stable. That needs fixing too. Cc: Pablo Neira Ayuso <pablo@netfilter.org> Cc: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests: shell: split merge nat optimization in two testsPablo Neira Ayuso2023-11-112-8/+8
| | | | | | | One without pipapo support and another with not to harm existing coverage. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* tests: shell: split single element in anonymous setPablo Neira Ayuso2023-11-112-1/+5
| | | | | | | Split this to move set stateful expression support into a separated test not to harm existing coverage. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* tests/shell: inline input data in "single_anon_set" testThomas Haller2023-10-241-38/+0
| | | | | | | | | | | | | | | | | The file "optimizations/dumps/single_anon_set.nft.input" was laying around, and it was unclear how it was used. Let's extend "check-patch.sh" to flag all unused files. But the script cannot understand how "single_anon_set.nft.input" is used (aside allow listing it). Instead, inline the script to keep it inside the test (script). We still write the data to a separate file and don't use `nft -f -` (because reading stdin uses a different code path we want to cover). Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* tests/shell: generate and add ".nft" dump files for existing testsThomas Haller2023-09-092-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | Several tests didn't have a ".nft" dump file committed. Generate one and commit it to git. While not all tests have a stable ruleset to compare, many have. Commit the .nft files for the tests where the output appears to be stable. This was generated by running `./tests/shell/run-tests.sh -g` twice, and commit the files that were identical both times. Note that 7 tests on my machine fail, so those are skipped. Also skip the files tests/shell/testcases/maps/dumps/0004interval_map_create_once_0.nft tests/shell/testcases/nft-f/dumps/0011manydefines_0.nft tests/shell/testcases/sets/dumps/0011add_many_elements_0.nft tests/shell/testcases/sets/dumps/0030add_many_elements_interval_0.nft tests/shell/testcases/sets/dumps/0068interval_stack_overflow_0.nft Those files are larger than 100KB, and I don't think we want to blow up the git repository this way. Even if they are only text files and compress well. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* libnftables: Drop cache in -c/--check modePablo Neira Ayuso2023-08-011-0/+11
| | | | | | | | | | | | | | | | | | | | | Extend e0aace943412 ("libnftables: Drop cache in error case") to also drop the cache with -c/--check, this is a dry run mode and kernel does not get any update. This fixes a bug with -o/--optimize, which first runs in an implicit -c/--check mode to validate that the ruleset is correct, then it provides the proposed optimization. In this case, if the cache is not emptied, old objects in the cache refer to scanner data that was already released, which triggers BUG like this: BUG: invalid input descriptor type 151665524 nft: erec.c:161: erec_print: Assertion `0' failed. Aborted This bug was triggered in a ruleset that contains a set for geoip filtering. This patch also extends tests/shell to cover this case. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: do not remove counter in verdict mapsPablo Neira Ayuso2023-05-101-0/+4
| | | | | | | | | | | | | Add counter to set element instead of dropping it: # nft -c -o -f test.nft Merging: test.nft:6:3-50: ip saddr 1.1.1.1 ip daddr 2.2.2.2 counter accept test.nft:7:3-48: ip saddr 1.1.1.2 ip daddr 3.3.3.3 counter drop into: ip daddr . ip saddr vmap { 2.2.2.2 . 1.1.1.1 counter : accept, 3.3.3.3 . 1.1.1.2 counter : drop } Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* evaluate: skip optimization if anonymous set uses stateful statementPablo Neira Ayuso2023-05-102-0/+4
| | | | | | | | fee6bda06403 ("evaluate: remove anon sets with exactly one element") introduces an optimization to remove use of sets with single element. Skip this optimization if set element contains stateful statements. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: support for redirect and masqueradePablo Neira Ayuso2023-04-051-0/+4
| | | | | | | | | | | | | | The redirect and masquerade statements can be handled as verdicts: - if redirect statement specifies no ports. - masquerade statement, in any case. Exceptions to the rule: If redirect statement specifies ports, then nat map transformation can be used iif both statements specify ports. Closes: https://bugzilla.netfilter.org/show_bug.cgi?id=1668 Fixes: 0a6dbfce6dc3 ("optimize: merge nat rules with same selectors into map") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: infer family for nat mappingPablo Neira Ayuso2023-02-211-0/+11
| | | | | | | | | | | | | | | Infer family from key in nat mapping, otherwise nat mapping via merge breaks since family is not specified. Merging: fw-test-bug2.nft:4:9-78: iifname enp2s0 ip daddr 72.2.3.66 tcp dport 53122 dnat to 10.1.1.10:22 fw-test-bug2.nft:5:9-77: iifname enp2s0 ip daddr 72.2.3.66 tcp dport 443 dnat to 10.1.1.52:443 fw-test-bug2.nft:6:9-75: iifname enp2s0 ip daddr 72.2.3.70 tcp dport 80 dnat to 10.1.1.52:80 into: dnat ip to iifname . ip daddr . tcp dport map { enp2s0 . 72.2.3.66 . 53122 : 10.1.1.10 . 22, enp2s0 . 72.2.3.66 . 443 : 10.1.1.52 . 443, enp2s0 . 72.2.3.70 . 80 : 10.1.1.52 . 80 } Closes: https://bugzilla.netfilter.org/show_bug.cgi?id=1657 Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: ignore existing nat mappingPablo Neira Ayuso2023-02-071-0/+1
| | | | | | | | | | | | | | | | User might be already using a nat mapping in their ruleset, use the unsupported statement when collecting statements in this case. # nft -c -o -f ruleset.nft nft: optimize.c:443: rule_build_stmt_matrix_stmts: Assertion `k >= 0' failed. Aborted The -o/--optimize feature only cares about linear rulesets at this stage, but do not hit assert() in this case. Closes: https://bugzilla.netfilter.org/show_bug.cgi?id=1656 Fixes: 0a6dbfce6dc3 ("optimize: merge nat rules with same selectors into map") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: select merge criteria based on candidates rulesPablo Neira Ayuso2023-02-061-0/+4
| | | | | | | | | | | | | Select the merge criteria based on the statements that are used in the candidate rules, instead of using the list of statements in the given chain. Update tests to include a rule with a verdict, which triggers the bug described in the bugzilla ticket. Closes: https://bugzilla.netfilter.org/show_bug.cgi?id=1657 Fixes: 0a6dbfce6dc3 ("optimize: merge nat rules with same selectors into map") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: fix incorrect expansion into concatenation with verdict mapPablo Neira Ayuso2023-02-021-0/+4
| | | | | | | | | | | | | | | | | | # nft -c -o -f ruleset.nft Merging: ruleset.nft:3:3-53: meta pkttype broadcast udp dport { 67, 547 } accept ruleset.nft:4:17-58: meta pkttype multicast udp dport 1900 drop into: meta pkttype . udp dport vmap { broadcast . { 67, 547 } : accept, multicast . 1900 : drop } ruleset.nft:3:38-39: Error: invalid data type, expected concatenation of (packet type, internet network service) meta pkttype broadcast udp dport { 67, 547 } accept ^^ Similar to 187c6d01d357 ("optimize: expand implicit set element when merging into concatenation") but for verdict maps. Reported-by: Simon G. Trajkovski <neur0armitage@proton.me> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: handle prefix and range when merging into set + concatenationPablo Neira Ayuso2022-11-051-1/+1
| | | | | | | | | | | | | | | | The following ruleset fails to be merged using set + concatenation: meta iifname eth1 ip saddr 1.1.1.2 ip daddr 2.2.3.0/24 accept meta iifname eth1 ip saddr 1.1.1.2 ip daddr 2.2.4.0-2.2.4.10 accept hitting the following assertion: nft: optimize.c:585: __merge_concat_stmts: Assertion `0' failed. Abort This patch also updates tests/shell. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* evaluate: bogus datatype assertion in binary operation evaluationPablo Neira Ayuso2022-10-061-0/+7
| | | | | | | | | | Use datatype_equal(), otherwise dynamically allocated datatype fails to fulfill the datatype pointer check, triggering the assertion: nft: evaluate.c:1249: expr_evaluate_binop: Assertion `expr_basetype(left) == expr_basetype(right)' failed. Closes: https://bugzilla.netfilter.org/show_bug.cgi?id=1636 Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: expand implicit set element when merging into concatenationPablo Neira Ayuso2022-08-301-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | Generalize the existing code to deal with implicit sets. When merging a ruleset like the following: udp dport 128 iifname "foo" #1 udp dport { 67, 123 } iifname "bar" #2 into a concatenation of statements, the following expansion need to be done for rule #2: 67 . "bar" 123 . "bar" The expansion logic consists of cloning the existing concatenation being built and then append each element in the implicit set. A list of ongoing concatenations being built is maintained, so further expansions are also supported. Extend test to cover for this use-case. Closes: https://bugzilla.netfilter.org/show_bug.cgi?id=1628 Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: check for mergeable rulesPablo Neira Ayuso2022-08-111-0/+12
| | | | | | Rules that are equal need to have at least one mergeable statement. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: merging concatenation is unsupportedPablo Neira Ayuso2022-08-111-0/+1
| | | | | | | | | | | Existing concatenation cannot be merge at this stage, skip them otherwise this assertion is hit: nft: optimize.c:434: rule_build_stmt_matrix_stmts: Assertion `k >= 0' failed Extend existing test to cover this. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: assume verdict is same when rules have no verdictPablo Neira Ayuso2022-06-231-0/+6
| | | | Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: only merge OP_IMPLICIT and OP_EQ relationalPablo Neira Ayuso2022-06-231-0/+6
| | | | | | Add test to cover this case. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: add unsupported statementPablo Neira Ayuso2022-06-231-0/+7
| | | | | | | | | | | | | | | | Do not try to merge rules with unsupported statements. This patch adds a dummy unsupported statement which is included in the statement collection and the rule vs statement matrix. When looking for possible rule mergers, rules using unsupported statements are discarded, otherwise bogus rule mergers might occur. Note that __stmt_type_eq() already returns false for unsupported statements. Add a test using meta mark statement, which is not yet supported. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: fix reject statementPablo Neira Ayuso2022-06-231-0/+7
| | | | | | | | Add missing code to the statement collection routine. Compare reject expressions when available. Add tests/shell. Fixes: fb298877ece2 ("src: add ruleset optimization infrastructure") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: do not merge rules with set reference in rhsPablo Neira Ayuso2022-06-231-0/+23
| | | | | | | Otherwise set reference ends up included in an anonymous set, as an element, which is not supported. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: segfault when releasing unsupported statementPablo Neira Ayuso2022-06-011-0/+7
| | | | | | | | | Call xfree() instead since stmt_alloc() does not initialize the statement type fields. Closes: https://bugzilla.netfilter.org/show_bug.cgi?id=1609 Fixes: ea1f1c9ff608 ("optimize: memleak in statement matrix") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: merge nat rules with same selectors into mapPablo Neira Ayuso2022-05-031-0/+20
| | | | | | | | | | | | | | | | | | | | | | | Verdict and nat are mutually exclusive, no need to support for this combination. # cat ruleset.nft table ip x { chain y { type nat hook postrouting priority srcnat; policy drop; ip saddr 1.1.1.1 tcp dport 8000 snat to 4.4.4.4:80 ip saddr 2.2.2.2 tcp dport 8001 snat to 5.5.5.5:90 } } # nft -o -c -f ruleset.nft Merging: ruleset.nft:4:3-52: ip saddr 1.1.1.1 tcp dport 8000 snat to 4.4.4.4:80 ruleset.nft:5:3-52: ip saddr 2.2.2.2 tcp dport 8001 snat to 5.5.5.5:90 into: snat to ip saddr . tcp dport map { 1.1.1.1 . 8000 : 4.4.4.4 . 80, 2.2.2.2 . 8001 : 5.5.5.5 . 90 } Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: Restore optimization for raw payload expressionsPablo Neira Ayuso2022-03-291-0/+31
| | | | | | | | This patch reverts d0f14b5337e7 ("optimize: do not merge raw payload expressions") after adding support for concatenation with variable length TYPE_INTEGER. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: do not assume log prefixPablo Neira Ayuso2022-03-041-0/+1
| | | | | | ... log prefix might not be present in log statements. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: more robust statement merge with vmapPablo Neira Ayuso2022-03-031-1/+1
| | | | | | | | | | | | | | Check expressions that are expected on the rhs rather than using a catch-all default case. Actually, lists and sets need to be their own routine, because this needs the set element key expression to be merged. This is a follow up to 99eb46969f3d ("optimize: fix vmap with anonymous sets"). Fixes: 1542082e259b ("optimize: merge same selector with different verdict into verdict map") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: fix vmap with anonymous setsPablo Neira Ayuso2022-03-031-0/+4
| | | | | | | | | | | | | | | | The following example ruleset crashes: table inet a { chain b { tcp dport { 1 } accept tcp dport 2-3 drop } } because handling for EXPR_SET is missing. Fixes: 1542082e259b ("optimize: merge same selector with different verdict into verdict map") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: merge verdict maps with same lookup keyPablo Neira Ayuso2022-01-261-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Merge two consecutive verdict maps with the same lookup key. For instance, merge the following: table inet x { chain filter_in_tcp { tcp dport vmap { 80 : accept, 81 : accept, 443 : accept, 931 : accept, 5001 : accept, 5201 : accept, } tcp dport vmap { 6800-6999 : accept, 33434-33499 : accept, } } } into: table inet x { chain filter_in_tcp { tcp dport vmap { 80 : accept, 81 : accept, 443 : accept, 931 : accept, 5001 : accept, 5201 : accept, 6800-6999 : accept, 33434-33499 : accept, } } } This patch updates statement comparison routine to inspect the verdict expression type to detect possible merger. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: merge several selectors with different verdict into verdict mapPablo Neira Ayuso2022-01-151-0/+5
| | | | | | | | | | | | | Transform: ip saddr 1.1.1.1 ip daddr 2.2.2.2 accept ip saddr 2.2.2.2 ip daddr 3.3.3.3 drop into: ip saddr . ip daddr vmap { 1.1.1.1 . 2.2.2.2 : accept, 2.2.2.2 . 3.3.3.3 : drop } Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: merge same selector with different verdict into verdict mapPablo Neira Ayuso2022-01-151-0/+5
| | | | | | | | | | | | | Transform: ct state invalid drop ct state established,related accept into: ct state vmap { established : accept, related : accept, invalid : drop } Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* optimize: merge rules with same selectors into a concatenationPablo Neira Ayuso2022-01-151-0/+5
| | | | | | | | | | | | | | | | | This patch extends the ruleset optimization infrastructure to collapse several rules with the same selectors into a concatenation. Transform: meta iifname eth1 ip saddr 1.1.1.1 ip daddr 2.2.2.3 accept meta iifname eth1 ip saddr 1.1.1.2 ip daddr 2.2.2.5 accept meta iifname eth2 ip saddr 1.1.1.3 ip daddr 2.2.2.6 accept into: meta iifname . ip saddr . ip daddr { eth1 . 1.1.1.1 . 2.2.2.6, eth1 . 1.1.1.2 . 2.2.2.5 , eth1 . 1.1.1.3 . 2.2.2.6 } accept Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* src: add ruleset optimization infrastructurePablo Neira Ayuso2022-01-151-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds a new -o/--optimize option to enable ruleset optimization. You can combine this option with the dry run mode (--check) to review the proposed ruleset updates without actually loading the ruleset, e.g. # nft -c -o -f ruleset.test Merging: ruleset.nft:16:3-37: ip daddr 192.168.0.1 counter accept ruleset.nft:17:3-37: ip daddr 192.168.0.2 counter accept ruleset.nft:18:3-37: ip daddr 192.168.0.3 counter accept into: ip daddr { 192.168.0.1, 192.168.0.2, 192.168.0.3 } counter packets 0 bytes 0 accept This infrastructure collects the common statements that are used in rules, then it builds a matrix of rules vs. statements. Then, it looks for common statements in consecutive rules which allows to merge rules. This ruleset optimization always performs an implicit dry run to validate that the original ruleset is correct. Then, on a second pass, it performs the ruleset optimization and add the rules into the kernel (unless --check has been specified by the user). From libnftables perspective, there is a new API to enable this feature: uint32_t nft_ctx_get_optimize(struct nft_ctx *ctx); void nft_ctx_set_optimize(struct nft_ctx *ctx, uint32_t flags); This patch adds support for the first optimization: Collapse a linear list of rules matching on a single selector into a set as exposed in the example above. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* netlink_delinearize: incorrect meta protocol dependency kill againPablo Neira Ayuso2021-09-031-0/+42
| | | | | | | | | | | | | This patch adds __meta_dependency_may_kill() to consolidate inspection of the meta protocol, nfproto and ether type expression to validate dependency removal on listings. Phil reports that 567ea4774e13 includes an update on the ip and ip6 families that is not described in the patch, moreover, it flips the default verdict from true to false. Fixes: 567ea4774e13 ("netlink_delinearize: incorrect meta protocol dependency kill") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* tests: add test case for removal of anon sets with only a single elementFlorian Westphal2021-06-072-0/+50
Also add a few examples that should not be changed: - anon set with 2 elements - anon map with 1 element - anon set with a concatenation The latter could be done with cmp but this currently triggers 'Error: Use concatenations with sets and maps, not singleton values' after removing the anon set. Signed-off-by: Florian Westphal <fw@strlen.de>