summaryrefslogtreecommitdiffstats
path: root/tests
Commit message (Collapse)AuthorAgeFilesLines
* evaluate: perform mark datatype compatibility check from mapsPablo Neira Ayuso2023-09-192-0/+64
| | | | | | | | | | | | | Wrap datatype compatibility check into a helper function and use it for map evaluation, otherwise the following bogus error message is displayed: Error: datatype mismatch, map expects packet mark, mapping expression has type integer Add unit tests to improve coverage for this usecase. Fixes: 5d8e33ddb112 ("evaluate: relax type-checking for integer arguments in mark statements") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* evaluate: expand sets and maps before evaluationPablo Neira Ayuso2023-09-194-0/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 3975430b12d9 ("src: expand table command before evaluation") moved ruleset expansion before evaluation, except for sets and maps. For sets and maps there is still a post_expand() phase. This patch moves sets and map expansion to allocate an independent CMD_OBJ_SETELEMS command to add elements to named set and maps which is evaluated, this consolidates the ruleset expansion to happen always before the evaluation step for all objects, except for anonymous sets and maps. This approach avoids an interference with the set interval code which detects overlaps and merges of adjacents ranges. This set interval routine uses set->init to maintain a cache of existing elements. Then, the post_expand() phase incorrectly expands set->init cache and it triggers a bogus ENOENT errors due to incorrect bytecode (placing element addition before set creation) in combination with user declared sets using the flat syntax notation. Since the evaluation step (coming after the expansion) creates implicit/anonymous sets and maps, those are not expanded anymore. These anonymous sets still need to be evaluated from set_evaluate() path and the netlink bytecode generation path, ie. do_add_set(), needs to deal with anonymous sets. Note that, for named sets, do_add_set() does not use set->init. Such content is part of the existing cache, and the CMD_OBJ_SETELEMS command is responsible for adding elements to named sets. Fixes: 3975430b12d9 ("src: expand table command before evaluation") Reported-by: Jann Haber <jannh@selfnet.de> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* tests/shell: simplify collecting error result in "test-wrapper.sh"Thomas Haller2023-09-181-8/+8
| | | | | | | | | | | | | | | | | | | The previous pattern was unnecessarily confusing. The "$rc_{dump,valgrind,tainted}" variable should only remember whether that particular check failed, not the overall exit code of the test wrapper. Otherwise, if you want to know in which case the wrapper exits with code 122, you have to oddly follow the rc_valgrind variable. This change will make more sense, when we add another such variable, but which will be assigned the non-zero value at multiple places. Assigning there the exit code of the wrapper, duplicates the places where the condition maps to the exit code. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: colorize NFT_TEST_HAS_SOCKET_LIMITSThomas Haller2023-09-181-11/+18
| | | | | | | | | | | | | | NFT_TEST_HAS_SOCKET_LIMITS= is similar to NFT_TEST_HAVE_* variables and indicates a feature (or lack thereof), except that it's inverted. Maybe this should be consolidated, however, NFT_TEST_HAS_SOCKET_LIMITS= is detected in the root namespace, unlike the shell scripts from features. So it's unclear how to consolidate them best. Anyway. Still highlight a lack of the capability, as it can cause tests to be skipped and we should see that easily. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: don't show the exit status for failed testsThomas Haller2023-09-181-6/+3
| | | | | | | | | | | | | | | | | | | | Previously, for failed tests we would print the exit code W: [FAILED] 2/2 tests/shell/testcases/listing/0013objects_0: got 1 This doesn't seem very useful. For one, we have special exit codes like 0 (OK), 77 (SKIPPED), 124 (DUMP FAIL), 123 (TAINTED), 122 (VALGRIND). Any other exit code is just an arbitrary failure. We don't define any special codes, and printing them is not useful. Note that further exit codes (118 - 121) are reserved, and could be special purposed, when there is a use. You can find the real exit code from the test in the result data in the "rc-failed" file. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: set C locale in "run-tests.sh"Thomas Haller2023-09-181-2/+6
| | | | | | | | | | The tests should run always the same, regardless of the user's language settings. Set LANG=C and LC_ALL=C and unset LANGUAGE. If some part wants to test a different language, it would set it explicitly. They anyway wouldn't want to depend on something from the user's environment. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: fix preserving ruleset diff after testThomas Haller2023-09-181-0/+1
| | | | | | | | | We want to delete the file in the case when there was no diff (and we expect the file to be empty). The condition was wrong. Fixes: 55fe071cd193 ('tests/shell: cleanup result handling in "test-wrapper.sh"') Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: check diff in "maps/typeof_maps_0" and "sets/typeof_sets_0" testThomas Haller2023-09-182-31/+177
| | | | | | | | | | | These tests run different variants based on NFT_TEST_HAVE_osf support. Consequently, we cannot check the pre-generated diff. Instead, construct what we expect dynamically in the script, and compare the ruleset against that. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: implement NFT_TEST_HAVE_json feature detection as scriptThomas Haller2023-09-182-14/+10
| | | | | | | | No more need to special case the "run a script" approach for detecting the json feature. Use the new mechanism instead. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: skip reset tests if kernel lacks supportFlorian Westphal2023-09-185-4/+43
| | | | | | | | | | | | reset is implemented via flush + extra attribute, so older kernels perform a flush. This means .nft doesn't work, we need to check if the individual set contents/sets are still in place post-reset. Make this generic and permit use of feat.sh in addition to the simpler foo.nft feature files. Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip test cases if ct expectation and/or timeout lacks supportFlorian Westphal2023-09-185-39/+33
| | | | | Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip test cases involving osf match if kernel lacks supportFlorian Westphal2023-09-183-10/+39
| | | | | Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip catchall tests if kernel lacks supportFlorian Westphal2023-09-186-2/+35
| | | | | Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip destroy tests if kernel lacks supportFlorian Westphal2023-09-186-0/+13
| | | | | | | | Destroy support was added for table/flowtable/chain etc. in a single commit, so no need to add capability tests for each destroy subtype. Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip inet ingress tests if kernel lacks supportFlorian Westphal2023-09-184-7/+18
| | | | | | | Split the bridge autoremove test to a new file. Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip some tests if kernel lacks netdev egress supportFlorian Westphal2023-09-183-1/+18
| | | | | Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip bitshift tests if kernel lacks supportFlorian Westphal2023-09-1811-0/+27
| | | | | Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip inner matching tests if unsupportedFlorian Westphal2023-09-182-0/+9
| | | | | Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip map query if kernel lacks supportFlorian Westphal2023-09-182-8/+38
| | | | | | | | | | | | | | On recent kernels one can perform a lookup in a map without a destination register (i.e., treat the map like a set -- pure existence check). Add a feature probe and work around the missing feature in typeof_maps_add_delete: do the test with a simplified ruleset, Indicate skipped even though a reduced test was run (earlier errors cause a failure) to not trigger dump validation error. Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip netdev_chain_0 if kernel requires netdev deviceFlorian Westphal2023-09-182-0/+9
| | | | | | | | | | | | | | This test case only works on kernel 6.4+. Add feature probe for this and tag the test accordingly using the scheme added by Thomas Haller in "tests/shell: skip tests if nft does not support JSON mode" so that run-test.sh skips it if kernel requires a device. Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: add and use chain binding feature probeFlorian Westphal2023-09-184-7/+62
| | | | | | | | | | Alter 30s-stress to suppress anon chains when its unuspported. Note that 30s-stress is optionally be run standalone, so also update the test script. Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: cleanup creating dummy interfaces in testsThomas Haller2023-09-186-31/+32
| | | | | | | | | | | | | | | | | | | | | | | In "tests/shell/testcases/chains/netdev_chain_0", calling "trap ... EXIT" multiple times does not work. Fix it, by calling one cleanup function. Note that we run in separate namespaces, so the cleanup is usually not necessary. Still do it, we might want to run without unshare (via NFT_TEST_UNSHARE_CMD=""). Without unshare, it's important that the cleanup always works. In practice it might not, for example, "trap ... EXIT" does not run for SIGTERM. A leaked interface might break the follow up test and tests interfere with each other. Try to workaround that by first trying to delete the interface. Also failures to create the interfaces are not considered fatal. I don't understand under what circumstances this might fail, note that there are other tests that create dummy interface and don't "exit 77" on failure. We want to know when something odd is going on. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: suggest 4Mb /proc/sys/net/core/{wmem_max,rmem_max} for rootlessThomas Haller2023-09-181-3/+3
| | | | | | | | | | | | | | | | 2Mb was not enough to pass "tests/shell/testcases/sets/0030add_many_elements_interval_0" in an unprivileged/rootless namespace. Instead, bump the suggestion to 4Mb, which lets the test pass. Note that the 4Mb are only the recommended value when running the test as rootless, and is used to autodetect NFT_TEST_HAS_SOCKET_LIMITS=y. You can set whatever values are suitable for your environment, and explicitly indicate whether the limits are appropriate or not via NFT_TEST_HAS_SOCKET_LIMITS=n|y. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests: shell: add sample ruleset reproducerFlorian Westphal2023-09-182-0/+501
| | | | | | | | | | | | Changes on kernel side no longer permit transactions that reference a chain after it is bound. This test case breaks when run with nftables 1.0.6 and earlier. Keep this as a test case in tree to catch any future problems in this area. Link: https://lore.kernel.org/netfilter-devel/20230911213750.5B4B663206F5@dd20004.kasserver.com/ Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: colorize NFT_TEST_SKIP_/NFT_TEST_HAVE_ in test outputThomas Haller2023-09-161-2/+10
| | | | | | | | | Having a "SKIP" option as "y" or a "HAVE" option as "n", is note worthy because tests may be skipped based on that. Colorize, to make it easier to see in the test output. Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: add feature probing via "features/*.nft" filesThomas Haller2023-09-161-0/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Running selftests on older kernels makes some of them fail very early because some tests use features that are not available on older kernels, e.g. -stable releases. Known examples: - inner header matching - anonymous chains - elem delete from packet path Also, some test cases might fail because a feature isn't compiled in, such as netdev chains. This adds a feature-probing mechanism to shell tests. Simply drop a 'nft -f' compatible file with a .nft suffix into "tests/shell/features". "run-tests.sh" will load it via `nft --check` and will export NFT_TEST_HAVE_${feature}=y|n Here ${feature} is the basename of the .nft file without file extension. It must be all lower-case. This extends the existing NFT_TEST_HAVE_json= feature detection. Similarly, NFT_TEST_REQUIRES(NFT_TEST_HAVE_*) tags work to easily skip a test. The test script that cannot fully work without the feature should either skip the test entirely (NFT_TEST_REQUIRES(NFT_TEST_HAVE_*)), or run a reduced/modified test. If a modified test was run and passes, it is still a good idea to mark the overall result as skipped (exit 77) instead of claiming success to the modified test. We want to know when not the full test was running, while we want to test as much as we can. This patch is based on Florian's feature probing patch. Originally-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests: shell: fix dump validation messageFlorian Westphal2023-09-162-13/+0
| | | | | | | | | | | | This test output depends on CONFIG_HZ: - update @y { ip saddr timeout 1d2h3m4s8ms } + update @y { ip saddr timeout 1d2h3m4s10ms } The dump record is with HZ=1000, on HZ=250 we get failure. Remove the dump file for now. Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/build: capture more output from "tests/build/run-tests.sh" scriptThomas Haller2023-09-151-6/+6
| | | | | | | | | Dropping stdout for various build tests makes it hard to understand what happens, when a build fails. Redirect both stdout and stderr to the log files for easier debugging. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: honor CLICOLOR_FORCE to force coloring in run-tests.shThomas Haller2023-09-151-5/+10
| | | | | | | | | | | | | We honor NO_COLOR= to disable coloring, let's also honor CLICOLOR_FORCE= to enable it. The purpose will be for `make` calling the script and redirecting to a file, while enabling colors. See-also: https://bixense.com/clicolors/ Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: accept $NFT_TEST_TMPDIR_TAG for the result directoryThomas Haller2023-09-151-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We allow the user to set "$TMPDIR" to affect where the "nft-test.*" directory is created. However, we don't allow the user to specify the exact location, so the user doesn't really know which directory was created. One remedy is that the test will also create the symlink "$TMPDIR/nft-test.latest.$USER" to point to the last test result. However, if you run multiple tests in parallel, that is not reliable to find the test results. Accept $NFT_TEST_TMPDIR_TAG and use it as part of the generated filename. That way, the caller can set it to a unique tag, and find the directory later based on that. For example export TMPDIR=/tmp export NFT_TEST_TMPDIR_TAG=".$(uuidgen)" ./tests/shell/run-tests.sh ls -lad "$TMPDIR/nft-test."*"$NFT_TEST_TMPDIR_TAG"*/ will work reliably -- as long as the tag is chosen uniquely. The reason to not allow the user to specify the directory name directly, is because we want that tests results follow the well-known pattern "/tmp/nft-test*". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: exit 77 from "run-tests.sh" if all tests were skippedThomas Haller2023-09-151-1/+7
| | | | | | | | | | | | | | If there are multiple tests and some of them pass and some are skipped, the overall result should be success (zero). Because likely the user just selected a bunch of tests (or all of them). So skipping some tests does not mean that the entire run is not a success. However, if all tests are skipped, then mark the overall result as skipped too. The more common case is if you only run one single test, then we want to know, that the test didn't run. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: in find_tests() use C locale for sorting tests namesThomas Haller2023-09-151-1/+1
| | | | | | | | It makes more sense, that the sort order does not depend on the user's locale. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: special handle base path starting with "./"Thomas Haller2023-09-151-1/+3
| | | | | | | | | | | | | | | | | | When we auto detect the tests with `tests/shell/run-tests.sh -L`, then commonly the NFT_TEST_BASEDIR starts with a redundant "./". That's a bit ugly. Instead, special handle that case and remove the prefix. The effect is that `tests/shell/run-tests.sh -L` shows tests/shell/testcases/bitwise/0040mark_binop_0 instead of ./tests/shell/testcases/bitwise/0040mark_binop_0 Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add missing nft/nodump files for testsThomas Haller2023-09-153-0/+29
| | | | | | | | | | Three tests didn't have a nft/nodump file, because previously I only generated files on Fedora kernel, where those tests are failing. Generate them on CentOS-Stream-9 with kernel 5.14.0-354.el9.x86_64. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: drop unstable dump for "transactions/0051map_0" testThomas Haller2023-09-152-7/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The file "tests/shell/testcases/transactions/dumps/0051map_0.nft" gets generated differently on Fedora 38 (6.4.14-200.fc38.x86_64) and CentOS-Stream-9 (5.14.0-354.el9.x86_64). It's not stable. diff --git c/tests/shell/testcases/transactions/dumps/0051map_0.nft w/tests/shell/testcases/transactions/dumps/0051map_0.nft index 59d69df70e61..fa7df9f93757 100644 --- c/tests/shell/testcases/transactions/dumps/0051map_0.nft +++ w/tests/shell/testcases/transactions/dumps/0051map_0.nft @@ -1,7 +1,11 @@ table ip x { + chain w { + } + chain m { } chain y { + ip saddr vmap { 1.1.1.1 : jump w, 2.2.2.2 : accept, 3.3.3.3 : goto m } } } Drop it. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: remove spurious .nft dump filesThomas Haller2023-09-152-16/+0
| | | | | | | | | | | These are left-over dumps ([1]), or dumps generated with the wrong name ([2]). Remove the files. [1] commit eb14363d44ce ('tests: shell: move chain priority and policy to chain folder') [2] commit b4775dec9f80 ('src: ingress inet support') Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add option to shuffle execution order of testsThomas Haller2023-09-151-0/+18
| | | | | | | | | | | | The user can set NFT_TEST_SHUFFLE_TESTS=y|n to have the tests shuffled randomly. The purpose of shuffling is to find tests that depend on each other, or would break when run in unexpected order. If unspecified, by default tests are shuffled if no tests are selected on the command line. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add "random-source.sh" helper for random-source for sort/shufThomas Haller2023-09-152-1/+41
| | | | | | | | | | | | | | | | | Commands `sort` and `shuf` have a "--random-source" argument. That's useful for generating stable, reproducible "random" output. However, we want to do this based on a fixed seed, while the "--random-source" expects a stream of randomness. Add a helper script for that. Also, use the stable randomness for shuf in the test "tests/shell/testcases/sets/automerge_0". See-also: https://www.gnu.org/software/coreutils/manual/html_node/Random-sources.html#Random-sources Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: export NFT_TEST_RANDOM_SEED variable for testsThomas Haller2023-09-151-0/+46
| | | | | | | | | | | | | | | | | | | | | | | | Let "run-tests.sh" export a NFT_TEST_RANDOM_SEED variable, set to a decimal, random integer (in the range of 0 to 0x7FFFFFFF). The purpose is to provide a seed to tests for randomization. Randomizing tests is very useful to increase the coverage while not testing all combinations (which might not be practical). The point of NFT_TEST_RANDOM_SEED is that the user can set the environment variable so that the same series of random events is used. That is useful for reproducing an issue, that is known to happen with a certain seed. - by default, if the user leaves NFT_TEST_RANDOM_SEED unset or empty, the script generates a number using $SRANDOM. - if the user sets NFT_TEST_RANDOM_SEED to an integer it is taken as is (modulo 0x80000000). - otherwise, calculate a number by hashing the value of $NFT_TEST_RANDOM_SEED. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: ensure vgdb-pipe files are deleted from "nft-valgrind-wrapper.sh"Thomas Haller2023-09-142-3/+11
| | | | | | | | | | | | | When the valgrind process gets killed, those files can be left over. They are located in the original $TMPDIR (usually /tmp). They should be cleaned up. I tried to cleanup the files from withing "nft-valgrind-wrapper.sh" itself via a `trap`, but it doesn't work. Instead, let "run-tests.sh" delete all files with a matching pattern. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: kill running child processes when aborting "run-tests.sh"Thomas Haller2023-09-141-5/+26
| | | | | | | | | | | | | | When aborting "run-tests.sh", child processes were left running. Kill them. It's surprisingly complicated to get this somewhat right. Do it by enabling monitor mode for each test call, so that they run in separate process groups and we can kill the entire group. Note that we cannot just `kill -- -$$`, because it's not clear who is in this process group. Also, we don't want to kill the `tee` process which handles our logging. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: typeof_integer/raw: prefer @nh for payload matchingFlorian Westphal2023-09-106-12/+12
| | | | | | | | | | @ih fails on kernels where payload expression doesn't support the 'inner' base offset. This test isn't about inner headers, so just use @nh which is universally available. Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: make delete_by_handle test work on older releasesFlorian Westphal2023-09-101-1/+1
| | | | | | | | | | This test fails on kernels that lack 05abe4456fa3 ("netfilter: nf_tables: allow to register flowtable with no devices") v5.8-rc1~165^2~27^2~1 Just add lo as dummy device. Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add "--quick" option to skip slow tests (via NFT_TEST_SKIP_slow=y)Thomas Haller2023-09-0910-12/+68
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It's important to run (a part) of the tests in a timely manner. Add an option to skip long running tests. Thereby, add a more general NFT_TEST_SKIP_* mechanism. This is related and inverse from "NFT_TEST_HAVE_json", where a test can require [ "$NFT_TEST_HAVE_json" != n ] to run, but is skipped when [ "$NFT_TEST_SKIP_slow" = y ]. Currently only NFT_TEST_SKIP_slow is supported. The user can set such environment variables (or use the -Q|--quick command line option). The configuration is printed in the test info. Tests should check for [ "$NFT_TEST_SKIP_slow" = y ] so that the variable has to be explicitly set to opt-out. For convenience, tests can also add a # NFT_TEST_SKIP(NFT_TEST_SKIP_slow) tag, which is evaluated by test-wrapper.sh. Or they can run a quick, reduced part of the test, but then should still indicate to be skipped. Mark 8 tests are as slow, that take longer than 5 seconds on my machine. With this, a parallel wall time for the non-slow tests is only 7 seconds (on my machine). The ultimate point is to integrate a call to "tests/shell/run-tests.sh" in a `make check` target. For development, you can then export NFT_TEST_SKIP_slow=y and have a fast `make check`. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: skip tests if nft does not support JSON modeThomas Haller2023-09-0911-9/+112
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We can build nft without JSON support, and some tests will fail without it. Instead, they should be skipped. Also note, that the test accepts any nft binary via the "NFT" environment variable. So it's not enough to make the skipping dependent on build configuration, but on the currently used $NFT variable. Let "run-test.sh" detect and export a "NFT_TEST_HAVE_json=y|n" variable. This is heavily inspired by Florian's feature probing patches. Tests that require JSON can check that variable, and skip. Note that they check in the form of [ "$NFT_TEST_HAVE_json" != n ], so the test is only skipped, if we explicitly detect lack of support. That is, don't check via [ "$NFT_TEST_HAVE_json" = y ]. Some of the tests still run parts of the tests that don't require JSON. Only towards the end of such partial run, mark the test as skipped. Some tests require JSON support throughout. For those, add a mechanism where tests can add a tag (in their first 10 lines): # NFT_TEST_REQUIRES(NFT_TEST_HAVE_json) This will be checked by "test-wrapper.sh", which will skip the test. The purpose of this is to make it low-effort to skip a test and to print the reason in the text output as Test skipped due to NFT_TEST_HAVE_json=n (test has "NFT_TEST_REQUIRES(NFT_TEST_HAVE_json)" tag) This is intentionally not shortened to NFT_TEST_REQUIRES(json), so that we can grep for NFT_TEST_HAVE_json to find all relevant places. Note that while NFT_TEST_HAVE_json is autodetected, the caller can also force it by setting the environment variable. This allows to see what would happen to such a test. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print number of completed tests to show progressThomas Haller2023-09-091-7/+13
| | | | | | | | | | | | | | | | | | Especially with VALGRIND=y, a full test run can take a long time. When looking at the output, it's interesting to get a feel how far along we are. Print the number of completed jobs vs. the number of total jobs, in the line showing the test result. It gives a nice progress status. Example: I: [OK] 1/373 ./tests/shell/testcases/bitwise/0040mark_binop_1 I: [OK] 2/373 ./tests/shell/testcases/bitwise/0040mark_binop_0 ... Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: set valgrind's "--vgdb-prefix=" to orignal TMPDIRThomas Haller2023-09-092-0/+6
| | | | | | | | | | | | | | | | | | | | | | | "test-wrapper.sh" sets TMPDIR="$NFT_TEST_TESTTMPDIR". That is useful, so that temporary files of the tests are placed inside the test result data. Sometimes tests miss to delete those files, which would result in piling up /tmp/tmp.XXXXXXXXXX files. By setting $TMPDIR, those files are clearly related to the test run that created them, and can be deleted together. However, valgrind likes to create files like "vgdb-pipe-from-vgdb-to-68-by-thom-on-???" inside $TMPDIR. These are pipes, so if you run `grep -R ^ /tmp/nft-test.latest` while the test is still running (to inspect the results), then the process hands reading from the pipe. Instead, tell valgrind to put those files in the original TMPDIR. For that purpose, export NFT_TEST_TMPDIR_ORIG from "run-tests.sh". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add ".nft" dump files for tests without dumps/ directoryThomas Haller2023-09-096-0/+0
| | | | | | | | | | | | | | DUMPGEN=y mode skips tests that don't have a corresponding "dumps/" directory. Add the "dumps/" directory for tests that lacked it, and generate ".nft" files by running `./tests/shell/run-tests.sh -g`. Yes, they are all empty. Not very exciting, but why not check for that too? Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add missing ".nodump" file for tests without dumpsThomas Haller2023-09-098-0/+0
| | | | | | | | | | | | These files are generated by running `./tests/shell/run-tests.sh -g`. Commit the .nodump files to git. The point is to explicitly make it known that no dump file should be there. This prevents `./tests/shell/run-tests.sh -g` from generating the files and proposing (over and over) to add them to git. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: generate and add ".nft" dump files for existing testsThomas Haller2023-09-09161-0/+2546
| | | | | | | | | | | | | | | | | | | | | | | | | | | Several tests didn't have a ".nft" dump file committed. Generate one and commit it to git. While not all tests have a stable ruleset to compare, many have. Commit the .nft files for the tests where the output appears to be stable. This was generated by running `./tests/shell/run-tests.sh -g` twice, and commit the files that were identical both times. Note that 7 tests on my machine fail, so those are skipped. Also skip the files tests/shell/testcases/maps/dumps/0004interval_map_create_once_0.nft tests/shell/testcases/nft-f/dumps/0011manydefines_0.nft tests/shell/testcases/sets/dumps/0011add_many_elements_0.nft tests/shell/testcases/sets/dumps/0030add_many_elements_interval_0.nft tests/shell/testcases/sets/dumps/0068interval_stack_overflow_0.nft Those files are larger than 100KB, and I don't think we want to blow up the git repository this way. Even if they are only text files and compress well. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>