summaryrefslogtreecommitdiffstats
path: root/tests
Commit message (Collapse)AuthorAgeFilesLines
* tests/shell: skip reset tests if kernel lacks supportFlorian Westphal2023-09-185-4/+43
| | | | | | | | | | | | reset is implemented via flush + extra attribute, so older kernels perform a flush. This means .nft doesn't work, we need to check if the individual set contents/sets are still in place post-reset. Make this generic and permit use of feat.sh in addition to the simpler foo.nft feature files. Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip test cases if ct expectation and/or timeout lacks supportFlorian Westphal2023-09-185-39/+33
| | | | | Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip test cases involving osf match if kernel lacks supportFlorian Westphal2023-09-183-10/+39
| | | | | Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip catchall tests if kernel lacks supportFlorian Westphal2023-09-186-2/+35
| | | | | Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip destroy tests if kernel lacks supportFlorian Westphal2023-09-186-0/+13
| | | | | | | | Destroy support was added for table/flowtable/chain etc. in a single commit, so no need to add capability tests for each destroy subtype. Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip inet ingress tests if kernel lacks supportFlorian Westphal2023-09-184-7/+18
| | | | | | | Split the bridge autoremove test to a new file. Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip some tests if kernel lacks netdev egress supportFlorian Westphal2023-09-183-1/+18
| | | | | Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip bitshift tests if kernel lacks supportFlorian Westphal2023-09-1811-0/+27
| | | | | Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip inner matching tests if unsupportedFlorian Westphal2023-09-182-0/+9
| | | | | Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip map query if kernel lacks supportFlorian Westphal2023-09-182-8/+38
| | | | | | | | | | | | | | On recent kernels one can perform a lookup in a map without a destination register (i.e., treat the map like a set -- pure existence check). Add a feature probe and work around the missing feature in typeof_maps_add_delete: do the test with a simplified ruleset, Indicate skipped even though a reduced test was run (earlier errors cause a failure) to not trigger dump validation error. Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: skip netdev_chain_0 if kernel requires netdev deviceFlorian Westphal2023-09-182-0/+9
| | | | | | | | | | | | | | This test case only works on kernel 6.4+. Add feature probe for this and tag the test accordingly using the scheme added by Thomas Haller in "tests/shell: skip tests if nft does not support JSON mode" so that run-test.sh skips it if kernel requires a device. Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: add and use chain binding feature probeFlorian Westphal2023-09-184-7/+62
| | | | | | | | | | Alter 30s-stress to suppress anon chains when its unuspported. Note that 30s-stress is optionally be run standalone, so also update the test script. Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: cleanup creating dummy interfaces in testsThomas Haller2023-09-186-31/+32
| | | | | | | | | | | | | | | | | | | | | | | In "tests/shell/testcases/chains/netdev_chain_0", calling "trap ... EXIT" multiple times does not work. Fix it, by calling one cleanup function. Note that we run in separate namespaces, so the cleanup is usually not necessary. Still do it, we might want to run without unshare (via NFT_TEST_UNSHARE_CMD=""). Without unshare, it's important that the cleanup always works. In practice it might not, for example, "trap ... EXIT" does not run for SIGTERM. A leaked interface might break the follow up test and tests interfere with each other. Try to workaround that by first trying to delete the interface. Also failures to create the interfaces are not considered fatal. I don't understand under what circumstances this might fail, note that there are other tests that create dummy interface and don't "exit 77" on failure. We want to know when something odd is going on. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: suggest 4Mb /proc/sys/net/core/{wmem_max,rmem_max} for rootlessThomas Haller2023-09-181-3/+3
| | | | | | | | | | | | | | | | 2Mb was not enough to pass "tests/shell/testcases/sets/0030add_many_elements_interval_0" in an unprivileged/rootless namespace. Instead, bump the suggestion to 4Mb, which lets the test pass. Note that the 4Mb are only the recommended value when running the test as rootless, and is used to autodetect NFT_TEST_HAS_SOCKET_LIMITS=y. You can set whatever values are suitable for your environment, and explicitly indicate whether the limits are appropriate or not via NFT_TEST_HAS_SOCKET_LIMITS=n|y. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests: shell: add sample ruleset reproducerFlorian Westphal2023-09-182-0/+501
| | | | | | | | | | | | Changes on kernel side no longer permit transactions that reference a chain after it is bound. This test case breaks when run with nftables 1.0.6 and earlier. Keep this as a test case in tree to catch any future problems in this area. Link: https://lore.kernel.org/netfilter-devel/20230911213750.5B4B663206F5@dd20004.kasserver.com/ Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: colorize NFT_TEST_SKIP_/NFT_TEST_HAVE_ in test outputThomas Haller2023-09-161-2/+10
| | | | | | | | | Having a "SKIP" option as "y" or a "HAVE" option as "n", is note worthy because tests may be skipped based on that. Colorize, to make it easier to see in the test output. Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: add feature probing via "features/*.nft" filesThomas Haller2023-09-161-0/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Running selftests on older kernels makes some of them fail very early because some tests use features that are not available on older kernels, e.g. -stable releases. Known examples: - inner header matching - anonymous chains - elem delete from packet path Also, some test cases might fail because a feature isn't compiled in, such as netdev chains. This adds a feature-probing mechanism to shell tests. Simply drop a 'nft -f' compatible file with a .nft suffix into "tests/shell/features". "run-tests.sh" will load it via `nft --check` and will export NFT_TEST_HAVE_${feature}=y|n Here ${feature} is the basename of the .nft file without file extension. It must be all lower-case. This extends the existing NFT_TEST_HAVE_json= feature detection. Similarly, NFT_TEST_REQUIRES(NFT_TEST_HAVE_*) tags work to easily skip a test. The test script that cannot fully work without the feature should either skip the test entirely (NFT_TEST_REQUIRES(NFT_TEST_HAVE_*)), or run a reduced/modified test. If a modified test was run and passes, it is still a good idea to mark the overall result as skipped (exit 77) instead of claiming success to the modified test. We want to know when not the full test was running, while we want to test as much as we can. This patch is based on Florian's feature probing patch. Originally-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests: shell: fix dump validation messageFlorian Westphal2023-09-162-13/+0
| | | | | | | | | | | | This test output depends on CONFIG_HZ: - update @y { ip saddr timeout 1d2h3m4s8ms } + update @y { ip saddr timeout 1d2h3m4s10ms } The dump record is with HZ=1000, on HZ=250 we get failure. Remove the dump file for now. Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/build: capture more output from "tests/build/run-tests.sh" scriptThomas Haller2023-09-151-6/+6
| | | | | | | | | Dropping stdout for various build tests makes it hard to understand what happens, when a build fails. Redirect both stdout and stderr to the log files for easier debugging. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: honor CLICOLOR_FORCE to force coloring in run-tests.shThomas Haller2023-09-151-5/+10
| | | | | | | | | | | | | We honor NO_COLOR= to disable coloring, let's also honor CLICOLOR_FORCE= to enable it. The purpose will be for `make` calling the script and redirecting to a file, while enabling colors. See-also: https://bixense.com/clicolors/ Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: accept $NFT_TEST_TMPDIR_TAG for the result directoryThomas Haller2023-09-151-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We allow the user to set "$TMPDIR" to affect where the "nft-test.*" directory is created. However, we don't allow the user to specify the exact location, so the user doesn't really know which directory was created. One remedy is that the test will also create the symlink "$TMPDIR/nft-test.latest.$USER" to point to the last test result. However, if you run multiple tests in parallel, that is not reliable to find the test results. Accept $NFT_TEST_TMPDIR_TAG and use it as part of the generated filename. That way, the caller can set it to a unique tag, and find the directory later based on that. For example export TMPDIR=/tmp export NFT_TEST_TMPDIR_TAG=".$(uuidgen)" ./tests/shell/run-tests.sh ls -lad "$TMPDIR/nft-test."*"$NFT_TEST_TMPDIR_TAG"*/ will work reliably -- as long as the tag is chosen uniquely. The reason to not allow the user to specify the directory name directly, is because we want that tests results follow the well-known pattern "/tmp/nft-test*". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: exit 77 from "run-tests.sh" if all tests were skippedThomas Haller2023-09-151-1/+7
| | | | | | | | | | | | | | If there are multiple tests and some of them pass and some are skipped, the overall result should be success (zero). Because likely the user just selected a bunch of tests (or all of them). So skipping some tests does not mean that the entire run is not a success. However, if all tests are skipped, then mark the overall result as skipped too. The more common case is if you only run one single test, then we want to know, that the test didn't run. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: in find_tests() use C locale for sorting tests namesThomas Haller2023-09-151-1/+1
| | | | | | | | It makes more sense, that the sort order does not depend on the user's locale. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: special handle base path starting with "./"Thomas Haller2023-09-151-1/+3
| | | | | | | | | | | | | | | | | | When we auto detect the tests with `tests/shell/run-tests.sh -L`, then commonly the NFT_TEST_BASEDIR starts with a redundant "./". That's a bit ugly. Instead, special handle that case and remove the prefix. The effect is that `tests/shell/run-tests.sh -L` shows tests/shell/testcases/bitwise/0040mark_binop_0 instead of ./tests/shell/testcases/bitwise/0040mark_binop_0 Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add missing nft/nodump files for testsThomas Haller2023-09-153-0/+29
| | | | | | | | | | Three tests didn't have a nft/nodump file, because previously I only generated files on Fedora kernel, where those tests are failing. Generate them on CentOS-Stream-9 with kernel 5.14.0-354.el9.x86_64. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: drop unstable dump for "transactions/0051map_0" testThomas Haller2023-09-152-7/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The file "tests/shell/testcases/transactions/dumps/0051map_0.nft" gets generated differently on Fedora 38 (6.4.14-200.fc38.x86_64) and CentOS-Stream-9 (5.14.0-354.el9.x86_64). It's not stable. diff --git c/tests/shell/testcases/transactions/dumps/0051map_0.nft w/tests/shell/testcases/transactions/dumps/0051map_0.nft index 59d69df70e61..fa7df9f93757 100644 --- c/tests/shell/testcases/transactions/dumps/0051map_0.nft +++ w/tests/shell/testcases/transactions/dumps/0051map_0.nft @@ -1,7 +1,11 @@ table ip x { + chain w { + } + chain m { } chain y { + ip saddr vmap { 1.1.1.1 : jump w, 2.2.2.2 : accept, 3.3.3.3 : goto m } } } Drop it. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: remove spurious .nft dump filesThomas Haller2023-09-152-16/+0
| | | | | | | | | | | These are left-over dumps ([1]), or dumps generated with the wrong name ([2]). Remove the files. [1] commit eb14363d44ce ('tests: shell: move chain priority and policy to chain folder') [2] commit b4775dec9f80 ('src: ingress inet support') Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add option to shuffle execution order of testsThomas Haller2023-09-151-0/+18
| | | | | | | | | | | | The user can set NFT_TEST_SHUFFLE_TESTS=y|n to have the tests shuffled randomly. The purpose of shuffling is to find tests that depend on each other, or would break when run in unexpected order. If unspecified, by default tests are shuffled if no tests are selected on the command line. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add "random-source.sh" helper for random-source for sort/shufThomas Haller2023-09-152-1/+41
| | | | | | | | | | | | | | | | | Commands `sort` and `shuf` have a "--random-source" argument. That's useful for generating stable, reproducible "random" output. However, we want to do this based on a fixed seed, while the "--random-source" expects a stream of randomness. Add a helper script for that. Also, use the stable randomness for shuf in the test "tests/shell/testcases/sets/automerge_0". See-also: https://www.gnu.org/software/coreutils/manual/html_node/Random-sources.html#Random-sources Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: export NFT_TEST_RANDOM_SEED variable for testsThomas Haller2023-09-151-0/+46
| | | | | | | | | | | | | | | | | | | | | | | | Let "run-tests.sh" export a NFT_TEST_RANDOM_SEED variable, set to a decimal, random integer (in the range of 0 to 0x7FFFFFFF). The purpose is to provide a seed to tests for randomization. Randomizing tests is very useful to increase the coverage while not testing all combinations (which might not be practical). The point of NFT_TEST_RANDOM_SEED is that the user can set the environment variable so that the same series of random events is used. That is useful for reproducing an issue, that is known to happen with a certain seed. - by default, if the user leaves NFT_TEST_RANDOM_SEED unset or empty, the script generates a number using $SRANDOM. - if the user sets NFT_TEST_RANDOM_SEED to an integer it is taken as is (modulo 0x80000000). - otherwise, calculate a number by hashing the value of $NFT_TEST_RANDOM_SEED. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: ensure vgdb-pipe files are deleted from "nft-valgrind-wrapper.sh"Thomas Haller2023-09-142-3/+11
| | | | | | | | | | | | | When the valgrind process gets killed, those files can be left over. They are located in the original $TMPDIR (usually /tmp). They should be cleaned up. I tried to cleanup the files from withing "nft-valgrind-wrapper.sh" itself via a `trap`, but it doesn't work. Instead, let "run-tests.sh" delete all files with a matching pattern. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: kill running child processes when aborting "run-tests.sh"Thomas Haller2023-09-141-5/+26
| | | | | | | | | | | | | | When aborting "run-tests.sh", child processes were left running. Kill them. It's surprisingly complicated to get this somewhat right. Do it by enabling monitor mode for each test call, so that they run in separate process groups and we can kill the entire group. Note that we cannot just `kill -- -$$`, because it's not clear who is in this process group. Also, we don't want to kill the `tee` process which handles our logging. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: typeof_integer/raw: prefer @nh for payload matchingFlorian Westphal2023-09-106-12/+12
| | | | | | | | | | @ih fails on kernels where payload expression doesn't support the 'inner' base offset. This test isn't about inner headers, so just use @nh which is universally available. Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: make delete_by_handle test work on older releasesFlorian Westphal2023-09-101-1/+1
| | | | | | | | | | This test fails on kernels that lack 05abe4456fa3 ("netfilter: nf_tables: allow to register flowtable with no devices") v5.8-rc1~165^2~27^2~1 Just add lo as dummy device. Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add "--quick" option to skip slow tests (via NFT_TEST_SKIP_slow=y)Thomas Haller2023-09-0910-12/+68
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It's important to run (a part) of the tests in a timely manner. Add an option to skip long running tests. Thereby, add a more general NFT_TEST_SKIP_* mechanism. This is related and inverse from "NFT_TEST_HAVE_json", where a test can require [ "$NFT_TEST_HAVE_json" != n ] to run, but is skipped when [ "$NFT_TEST_SKIP_slow" = y ]. Currently only NFT_TEST_SKIP_slow is supported. The user can set such environment variables (or use the -Q|--quick command line option). The configuration is printed in the test info. Tests should check for [ "$NFT_TEST_SKIP_slow" = y ] so that the variable has to be explicitly set to opt-out. For convenience, tests can also add a # NFT_TEST_SKIP(NFT_TEST_SKIP_slow) tag, which is evaluated by test-wrapper.sh. Or they can run a quick, reduced part of the test, but then should still indicate to be skipped. Mark 8 tests are as slow, that take longer than 5 seconds on my machine. With this, a parallel wall time for the non-slow tests is only 7 seconds (on my machine). The ultimate point is to integrate a call to "tests/shell/run-tests.sh" in a `make check` target. For development, you can then export NFT_TEST_SKIP_slow=y and have a fast `make check`. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: skip tests if nft does not support JSON modeThomas Haller2023-09-0911-9/+112
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We can build nft without JSON support, and some tests will fail without it. Instead, they should be skipped. Also note, that the test accepts any nft binary via the "NFT" environment variable. So it's not enough to make the skipping dependent on build configuration, but on the currently used $NFT variable. Let "run-test.sh" detect and export a "NFT_TEST_HAVE_json=y|n" variable. This is heavily inspired by Florian's feature probing patches. Tests that require JSON can check that variable, and skip. Note that they check in the form of [ "$NFT_TEST_HAVE_json" != n ], so the test is only skipped, if we explicitly detect lack of support. That is, don't check via [ "$NFT_TEST_HAVE_json" = y ]. Some of the tests still run parts of the tests that don't require JSON. Only towards the end of such partial run, mark the test as skipped. Some tests require JSON support throughout. For those, add a mechanism where tests can add a tag (in their first 10 lines): # NFT_TEST_REQUIRES(NFT_TEST_HAVE_json) This will be checked by "test-wrapper.sh", which will skip the test. The purpose of this is to make it low-effort to skip a test and to print the reason in the text output as Test skipped due to NFT_TEST_HAVE_json=n (test has "NFT_TEST_REQUIRES(NFT_TEST_HAVE_json)" tag) This is intentionally not shortened to NFT_TEST_REQUIRES(json), so that we can grep for NFT_TEST_HAVE_json to find all relevant places. Note that while NFT_TEST_HAVE_json is autodetected, the caller can also force it by setting the environment variable. This allows to see what would happen to such a test. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print number of completed tests to show progressThomas Haller2023-09-091-7/+13
| | | | | | | | | | | | | | | | | | Especially with VALGRIND=y, a full test run can take a long time. When looking at the output, it's interesting to get a feel how far along we are. Print the number of completed jobs vs. the number of total jobs, in the line showing the test result. It gives a nice progress status. Example: I: [OK] 1/373 ./tests/shell/testcases/bitwise/0040mark_binop_1 I: [OK] 2/373 ./tests/shell/testcases/bitwise/0040mark_binop_0 ... Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: set valgrind's "--vgdb-prefix=" to orignal TMPDIRThomas Haller2023-09-092-0/+6
| | | | | | | | | | | | | | | | | | | | | | | "test-wrapper.sh" sets TMPDIR="$NFT_TEST_TESTTMPDIR". That is useful, so that temporary files of the tests are placed inside the test result data. Sometimes tests miss to delete those files, which would result in piling up /tmp/tmp.XXXXXXXXXX files. By setting $TMPDIR, those files are clearly related to the test run that created them, and can be deleted together. However, valgrind likes to create files like "vgdb-pipe-from-vgdb-to-68-by-thom-on-???" inside $TMPDIR. These are pipes, so if you run `grep -R ^ /tmp/nft-test.latest` while the test is still running (to inspect the results), then the process hands reading from the pipe. Instead, tell valgrind to put those files in the original TMPDIR. For that purpose, export NFT_TEST_TMPDIR_ORIG from "run-tests.sh". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add ".nft" dump files for tests without dumps/ directoryThomas Haller2023-09-096-0/+0
| | | | | | | | | | | | | | DUMPGEN=y mode skips tests that don't have a corresponding "dumps/" directory. Add the "dumps/" directory for tests that lacked it, and generate ".nft" files by running `./tests/shell/run-tests.sh -g`. Yes, they are all empty. Not very exciting, but why not check for that too? Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add missing ".nodump" file for tests without dumpsThomas Haller2023-09-098-0/+0
| | | | | | | | | | | | These files are generated by running `./tests/shell/run-tests.sh -g`. Commit the .nodump files to git. The point is to explicitly make it known that no dump file should be there. This prevents `./tests/shell/run-tests.sh -g` from generating the files and proposing (over and over) to add them to git. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: generate and add ".nft" dump files for existing testsThomas Haller2023-09-09161-0/+2546
| | | | | | | | | | | | | | | | | | | | | | | | | | | Several tests didn't have a ".nft" dump file committed. Generate one and commit it to git. While not all tests have a stable ruleset to compare, many have. Commit the .nft files for the tests where the output appears to be stable. This was generated by running `./tests/shell/run-tests.sh -g` twice, and commit the files that were identical both times. Note that 7 tests on my machine fail, so those are skipped. Also skip the files tests/shell/testcases/maps/dumps/0004interval_map_create_once_0.nft tests/shell/testcases/nft-f/dumps/0011manydefines_0.nft tests/shell/testcases/sets/dumps/0011add_many_elements_0.nft tests/shell/testcases/sets/dumps/0030add_many_elements_interval_0.nft tests/shell/testcases/sets/dumps/0068interval_stack_overflow_0.nft Those files are larger than 100KB, and I don't think we want to blow up the git repository this way. Even if they are only text files and compress well. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: honor .nodump file for tests without nft dumpsThomas Haller2023-09-091-2/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | For some tests, the dump is not stable or useful to test. For example, if they have an "expires" timestamps. Those tests don't have a .nft file in the dumps directory, and don't have it checked. DUMPGEN=y generates a new dump file, if the "dumps/" directory exists. Omitting that directory is a way to prevent the generation of the file. However, many such tests share their directory with tests that do have dumps. When running tests with DUMPGEN=y, new files for old tests are generated. Those files are not meant to be compared or committed to git because it's known to not work. Whether a test has a dump file, is part of the test. The absence of the dump file should also be recorded and committed to git. Add a way to opt-out from such generating such dumps by having .nodump files instead of the .nft dump. Later we should add unit tests that checks that no test has both a .nft and a .nodump file in git, that the .nodump file is always empty, and that every .nft/.nodump file has a corresponding test committed to git. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: set NFT_TEST_JOBS based on $(nproc)Thomas Haller2023-09-081-3/+7
| | | | | | | | Choose 150% of $(nproc) for the default vlaue of NFT_TEST_JOBS (rounded up). The minimal value chosen by default is 2. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: record wall time of test run in result dataThomas Haller2023-09-081-0/+16
| | | | | | | | | | | | | | | | | | | | | When running tests, it's useful to see how long it took. Keep track if the timestamps in a "times" file. Try: ( \ for d in /tmp/nft-test.latest.*/test-*/ ; do \ printf '%10.2f %s\n' \ "$(sed '1!d' "$d/times")" \ "$(cat "$d/name")" ; \ done \ | sort -n \ | awk '{print $0; s+=$1} END{printf("%10.2f\n", s)}' ; \ printf '%10.2f wall time\n' "$(sed '1!d' /tmp/nft-test.latest.*/times)" \ ) Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: no longer enable verbose output when selecting a testThomas Haller2023-09-081-4/+1
| | | | | | | | | | | | | | | | | | | | Previously, when selecting a test on the command line, it would also enable verbose output (except if the "--" separator was used). This convenience feature seems not great because the output from the test badly clutters the "run-test.sh" output. Now that the test results are all on disk, you can search them after the run with great flexibility (grep). Additionally, in previous versions, command line argument parsing was more restrictive, requiring that "-v" always be placed first. Now, the order does not matter, so it's easy to edit the command prompt and append a "-v", if that is what you want. Or if you really like verbose output, then `export VERBOSE=y`. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print "kernel is tainted" separate from test resultThomas Haller2023-09-081-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Once the kernel is tainted, it stays until reboot. It would not be useful to fail the entire test run based on that (and we don't do that). But then, it seems odd to print this in the same style as the test results, because a [FAILED] of a test counts as an overall failure. Instead, print this warning in a different style. Previously: $ ./tests/shell/run-tests.sh -- /usr/bin/true ... W: [FAILED] kernel is tainted I: [OK] /usr/bin/true I: results: [OK] 1 [SKIPPED] 0 [FAILED] 0 [TOTAL] 1 Now: $ ./tests/shell/run-tests.sh -- /usr/bin/true ... W: kernel is tainted I: [OK] /usr/bin/true I: results: [OK] 1 [SKIPPED] 0 [FAILED] 0 [TOTAL] 1 Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: redirect output of test script to file tooThomas Haller2023-09-081-0/+2
| | | | | | | | | | | It's useful to keep around for later. Redirect to the temporary directory. Note that the file content may be colorized too. `less -R` helps with that. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: don't redirect error/warning messages to stderrThomas Haller2023-09-081-2/+2
| | | | | | | | | | | | | | Writing some messages to stderr and some to stdout is not helpful. Once they are written to separate streams, it's hard to be sure about their relative order. Use grep to filter messages. Also, next we will redirect the entire output also to a file. There the output is also not split in two files. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print the NFT setting with the VALGRIND=y wrapperThomas Haller2023-09-081-4/+5
| | | | | | | | | With this we see in the info output I: info: NFT=./tests/shell/helpers/nft-valgrind-wrapper.sh Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: fix handling failures with VALGRIND=yThomas Haller2023-09-083-6/+26
| | | | | | | | | | | | | | | | | | With VALGRIND=y, on memleaks the tests did not fail. Fix that by passing "--error-exitcode=122" to valgrind. But just returning 122 from $NFT command may not correctly fail the test. Instead, ensure to write a "rc-failed-valrind" file, which is picked up by "test-wrapper.sh" to properly handle the valgrind failure (and fail with error code 122 itself). Also, accept NFT_TEST_VALGRIND_OPTS variable to a pass additional arguments to valgrind. For example a "--suppressions" file. Also show the special error code [VALGRIND] in "run-test.sh". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>