summaryrefslogtreecommitdiffstats
path: root/tests/shell
Commit message (Collapse)AuthorAgeFilesLines
...
* tests/shell: ensure vgdb-pipe files are deleted from "nft-valgrind-wrapper.sh"Thomas Haller2023-09-142-3/+11
| | | | | | | | | | | | | When the valgrind process gets killed, those files can be left over. They are located in the original $TMPDIR (usually /tmp). They should be cleaned up. I tried to cleanup the files from withing "nft-valgrind-wrapper.sh" itself via a `trap`, but it doesn't work. Instead, let "run-tests.sh" delete all files with a matching pattern. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: kill running child processes when aborting "run-tests.sh"Thomas Haller2023-09-141-5/+26
| | | | | | | | | | | | | | When aborting "run-tests.sh", child processes were left running. Kill them. It's surprisingly complicated to get this somewhat right. Do it by enabling monitor mode for each test call, so that they run in separate process groups and we can kill the entire group. Note that we cannot just `kill -- -$$`, because it's not clear who is in this process group. Also, we don't want to kill the `tee` process which handles our logging. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: typeof_integer/raw: prefer @nh for payload matchingFlorian Westphal2023-09-106-12/+12
| | | | | | | | | | @ih fails on kernels where payload expression doesn't support the 'inner' base offset. This test isn't about inner headers, so just use @nh which is universally available. Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: make delete_by_handle test work on older releasesFlorian Westphal2023-09-101-1/+1
| | | | | | | | | | This test fails on kernels that lack 05abe4456fa3 ("netfilter: nf_tables: allow to register flowtable with no devices") v5.8-rc1~165^2~27^2~1 Just add lo as dummy device. Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add "--quick" option to skip slow tests (via NFT_TEST_SKIP_slow=y)Thomas Haller2023-09-0910-12/+68
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It's important to run (a part) of the tests in a timely manner. Add an option to skip long running tests. Thereby, add a more general NFT_TEST_SKIP_* mechanism. This is related and inverse from "NFT_TEST_HAVE_json", where a test can require [ "$NFT_TEST_HAVE_json" != n ] to run, but is skipped when [ "$NFT_TEST_SKIP_slow" = y ]. Currently only NFT_TEST_SKIP_slow is supported. The user can set such environment variables (or use the -Q|--quick command line option). The configuration is printed in the test info. Tests should check for [ "$NFT_TEST_SKIP_slow" = y ] so that the variable has to be explicitly set to opt-out. For convenience, tests can also add a # NFT_TEST_SKIP(NFT_TEST_SKIP_slow) tag, which is evaluated by test-wrapper.sh. Or they can run a quick, reduced part of the test, but then should still indicate to be skipped. Mark 8 tests are as slow, that take longer than 5 seconds on my machine. With this, a parallel wall time for the non-slow tests is only 7 seconds (on my machine). The ultimate point is to integrate a call to "tests/shell/run-tests.sh" in a `make check` target. For development, you can then export NFT_TEST_SKIP_slow=y and have a fast `make check`. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: skip tests if nft does not support JSON modeThomas Haller2023-09-0911-9/+112
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We can build nft without JSON support, and some tests will fail without it. Instead, they should be skipped. Also note, that the test accepts any nft binary via the "NFT" environment variable. So it's not enough to make the skipping dependent on build configuration, but on the currently used $NFT variable. Let "run-test.sh" detect and export a "NFT_TEST_HAVE_json=y|n" variable. This is heavily inspired by Florian's feature probing patches. Tests that require JSON can check that variable, and skip. Note that they check in the form of [ "$NFT_TEST_HAVE_json" != n ], so the test is only skipped, if we explicitly detect lack of support. That is, don't check via [ "$NFT_TEST_HAVE_json" = y ]. Some of the tests still run parts of the tests that don't require JSON. Only towards the end of such partial run, mark the test as skipped. Some tests require JSON support throughout. For those, add a mechanism where tests can add a tag (in their first 10 lines): # NFT_TEST_REQUIRES(NFT_TEST_HAVE_json) This will be checked by "test-wrapper.sh", which will skip the test. The purpose of this is to make it low-effort to skip a test and to print the reason in the text output as Test skipped due to NFT_TEST_HAVE_json=n (test has "NFT_TEST_REQUIRES(NFT_TEST_HAVE_json)" tag) This is intentionally not shortened to NFT_TEST_REQUIRES(json), so that we can grep for NFT_TEST_HAVE_json to find all relevant places. Note that while NFT_TEST_HAVE_json is autodetected, the caller can also force it by setting the environment variable. This allows to see what would happen to such a test. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print number of completed tests to show progressThomas Haller2023-09-091-7/+13
| | | | | | | | | | | | | | | | | | Especially with VALGRIND=y, a full test run can take a long time. When looking at the output, it's interesting to get a feel how far along we are. Print the number of completed jobs vs. the number of total jobs, in the line showing the test result. It gives a nice progress status. Example: I: [OK] 1/373 ./tests/shell/testcases/bitwise/0040mark_binop_1 I: [OK] 2/373 ./tests/shell/testcases/bitwise/0040mark_binop_0 ... Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: set valgrind's "--vgdb-prefix=" to orignal TMPDIRThomas Haller2023-09-092-0/+6
| | | | | | | | | | | | | | | | | | | | | | | "test-wrapper.sh" sets TMPDIR="$NFT_TEST_TESTTMPDIR". That is useful, so that temporary files of the tests are placed inside the test result data. Sometimes tests miss to delete those files, which would result in piling up /tmp/tmp.XXXXXXXXXX files. By setting $TMPDIR, those files are clearly related to the test run that created them, and can be deleted together. However, valgrind likes to create files like "vgdb-pipe-from-vgdb-to-68-by-thom-on-???" inside $TMPDIR. These are pipes, so if you run `grep -R ^ /tmp/nft-test.latest` while the test is still running (to inspect the results), then the process hands reading from the pipe. Instead, tell valgrind to put those files in the original TMPDIR. For that purpose, export NFT_TEST_TMPDIR_ORIG from "run-tests.sh". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add ".nft" dump files for tests without dumps/ directoryThomas Haller2023-09-096-0/+0
| | | | | | | | | | | | | | DUMPGEN=y mode skips tests that don't have a corresponding "dumps/" directory. Add the "dumps/" directory for tests that lacked it, and generate ".nft" files by running `./tests/shell/run-tests.sh -g`. Yes, they are all empty. Not very exciting, but why not check for that too? Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add missing ".nodump" file for tests without dumpsThomas Haller2023-09-098-0/+0
| | | | | | | | | | | | These files are generated by running `./tests/shell/run-tests.sh -g`. Commit the .nodump files to git. The point is to explicitly make it known that no dump file should be there. This prevents `./tests/shell/run-tests.sh -g` from generating the files and proposing (over and over) to add them to git. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: generate and add ".nft" dump files for existing testsThomas Haller2023-09-09161-0/+2546
| | | | | | | | | | | | | | | | | | | | | | | | | | | Several tests didn't have a ".nft" dump file committed. Generate one and commit it to git. While not all tests have a stable ruleset to compare, many have. Commit the .nft files for the tests where the output appears to be stable. This was generated by running `./tests/shell/run-tests.sh -g` twice, and commit the files that were identical both times. Note that 7 tests on my machine fail, so those are skipped. Also skip the files tests/shell/testcases/maps/dumps/0004interval_map_create_once_0.nft tests/shell/testcases/nft-f/dumps/0011manydefines_0.nft tests/shell/testcases/sets/dumps/0011add_many_elements_0.nft tests/shell/testcases/sets/dumps/0030add_many_elements_interval_0.nft tests/shell/testcases/sets/dumps/0068interval_stack_overflow_0.nft Those files are larger than 100KB, and I don't think we want to blow up the git repository this way. Even if they are only text files and compress well. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: honor .nodump file for tests without nft dumpsThomas Haller2023-09-091-2/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | For some tests, the dump is not stable or useful to test. For example, if they have an "expires" timestamps. Those tests don't have a .nft file in the dumps directory, and don't have it checked. DUMPGEN=y generates a new dump file, if the "dumps/" directory exists. Omitting that directory is a way to prevent the generation of the file. However, many such tests share their directory with tests that do have dumps. When running tests with DUMPGEN=y, new files for old tests are generated. Those files are not meant to be compared or committed to git because it's known to not work. Whether a test has a dump file, is part of the test. The absence of the dump file should also be recorded and committed to git. Add a way to opt-out from such generating such dumps by having .nodump files instead of the .nft dump. Later we should add unit tests that checks that no test has both a .nft and a .nodump file in git, that the .nodump file is always empty, and that every .nft/.nodump file has a corresponding test committed to git. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: set NFT_TEST_JOBS based on $(nproc)Thomas Haller2023-09-081-3/+7
| | | | | | | | Choose 150% of $(nproc) for the default vlaue of NFT_TEST_JOBS (rounded up). The minimal value chosen by default is 2. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: record wall time of test run in result dataThomas Haller2023-09-081-0/+16
| | | | | | | | | | | | | | | | | | | | | When running tests, it's useful to see how long it took. Keep track if the timestamps in a "times" file. Try: ( \ for d in /tmp/nft-test.latest.*/test-*/ ; do \ printf '%10.2f %s\n' \ "$(sed '1!d' "$d/times")" \ "$(cat "$d/name")" ; \ done \ | sort -n \ | awk '{print $0; s+=$1} END{printf("%10.2f\n", s)}' ; \ printf '%10.2f wall time\n' "$(sed '1!d' /tmp/nft-test.latest.*/times)" \ ) Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: no longer enable verbose output when selecting a testThomas Haller2023-09-081-4/+1
| | | | | | | | | | | | | | | | | | | | Previously, when selecting a test on the command line, it would also enable verbose output (except if the "--" separator was used). This convenience feature seems not great because the output from the test badly clutters the "run-test.sh" output. Now that the test results are all on disk, you can search them after the run with great flexibility (grep). Additionally, in previous versions, command line argument parsing was more restrictive, requiring that "-v" always be placed first. Now, the order does not matter, so it's easy to edit the command prompt and append a "-v", if that is what you want. Or if you really like verbose output, then `export VERBOSE=y`. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print "kernel is tainted" separate from test resultThomas Haller2023-09-081-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Once the kernel is tainted, it stays until reboot. It would not be useful to fail the entire test run based on that (and we don't do that). But then, it seems odd to print this in the same style as the test results, because a [FAILED] of a test counts as an overall failure. Instead, print this warning in a different style. Previously: $ ./tests/shell/run-tests.sh -- /usr/bin/true ... W: [FAILED] kernel is tainted I: [OK] /usr/bin/true I: results: [OK] 1 [SKIPPED] 0 [FAILED] 0 [TOTAL] 1 Now: $ ./tests/shell/run-tests.sh -- /usr/bin/true ... W: kernel is tainted I: [OK] /usr/bin/true I: results: [OK] 1 [SKIPPED] 0 [FAILED] 0 [TOTAL] 1 Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: redirect output of test script to file tooThomas Haller2023-09-081-0/+2
| | | | | | | | | | | It's useful to keep around for later. Redirect to the temporary directory. Note that the file content may be colorized too. `less -R` helps with that. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: don't redirect error/warning messages to stderrThomas Haller2023-09-081-2/+2
| | | | | | | | | | | | | | Writing some messages to stderr and some to stdout is not helpful. Once they are written to separate streams, it's hard to be sure about their relative order. Use grep to filter messages. Also, next we will redirect the entire output also to a file. There the output is also not split in two files. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print the NFT setting with the VALGRIND=y wrapperThomas Haller2023-09-081-4/+5
| | | | | | | | | With this we see in the info output I: info: NFT=./tests/shell/helpers/nft-valgrind-wrapper.sh Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: fix handling failures with VALGRIND=yThomas Haller2023-09-083-6/+26
| | | | | | | | | | | | | | | | | | With VALGRIND=y, on memleaks the tests did not fail. Fix that by passing "--error-exitcode=122" to valgrind. But just returning 122 from $NFT command may not correctly fail the test. Instead, ensure to write a "rc-failed-valrind" file, which is picked up by "test-wrapper.sh" to properly handle the valgrind failure (and fail with error code 122 itself). Also, accept NFT_TEST_VALGRIND_OPTS variable to a pass additional arguments to valgrind. For example a "--suppressions" file. Also show the special error code [VALGRIND] in "run-test.sh". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: colorize terminal output with test resultThomas Haller2023-09-081-10/+61
| | | | | | | | | | Colors help to see what is important. It honors the common NO_COLOR=<anything> to disable coloring. It also does not colorize, if [ -t 1 ] indicates that stdout is not a terminal. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: cleanup print_test_result() and show TAINTED error codeThomas Haller2023-09-081-8/+10
| | | | | | | | | | | We will add more special error codes (122 for VALGRIND). Minor refactor of print_test_result() to make it easier to extend for that. Also, we will soon colorize the output. This preparation patch makes that easier too. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: cleanup result handling in "test-wrapper.sh"Thomas Haller2023-09-081-24/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | The previous code was mostly correct, but hard to understand. Rework it. Also, on failure now always write "rc-failed-exit", which is the exit code that "test-wrapper.sh" reports to "run-test.sh". Note that this error code may not be the same as the one returned by the TEST binary. The latter you can find in one of the files "rc-{ok,skipped,failed}". In general, you can search the directory with test results for those "rc-*" files. If you find a "rc-failed" file, it was counted as failure. There might be other "rc-failed-*" files, depending on whether the diff failed or kernel got tainted. Also, reserve all the error codes 118 - 124 for the "test-wrapper.sh". For example, 124 means a dump difference and 123 means kernel got tainted. In the future, 122 will mean a valgrind error. Other numbers are not reserved. If a test command fails with such an reserved code, "test-wrapper.sh" modifies it to 125, so that "run-test.sh" does not get the wrong idea about the failure reason. This is not new in this patch, except that the reserved range was extended for future additions. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: return 77/skip for tests that fail to create dummy deviceThomas Haller2023-09-083-5/+5
| | | | | | | | | | | | | There are some existing tests, that skip operation when they fail to create a dummy interface. Use the new exit code 77 to indicate "SKIPPED". I wonder why creating a dummy device would ever fail and why we don't just fail the test altogether in that case. But the patch does not change that. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: set TMPDIR for tests in "test-wrapper.sh"Thomas Haller2023-09-071-0/+2
| | | | | | | | | | | | | | Various tests create additional temporary files. They really should just use "$NFT_TEST_TESTTMPDIR" for that. However, they mostly use `mktemp`. The scripts are supposed to cleanup those files afterwards. However, often that does not work correctly and /tmp gets full of left over temporary files. Export "TMPDIR" so that they use the test-specific temporary directory. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: fix "0003includepath_0" for different TMPDIRThomas Haller2023-09-071-2/+2
| | | | | | | | We are going to set $TMPDIR to another location. The previous code made assumptions that the generated path would always be in /tmp. Fix that. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: record the test duration (wall time) in the result dataThomas Haller2023-09-071-0/+6
| | | | | | | | | | | | | | | | | Runtimes are important. Add a way to find out how long tests took. $ ./tests/shell/run-tests.sh ... $ for d in /tmp/nft-test.latest.*/test-*/ ; do \ printf '%10.2f %s\n' \ "$(sed '1!d' "$d/times")" \ "$(cat "$d/name")" ; \ done \ | sort -n \ | awk '{print $0; s+=$1} END{printf("%10.2f\n", s)}' Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: skip test in rootless that hit socket buffer size limitThomas Haller2023-09-0710-5/+167
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The socket buffer limits like /proc/sys/net/core/{rmem_max,wmem_max} can cause tests to fail, when running rootless. That's because real-root can override those limits, rootless cannot. Add an environment variable NFT_TEST_HAS_SOCKET_LIMITS=*|n which is automatically set by "run-tests.sh". Certain tests will check for [ "$NFT_TEST_HAS_SOCKET_LIMITS" = y ] and skip the test. The user may manually bump those limits (requires root), and set NFT_TEST_HAS_SOCKET_LIMITS=n to get the tests to pass even as rootless. For example, the test passes with root: sudo ./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0 Without root, it would fail. Skip it instead: ./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0 ... I: [SKIPPED] tests/shell/testcases/sets/automerge_0 Or bump the limit: $ echo 3000000 | sudo tee /proc/sys/net/core/wmem_max $ NFT_TEST_HAS_SOCKET_LIMITS=n ./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0 ... I: [OK] tests/shell/testcases/sets/automerge_0 Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: bind mount private /var/run/netns in test containerThomas Haller2023-09-072-4/+51
| | | | | | | | | | | | | | | | | | | | | | Some tests want to run `ip netns add`, which requires write permissions to /var/run/netns. Also, /var/run/netns would be a systemwide mount path, and shared between the tests. We would want to isolate that. Fix that by bind mount a tmpfs inside the test wrapper, if we appear to have a private mount namespace. Fixes $ ./tests/shell/run-tests.sh -- tests/shell/testcases/netns/0001nft-f_0 Optimally, `ip netns add` would allow to specify a private location for those bind mounts. It seems that iproute2 is build with /var/run/netns, instead the more common /run/netns. Hence, handle /var/run instead of /run. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: support running tests in parallelThomas Haller2023-09-071-13/+73
| | | | | | | | | | | | | | | Add option to enable running jobs in parallel. The purpose is to speed up the run time of the tests. The global cleanup (removal of kernel modules) interferes with parallel jobs (or even with, unrelated jobs on the system). By setting NFT_TEST_JOBS= to a positive number, that cleanup is skipped. This option is too good to miss. Hence parallel execution is enabled by default, and you have to opt-out from it. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: move valgrind wrapper script to separate scriptThomas Haller2023-09-072-34/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, in valgrind mode we would generate one script, which had "$NFT" variable and the temp directory hard coded. Soon, we will run jobs in parallel, so they would need at least different temp directories. Also, we want to put the valgrind results are inside "$NFT_TEST_TESTTMPDIR", along the test data. Extract the wrapper script to a separate script. It does not need to be generated ad-hoc, instead it uses the environment variables "$NFT_REAL" and "$NFT_TEST_TESTTMPDIR", as "run-tests.sh" prepares them. Also, add a "$NFT_REAL" variable for the actual NFT binary. We wrap the "$NFT" variable with VALGRIND=y or the user may pass "NFT='valgrind nft'". We should have access to the real binary. That might be useful for example to call `ldd "$NFT_REAL" | grep libjansson` to check for JSON support. Also, we use libtool. So quite possible the nft binary is actually a shell script. Calling valgrind on that script results in a lot of leak reports from shell (and slows down the command). Instead, use `libtool --mode=execute`. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: move taint check to "test-wrapper.sh"Thomas Haller2023-09-072-15/+19
| | | | | | | | | We will run tests in parallel. That means, we have multiple tests data and results in fly. That becomes simpler, if we move more result data to the test-wrapper and out of "run-tests.sh". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: rework printing of test resultsThomas Haller2023-09-072-40/+81
| | | | | | | | | | | | | | | | | | | | - "test-wrapper.sh" no longer will print the test output to its stdout. Instead, it only writes the testout.log file. - rework the loop "run-tests.sh" for printing the test results. It no longer captures the output of the test, as the wrapper is expected to be silent. Instead, they get the output from the result directory. The benefit is, that there is no duplication in what we print and the captured data in the result directory. The verbose mode is only for convenience, to safe looking at the test data. It's not essential otherwise. - also move the evaluation of the test result (and printing of the information) to a separate function. Later we want to run tests in parallel, so the steps need to be clearly separated. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: move the dump diff handling inside "test-wrapper.sh"Thomas Haller2023-09-072-35/+66
| | | | | | | | | | | | | | | | | | | | | | | | | | | This fits there better. At this point, we are still inside the unshared namespace and right after the test. The test-wrapper.sh should compare (and generate) the dumps. Also change behavior for DUMPGEN=y. - Previously it would only rewrite the dump if the dumpfile didn't exist yet. Now instead, always rewrite the file with DUMPGEN=y. The mode of operation is anyway, that the developer afterwards checks `git diff|status` to pick up the changes. There should be no changes to existing files (as existing tests are supposed to pass). So a diff there either means something went wrong (and we should see it) or it just means the dumps correctly should be regenerated. - also, only generate the file if the "dumps/" directory exists. This allows to write tests that don't have a dump file and don't get it automatically generated. The test wrapper will return a special error code 124 to indicate that the test passed, but the dumps file differed. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: support --keep-logs option (NFT_TEST_KEEP_LOGS=y) to preserve ↵Thomas Haller2023-09-071-3/+19
| | | | | | | | | | | test output The test output is now all collected in the temporary directory. On success, that directory is deleted. Add an option to always preserve that directory. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: interpret an exit code of 77 from scripts as "skipped"Thomas Haller2023-09-072-1/+12
| | | | | | | | | | | | Allow scripts to indicate that a test could not run by exiting 77. "77" is chosen as exit code from automake's testsuites ([1]). Compare to git-bisect which chooses 125 to indicate skipped. [1] https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: run each test in separate namespace and allow rootlessThomas Haller2023-09-072-17/+130
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Don't unshare the entire shell script. Instead, call unshare each test separately. That means, all tests use now a different sandbox and will also allow (with further changes) to run them in parallel. Also, allow to run rootless/unprivileged. The script first tries to run a separate PID+USER+NET namespace. If that fails, it downgrades to USER+NET. If that fails, it downgrades to a separate NET namespace. If unshare still fails, the script fails entirely. That differs from before, where the script would proceed without sandboxing. The script will now always require that unsharing works, unless the user opts-out. If the user cannot unshare, they can set NFT_TEST_UNSHARE_CMD to the command used for unsharing. It may be empty for no unshare. The command line arguments -U/--no-unshare are a shortcut for setting NFT_TEST_UNSHARE_CMD="". If we are able to create a separate USER namespace, then this mode allows to run the test as rootless/unprivileged. We no longer require [ `id -u` = 0 ]. Some tests may not work as rootless. For example, the socket buffers is limited by /proc/sys/net/core/{wmem_max,rmem_max} which real-root can override, but rootless tests cannot. Such tests should check for [ "$NFT_TEST_HAS_REALROOT" != y ] and skip gracefully. Usually, the user doesn't need to tell the script whether they have real-root. The script will autodetect it via [ `id -u` = 0 ]. But that won't work when run inside a rootless container already. In that case, the user would want to tell the script that there is no real-root. They can do so via the -R/--without-root option or NFT_TEST_HAS_REALROOT=n. If tests wish, the can know whether they run inside "unshare" environment by checking for [ "$NFT_TEST_HAS_UNSHARED" = y ]. When setting NFT_TEST_UNSHARE_CMD to override the unshare command, users may want to also set NFT_TEST_HAS_UNSHARED= and NFT_TEST_HAS_REALROOT= correctly. As we run each test in a separate unshare environment, we need a wrapper "tests/shell/helpers/test-wrapper.sh" around the test, which executes inside the tested environment. Also, each test gets its own temp directory prepared in NFT_TEST_TESTTMPDIR. This is also the place, where test artifacts and results will be collected. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print test configurationThomas Haller2023-09-071-5/+10
| | | | | | | | | As the script can be configured via environment variables or command line option, it's useful to show the environment variables that we received or set during the test setup. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: normalize boolean configuration in environment variablesThomas Haller2023-09-071-4/+15
| | | | | | | | | | | | Previously, we would honor "y" as opt-in, and all other values meant false. - accept alternatives to "y", like "1" or "true". - normalize the value, to either be "y" or "n". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: export NFT_TEST_BASEDIR and NFT_TEST_TMPDIR for testsThomas Haller2023-09-071-24/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | Let the test wrapper prepare and export two environment variables for the test: - "$NFT_TEST_BASEDIR" is just the top directory where the test scripts lie. - "$NFT_TEST_TMPDIR" is a `mktemp` directory created by "run-tests.sh" and removed at the end. Tests may use that to leave data there. This directory will be used for various things, like the "nft" wrapper in valgrind mode, the results of the tests and possibly as cache for feature detection. The "$NFT_TEST_TMPDIR" was already used before with the "VALGRIND=y" mode. It's only renamed and got an extended purpose. Also drop the unnecessary first detection of "$DIFF" and the "$SRC_NFT" variable. Also, note that the mktemp creates the temporary directory under /tmp. Which is commonly a tempfs. The user can override that by exporting TMPDIR. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: check test names before start and support directoriesThomas Haller2023-09-071-0/+12
| | | | | | | | | | | Check for valid test names early. That's useful because we treat any unrecognized options as test names. We should detect a mistake early. While at it, also support specifying directory names instead of executable files. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: rework finding tests and add "--list-tests" optionThomas Haller2023-09-071-28/+30
| | | | | | | | | | | | | | Cleanup finding the test files. Also add a "--list-tests" option to see which tests are found and would run. Also get rid of the FIND="$(which find)" detection. Which system doesn't have a working find? Also, we can just fail when we try to use find, and don't need a check first. This is still after "unshare", which will be addressed next. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: rework command line parsing in "run-tests.sh"Thomas Haller2023-09-071-30/+68
| | | | | | | | | | | | Parse the arguments in a loop, so that their order does not matter. Also, soon more command line arguments will be added, and this way of parsing seems more maintainable and flexible. Currently this is still after the is-root check and after unshare. That will be addressed later. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests: shell: 0043concatenated_ranges_0: re-enable all testsFlorian Westphal2023-09-061-6/+1
| | | | | | | | This script suppressed a few tests when ran via run-tests.sh, don't do that, it would have caught the previous 'get' bug years ago. Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: expand vmap test case to also cause batch abortFlorian Westphal2023-08-292-6/+22
| | | | | | | | | | Let the last few batches also push an update that contains elements twice. This is expected to cause the batch to be aborted, which increases code coverage on kernel side. Signed-off-by: Florian Westphal <fw@strlen.de>
* tests: 30s-stress: add failslab and abort phase testsFlorian Westphal2023-08-241-15/+375
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pablo suggested to also cover abort phase by intentionally deleting non-existent or adding clashing keys. While at it: add rules with anon sets and jumps to anonymous chains and a few constant sets. Pick different key sizes so there is a higher chance kernel picks different backend storages such as bitmap or hash_fast. add failslab support, this also covers unlikely or "impossible" cases like failing GFP_KERNEL allocations. randomly spawn 'nft monitor' in the background for a random duration to cover notification path. Try to randomly delete a set or chain from control plane. Randomly set a table as dormant (and back to normal). Allow to pass the test runtime as argument, so one can now do ./30s-stress 3600 to have the test run for one hour. For such long test durations, make sure the ruleset gets regenerated periodically. Signed-off-by: Florian Westphal <fw@strlen.de>
* tests: shell: use minutes granularity in sets/0036add_set_element_expiration_0Pablo Neira Ayuso2023-08-231-2/+6
| | | | | | | Use minute granularity to fix bogus failures of this test on slow testbed. Acked-by: Phil Sutter <phil@nwl.cc> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* tests: shell: Stabilize sets/0043concatenated_ranges_0 testPhil Sutter2023-08-231-2/+6
| | | | | | | | | | | On a slow system, one of the 'delete element' commands would occasionally fail. Assuming it can only happen if the 2s timeout passes "too quickly", work around it by adding elements with a 2m timeout instead and when wanting to test the element expiry just drop and add the element again with a short timeout. Fixes: 6231d3fa4af1e ("tests: shell: Fix for unstable sets/0043concatenated_ranges_0") Signed-off-by: Phil Sutter <phil@nwl.cc>
* tests: shell: Stabilize sets/reset_command_0 testPhil Sutter2023-08-231-39/+48
| | | | | | | | | | | | | | | | | | Timeout/expiry value testing based on seconds is way too fragile, especially with slow debug kernels. Rewrite the unit to test minute-based values. This means it is no longer feasible to wait for values to sufficiently change, so instead specify an 'expires' value when creating the ruleset and drop the 'sleep' call. While being at it: - Combine 'get element' and 'reset element' calls into one, assert the relevant (sanitized) line appears twice in output instead of comparing with 'diff'. - Turn comments into 'echo' calls to help debugging if the test fails. Reported-by: Florian Westphal <fw@strlen.de> Signed-off-by: Phil Sutter <phil@nwl.cc>
* tests: update bad_expression test caseFlorian Westphal2023-08-171-0/+9
| | | | | | | | Check that the ruleset also fails to validate if there is another table that passes validation checks. Reported-by: Pablo Neira Ayuso <pablo@netfilter.org> Signed-off-by: Florian Westphal <fw@strlen.de>