summaryrefslogtreecommitdiffstats
path: root/tests/shell/run-tests.sh
Commit message (Collapse)AuthorAgeFilesLines
* tests/shell: run `nft --check` on persisted dump filesThomas Haller2023-09-191-1/+3
| | | | | | | | | | | | | | | | "nft --check" will trigger a rollback in kernel. The existing dump files might hit new code paths. Take the opportunity to call the command on the existing files. And alternative would be to write a separate tests, that iterates over all files. However, then we can only run all the commands sequentially (unless we do something smart). That might be slower than the opportunity to run the checks in parallel. More importantly, it would be nice if the check for the dump file is clearly tied to the file's test. So run it right after the test, from the test wrapper. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: colorize NFT_TEST_HAS_SOCKET_LIMITSThomas Haller2023-09-181-11/+18
| | | | | | | | | | | | | | NFT_TEST_HAS_SOCKET_LIMITS= is similar to NFT_TEST_HAVE_* variables and indicates a feature (or lack thereof), except that it's inverted. Maybe this should be consolidated, however, NFT_TEST_HAS_SOCKET_LIMITS= is detected in the root namespace, unlike the shell scripts from features. So it's unclear how to consolidate them best. Anyway. Still highlight a lack of the capability, as it can cause tests to be skipped and we should see that easily. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: don't show the exit status for failed testsThomas Haller2023-09-181-6/+3
| | | | | | | | | | | | | | | | | | | | Previously, for failed tests we would print the exit code W: [FAILED] 2/2 tests/shell/testcases/listing/0013objects_0: got 1 This doesn't seem very useful. For one, we have special exit codes like 0 (OK), 77 (SKIPPED), 124 (DUMP FAIL), 123 (TAINTED), 122 (VALGRIND). Any other exit code is just an arbitrary failure. We don't define any special codes, and printing them is not useful. Note that further exit codes (118 - 121) are reserved, and could be special purposed, when there is a use. You can find the real exit code from the test in the result data in the "rc-failed" file. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: set C locale in "run-tests.sh"Thomas Haller2023-09-181-2/+6
| | | | | | | | | | The tests should run always the same, regardless of the user's language settings. Set LANG=C and LC_ALL=C and unset LANGUAGE. If some part wants to test a different language, it would set it explicitly. They anyway wouldn't want to depend on something from the user's environment. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: implement NFT_TEST_HAVE_json feature detection as scriptThomas Haller2023-09-181-14/+4
| | | | | | | | No more need to special case the "run a script" approach for detecting the json feature. Use the new mechanism instead. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: skip reset tests if kernel lacks supportFlorian Westphal2023-09-181-4/+21
| | | | | | | | | | | | reset is implemented via flush + extra attribute, so older kernels perform a flush. This means .nft doesn't work, we need to check if the individual set contents/sets are still in place post-reset. Make this generic and permit use of feat.sh in addition to the simpler foo.nft feature files. Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: suggest 4Mb /proc/sys/net/core/{wmem_max,rmem_max} for rootlessThomas Haller2023-09-181-3/+3
| | | | | | | | | | | | | | | | 2Mb was not enough to pass "tests/shell/testcases/sets/0030add_many_elements_interval_0" in an unprivileged/rootless namespace. Instead, bump the suggestion to 4Mb, which lets the test pass. Note that the 4Mb are only the recommended value when running the test as rootless, and is used to autodetect NFT_TEST_HAS_SOCKET_LIMITS=y. You can set whatever values are suitable for your environment, and explicitly indicate whether the limits are appropriate or not via NFT_TEST_HAS_SOCKET_LIMITS=n|y. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: colorize NFT_TEST_SKIP_/NFT_TEST_HAVE_ in test outputThomas Haller2023-09-161-2/+10
| | | | | | | | | Having a "SKIP" option as "y" or a "HAVE" option as "n", is note worthy because tests may be skipped based on that. Colorize, to make it easier to see in the test output. Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: add feature probing via "features/*.nft" filesThomas Haller2023-09-161-0/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Running selftests on older kernels makes some of them fail very early because some tests use features that are not available on older kernels, e.g. -stable releases. Known examples: - inner header matching - anonymous chains - elem delete from packet path Also, some test cases might fail because a feature isn't compiled in, such as netdev chains. This adds a feature-probing mechanism to shell tests. Simply drop a 'nft -f' compatible file with a .nft suffix into "tests/shell/features". "run-tests.sh" will load it via `nft --check` and will export NFT_TEST_HAVE_${feature}=y|n Here ${feature} is the basename of the .nft file without file extension. It must be all lower-case. This extends the existing NFT_TEST_HAVE_json= feature detection. Similarly, NFT_TEST_REQUIRES(NFT_TEST_HAVE_*) tags work to easily skip a test. The test script that cannot fully work without the feature should either skip the test entirely (NFT_TEST_REQUIRES(NFT_TEST_HAVE_*)), or run a reduced/modified test. If a modified test was run and passes, it is still a good idea to mark the overall result as skipped (exit 77) instead of claiming success to the modified test. We want to know when not the full test was running, while we want to test as much as we can. This patch is based on Florian's feature probing patch. Originally-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: honor CLICOLOR_FORCE to force coloring in run-tests.shThomas Haller2023-09-151-5/+10
| | | | | | | | | | | | | We honor NO_COLOR= to disable coloring, let's also honor CLICOLOR_FORCE= to enable it. The purpose will be for `make` calling the script and redirecting to a file, while enabling colors. See-also: https://bixense.com/clicolors/ Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: accept $NFT_TEST_TMPDIR_TAG for the result directoryThomas Haller2023-09-151-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We allow the user to set "$TMPDIR" to affect where the "nft-test.*" directory is created. However, we don't allow the user to specify the exact location, so the user doesn't really know which directory was created. One remedy is that the test will also create the symlink "$TMPDIR/nft-test.latest.$USER" to point to the last test result. However, if you run multiple tests in parallel, that is not reliable to find the test results. Accept $NFT_TEST_TMPDIR_TAG and use it as part of the generated filename. That way, the caller can set it to a unique tag, and find the directory later based on that. For example export TMPDIR=/tmp export NFT_TEST_TMPDIR_TAG=".$(uuidgen)" ./tests/shell/run-tests.sh ls -lad "$TMPDIR/nft-test."*"$NFT_TEST_TMPDIR_TAG"*/ will work reliably -- as long as the tag is chosen uniquely. The reason to not allow the user to specify the directory name directly, is because we want that tests results follow the well-known pattern "/tmp/nft-test*". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: exit 77 from "run-tests.sh" if all tests were skippedThomas Haller2023-09-151-1/+7
| | | | | | | | | | | | | | If there are multiple tests and some of them pass and some are skipped, the overall result should be success (zero). Because likely the user just selected a bunch of tests (or all of them). So skipping some tests does not mean that the entire run is not a success. However, if all tests are skipped, then mark the overall result as skipped too. The more common case is if you only run one single test, then we want to know, that the test didn't run. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: in find_tests() use C locale for sorting tests namesThomas Haller2023-09-151-1/+1
| | | | | | | | It makes more sense, that the sort order does not depend on the user's locale. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: special handle base path starting with "./"Thomas Haller2023-09-151-1/+3
| | | | | | | | | | | | | | | | | | When we auto detect the tests with `tests/shell/run-tests.sh -L`, then commonly the NFT_TEST_BASEDIR starts with a redundant "./". That's a bit ugly. Instead, special handle that case and remove the prefix. The effect is that `tests/shell/run-tests.sh -L` shows tests/shell/testcases/bitwise/0040mark_binop_0 instead of ./tests/shell/testcases/bitwise/0040mark_binop_0 Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add option to shuffle execution order of testsThomas Haller2023-09-151-0/+18
| | | | | | | | | | | | The user can set NFT_TEST_SHUFFLE_TESTS=y|n to have the tests shuffled randomly. The purpose of shuffling is to find tests that depend on each other, or would break when run in unexpected order. If unspecified, by default tests are shuffled if no tests are selected on the command line. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: export NFT_TEST_RANDOM_SEED variable for testsThomas Haller2023-09-151-0/+46
| | | | | | | | | | | | | | | | | | | | | | | | Let "run-tests.sh" export a NFT_TEST_RANDOM_SEED variable, set to a decimal, random integer (in the range of 0 to 0x7FFFFFFF). The purpose is to provide a seed to tests for randomization. Randomizing tests is very useful to increase the coverage while not testing all combinations (which might not be practical). The point of NFT_TEST_RANDOM_SEED is that the user can set the environment variable so that the same series of random events is used. That is useful for reproducing an issue, that is known to happen with a certain seed. - by default, if the user leaves NFT_TEST_RANDOM_SEED unset or empty, the script generates a number using $SRANDOM. - if the user sets NFT_TEST_RANDOM_SEED to an integer it is taken as is (modulo 0x80000000). - otherwise, calculate a number by hashing the value of $NFT_TEST_RANDOM_SEED. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: ensure vgdb-pipe files are deleted from "nft-valgrind-wrapper.sh"Thomas Haller2023-09-141-1/+9
| | | | | | | | | | | | | When the valgrind process gets killed, those files can be left over. They are located in the original $TMPDIR (usually /tmp). They should be cleaned up. I tried to cleanup the files from withing "nft-valgrind-wrapper.sh" itself via a `trap`, but it doesn't work. Instead, let "run-tests.sh" delete all files with a matching pattern. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: kill running child processes when aborting "run-tests.sh"Thomas Haller2023-09-141-5/+26
| | | | | | | | | | | | | | When aborting "run-tests.sh", child processes were left running. Kill them. It's surprisingly complicated to get this somewhat right. Do it by enabling monitor mode for each test call, so that they run in separate process groups and we can kill the entire group. Note that we cannot just `kill -- -$$`, because it's not clear who is in this process group. Also, we don't want to kill the `tee` process which handles our logging. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add "--quick" option to skip slow tests (via NFT_TEST_SKIP_slow=y)Thomas Haller2023-09-091-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It's important to run (a part) of the tests in a timely manner. Add an option to skip long running tests. Thereby, add a more general NFT_TEST_SKIP_* mechanism. This is related and inverse from "NFT_TEST_HAVE_json", where a test can require [ "$NFT_TEST_HAVE_json" != n ] to run, but is skipped when [ "$NFT_TEST_SKIP_slow" = y ]. Currently only NFT_TEST_SKIP_slow is supported. The user can set such environment variables (or use the -Q|--quick command line option). The configuration is printed in the test info. Tests should check for [ "$NFT_TEST_SKIP_slow" = y ] so that the variable has to be explicitly set to opt-out. For convenience, tests can also add a # NFT_TEST_SKIP(NFT_TEST_SKIP_slow) tag, which is evaluated by test-wrapper.sh. Or they can run a quick, reduced part of the test, but then should still indicate to be skipped. Mark 8 tests are as slow, that take longer than 5 seconds on my machine. With this, a parallel wall time for the non-slow tests is only 7 seconds (on my machine). The ultimate point is to integrate a call to "tests/shell/run-tests.sh" in a `make check` target. For development, you can then export NFT_TEST_SKIP_slow=y and have a fast `make check`. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: skip tests if nft does not support JSON modeThomas Haller2023-09-091-2/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We can build nft without JSON support, and some tests will fail without it. Instead, they should be skipped. Also note, that the test accepts any nft binary via the "NFT" environment variable. So it's not enough to make the skipping dependent on build configuration, but on the currently used $NFT variable. Let "run-test.sh" detect and export a "NFT_TEST_HAVE_json=y|n" variable. This is heavily inspired by Florian's feature probing patches. Tests that require JSON can check that variable, and skip. Note that they check in the form of [ "$NFT_TEST_HAVE_json" != n ], so the test is only skipped, if we explicitly detect lack of support. That is, don't check via [ "$NFT_TEST_HAVE_json" = y ]. Some of the tests still run parts of the tests that don't require JSON. Only towards the end of such partial run, mark the test as skipped. Some tests require JSON support throughout. For those, add a mechanism where tests can add a tag (in their first 10 lines): # NFT_TEST_REQUIRES(NFT_TEST_HAVE_json) This will be checked by "test-wrapper.sh", which will skip the test. The purpose of this is to make it low-effort to skip a test and to print the reason in the text output as Test skipped due to NFT_TEST_HAVE_json=n (test has "NFT_TEST_REQUIRES(NFT_TEST_HAVE_json)" tag) This is intentionally not shortened to NFT_TEST_REQUIRES(json), so that we can grep for NFT_TEST_HAVE_json to find all relevant places. Note that while NFT_TEST_HAVE_json is autodetected, the caller can also force it by setting the environment variable. This allows to see what would happen to such a test. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print number of completed tests to show progressThomas Haller2023-09-091-7/+13
| | | | | | | | | | | | | | | | | | Especially with VALGRIND=y, a full test run can take a long time. When looking at the output, it's interesting to get a feel how far along we are. Print the number of completed jobs vs. the number of total jobs, in the line showing the test result. It gives a nice progress status. Example: I: [OK] 1/373 ./tests/shell/testcases/bitwise/0040mark_binop_1 I: [OK] 2/373 ./tests/shell/testcases/bitwise/0040mark_binop_0 ... Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: set valgrind's "--vgdb-prefix=" to orignal TMPDIRThomas Haller2023-09-091-0/+5
| | | | | | | | | | | | | | | | | | | | | | | "test-wrapper.sh" sets TMPDIR="$NFT_TEST_TESTTMPDIR". That is useful, so that temporary files of the tests are placed inside the test result data. Sometimes tests miss to delete those files, which would result in piling up /tmp/tmp.XXXXXXXXXX files. By setting $TMPDIR, those files are clearly related to the test run that created them, and can be deleted together. However, valgrind likes to create files like "vgdb-pipe-from-vgdb-to-68-by-thom-on-???" inside $TMPDIR. These are pipes, so if you run `grep -R ^ /tmp/nft-test.latest` while the test is still running (to inspect the results), then the process hands reading from the pipe. Instead, tell valgrind to put those files in the original TMPDIR. For that purpose, export NFT_TEST_TMPDIR_ORIG from "run-tests.sh". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: set NFT_TEST_JOBS based on $(nproc)Thomas Haller2023-09-081-3/+7
| | | | | | | | Choose 150% of $(nproc) for the default vlaue of NFT_TEST_JOBS (rounded up). The minimal value chosen by default is 2. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: record wall time of test run in result dataThomas Haller2023-09-081-0/+16
| | | | | | | | | | | | | | | | | | | | | When running tests, it's useful to see how long it took. Keep track if the timestamps in a "times" file. Try: ( \ for d in /tmp/nft-test.latest.*/test-*/ ; do \ printf '%10.2f %s\n' \ "$(sed '1!d' "$d/times")" \ "$(cat "$d/name")" ; \ done \ | sort -n \ | awk '{print $0; s+=$1} END{printf("%10.2f\n", s)}' ; \ printf '%10.2f wall time\n' "$(sed '1!d' /tmp/nft-test.latest.*/times)" \ ) Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: no longer enable verbose output when selecting a testThomas Haller2023-09-081-4/+1
| | | | | | | | | | | | | | | | | | | | Previously, when selecting a test on the command line, it would also enable verbose output (except if the "--" separator was used). This convenience feature seems not great because the output from the test badly clutters the "run-test.sh" output. Now that the test results are all on disk, you can search them after the run with great flexibility (grep). Additionally, in previous versions, command line argument parsing was more restrictive, requiring that "-v" always be placed first. Now, the order does not matter, so it's easy to edit the command prompt and append a "-v", if that is what you want. Or if you really like verbose output, then `export VERBOSE=y`. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print "kernel is tainted" separate from test resultThomas Haller2023-09-081-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Once the kernel is tainted, it stays until reboot. It would not be useful to fail the entire test run based on that (and we don't do that). But then, it seems odd to print this in the same style as the test results, because a [FAILED] of a test counts as an overall failure. Instead, print this warning in a different style. Previously: $ ./tests/shell/run-tests.sh -- /usr/bin/true ... W: [FAILED] kernel is tainted I: [OK] /usr/bin/true I: results: [OK] 1 [SKIPPED] 0 [FAILED] 0 [TOTAL] 1 Now: $ ./tests/shell/run-tests.sh -- /usr/bin/true ... W: kernel is tainted I: [OK] /usr/bin/true I: results: [OK] 1 [SKIPPED] 0 [FAILED] 0 [TOTAL] 1 Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: redirect output of test script to file tooThomas Haller2023-09-081-0/+2
| | | | | | | | | | | It's useful to keep around for later. Redirect to the temporary directory. Note that the file content may be colorized too. `less -R` helps with that. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: don't redirect error/warning messages to stderrThomas Haller2023-09-081-2/+2
| | | | | | | | | | | | | | Writing some messages to stderr and some to stdout is not helpful. Once they are written to separate streams, it's hard to be sure about their relative order. Use grep to filter messages. Also, next we will redirect the entire output also to a file. There the output is also not split in two files. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print the NFT setting with the VALGRIND=y wrapperThomas Haller2023-09-081-4/+5
| | | | | | | | | With this we see in the info output I: info: NFT=./tests/shell/helpers/nft-valgrind-wrapper.sh Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: fix handling failures with VALGRIND=yThomas Haller2023-09-081-1/+3
| | | | | | | | | | | | | | | | | | With VALGRIND=y, on memleaks the tests did not fail. Fix that by passing "--error-exitcode=122" to valgrind. But just returning 122 from $NFT command may not correctly fail the test. Instead, ensure to write a "rc-failed-valrind" file, which is picked up by "test-wrapper.sh" to properly handle the valgrind failure (and fail with error code 122 itself). Also, accept NFT_TEST_VALGRIND_OPTS variable to a pass additional arguments to valgrind. For example a "--suppressions" file. Also show the special error code [VALGRIND] in "run-test.sh". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: colorize terminal output with test resultThomas Haller2023-09-081-10/+61
| | | | | | | | | | Colors help to see what is important. It honors the common NO_COLOR=<anything> to disable coloring. It also does not colorize, if [ -t 1 ] indicates that stdout is not a terminal. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: cleanup print_test_result() and show TAINTED error codeThomas Haller2023-09-081-8/+10
| | | | | | | | | | | We will add more special error codes (122 for VALGRIND). Minor refactor of print_test_result() to make it easier to extend for that. Also, we will soon colorize the output. This preparation patch makes that easier too. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: skip test in rootless that hit socket buffer size limitThomas Haller2023-09-071-0/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The socket buffer limits like /proc/sys/net/core/{rmem_max,wmem_max} can cause tests to fail, when running rootless. That's because real-root can override those limits, rootless cannot. Add an environment variable NFT_TEST_HAS_SOCKET_LIMITS=*|n which is automatically set by "run-tests.sh". Certain tests will check for [ "$NFT_TEST_HAS_SOCKET_LIMITS" = y ] and skip the test. The user may manually bump those limits (requires root), and set NFT_TEST_HAS_SOCKET_LIMITS=n to get the tests to pass even as rootless. For example, the test passes with root: sudo ./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0 Without root, it would fail. Skip it instead: ./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0 ... I: [SKIPPED] tests/shell/testcases/sets/automerge_0 Or bump the limit: $ echo 3000000 | sudo tee /proc/sys/net/core/wmem_max $ NFT_TEST_HAS_SOCKET_LIMITS=n ./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0 ... I: [OK] tests/shell/testcases/sets/automerge_0 Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: bind mount private /var/run/netns in test containerThomas Haller2023-09-071-4/+28
| | | | | | | | | | | | | | | | | | | | | | Some tests want to run `ip netns add`, which requires write permissions to /var/run/netns. Also, /var/run/netns would be a systemwide mount path, and shared between the tests. We would want to isolate that. Fix that by bind mount a tmpfs inside the test wrapper, if we appear to have a private mount namespace. Fixes $ ./tests/shell/run-tests.sh -- tests/shell/testcases/netns/0001nft-f_0 Optimally, `ip netns add` would allow to specify a private location for those bind mounts. It seems that iproute2 is build with /var/run/netns, instead the more common /run/netns. Hence, handle /var/run instead of /run. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: support running tests in parallelThomas Haller2023-09-071-13/+73
| | | | | | | | | | | | | | | Add option to enable running jobs in parallel. The purpose is to speed up the run time of the tests. The global cleanup (removal of kernel modules) interferes with parallel jobs (or even with, unrelated jobs on the system). By setting NFT_TEST_JOBS= to a positive number, that cleanup is skipped. This option is too good to miss. Hence parallel execution is enabled by default, and you have to opt-out from it. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: move valgrind wrapper script to separate scriptThomas Haller2023-09-071-34/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, in valgrind mode we would generate one script, which had "$NFT" variable and the temp directory hard coded. Soon, we will run jobs in parallel, so they would need at least different temp directories. Also, we want to put the valgrind results are inside "$NFT_TEST_TESTTMPDIR", along the test data. Extract the wrapper script to a separate script. It does not need to be generated ad-hoc, instead it uses the environment variables "$NFT_REAL" and "$NFT_TEST_TESTTMPDIR", as "run-tests.sh" prepares them. Also, add a "$NFT_REAL" variable for the actual NFT binary. We wrap the "$NFT" variable with VALGRIND=y or the user may pass "NFT='valgrind nft'". We should have access to the real binary. That might be useful for example to call `ldd "$NFT_REAL" | grep libjansson` to check for JSON support. Also, we use libtool. So quite possible the nft binary is actually a shell script. Calling valgrind on that script results in a lot of leak reports from shell (and slows down the command). Instead, use `libtool --mode=execute`. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: move taint check to "test-wrapper.sh"Thomas Haller2023-09-071-12/+4
| | | | | | | | | We will run tests in parallel. That means, we have multiple tests data and results in fly. That becomes simpler, if we move more result data to the test-wrapper and out of "run-tests.sh". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: rework printing of test resultsThomas Haller2023-09-071-39/+80
| | | | | | | | | | | | | | | | | | | | - "test-wrapper.sh" no longer will print the test output to its stdout. Instead, it only writes the testout.log file. - rework the loop "run-tests.sh" for printing the test results. It no longer captures the output of the test, as the wrapper is expected to be silent. Instead, they get the output from the result directory. The benefit is, that there is no duplication in what we print and the captured data in the result directory. The verbose mode is only for convenience, to safe looking at the test data. It's not essential otherwise. - also move the evaluation of the test result (and printing of the information) to a separate function. Later we want to run tests in parallel, so the steps need to be clearly separated. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: move the dump diff handling inside "test-wrapper.sh"Thomas Haller2023-09-071-28/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | This fits there better. At this point, we are still inside the unshared namespace and right after the test. The test-wrapper.sh should compare (and generate) the dumps. Also change behavior for DUMPGEN=y. - Previously it would only rewrite the dump if the dumpfile didn't exist yet. Now instead, always rewrite the file with DUMPGEN=y. The mode of operation is anyway, that the developer afterwards checks `git diff|status` to pick up the changes. There should be no changes to existing files (as existing tests are supposed to pass). So a diff there either means something went wrong (and we should see it) or it just means the dumps correctly should be regenerated. - also, only generate the file if the "dumps/" directory exists. This allows to write tests that don't have a dump file and don't get it automatically generated. The test wrapper will return a special error code 124 to indicate that the test passed, but the dumps file differed. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: support --keep-logs option (NFT_TEST_KEEP_LOGS=y) to preserve ↵Thomas Haller2023-09-071-3/+19
| | | | | | | | | | | test output The test output is now all collected in the temporary directory. On success, that directory is deleted. Add an option to always preserve that directory. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: interpret an exit code of 77 from scripts as "skipped"Thomas Haller2023-09-071-1/+10
| | | | | | | | | | | | Allow scripts to indicate that a test could not run by exiting 77. "77" is chosen as exit code from automake's testsuites ([1]). Compare to git-bisect which chooses 125 to indicate skipped. [1] https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: run each test in separate namespace and allow rootlessThomas Haller2023-09-071-17/+107
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Don't unshare the entire shell script. Instead, call unshare each test separately. That means, all tests use now a different sandbox and will also allow (with further changes) to run them in parallel. Also, allow to run rootless/unprivileged. The script first tries to run a separate PID+USER+NET namespace. If that fails, it downgrades to USER+NET. If that fails, it downgrades to a separate NET namespace. If unshare still fails, the script fails entirely. That differs from before, where the script would proceed without sandboxing. The script will now always require that unsharing works, unless the user opts-out. If the user cannot unshare, they can set NFT_TEST_UNSHARE_CMD to the command used for unsharing. It may be empty for no unshare. The command line arguments -U/--no-unshare are a shortcut for setting NFT_TEST_UNSHARE_CMD="". If we are able to create a separate USER namespace, then this mode allows to run the test as rootless/unprivileged. We no longer require [ `id -u` = 0 ]. Some tests may not work as rootless. For example, the socket buffers is limited by /proc/sys/net/core/{wmem_max,rmem_max} which real-root can override, but rootless tests cannot. Such tests should check for [ "$NFT_TEST_HAS_REALROOT" != y ] and skip gracefully. Usually, the user doesn't need to tell the script whether they have real-root. The script will autodetect it via [ `id -u` = 0 ]. But that won't work when run inside a rootless container already. In that case, the user would want to tell the script that there is no real-root. They can do so via the -R/--without-root option or NFT_TEST_HAS_REALROOT=n. If tests wish, the can know whether they run inside "unshare" environment by checking for [ "$NFT_TEST_HAS_UNSHARED" = y ]. When setting NFT_TEST_UNSHARE_CMD to override the unshare command, users may want to also set NFT_TEST_HAS_UNSHARED= and NFT_TEST_HAS_REALROOT= correctly. As we run each test in a separate unshare environment, we need a wrapper "tests/shell/helpers/test-wrapper.sh" around the test, which executes inside the tested environment. Also, each test gets its own temp directory prepared in NFT_TEST_TESTTMPDIR. This is also the place, where test artifacts and results will be collected. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print test configurationThomas Haller2023-09-071-5/+10
| | | | | | | | | As the script can be configured via environment variables or command line option, it's useful to show the environment variables that we received or set during the test setup. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: normalize boolean configuration in environment variablesThomas Haller2023-09-071-4/+15
| | | | | | | | | | | | Previously, we would honor "y" as opt-in, and all other values meant false. - accept alternatives to "y", like "1" or "true". - normalize the value, to either be "y" or "n". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: export NFT_TEST_BASEDIR and NFT_TEST_TMPDIR for testsThomas Haller2023-09-071-24/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | Let the test wrapper prepare and export two environment variables for the test: - "$NFT_TEST_BASEDIR" is just the top directory where the test scripts lie. - "$NFT_TEST_TMPDIR" is a `mktemp` directory created by "run-tests.sh" and removed at the end. Tests may use that to leave data there. This directory will be used for various things, like the "nft" wrapper in valgrind mode, the results of the tests and possibly as cache for feature detection. The "$NFT_TEST_TMPDIR" was already used before with the "VALGRIND=y" mode. It's only renamed and got an extended purpose. Also drop the unnecessary first detection of "$DIFF" and the "$SRC_NFT" variable. Also, note that the mktemp creates the temporary directory under /tmp. Which is commonly a tempfs. The user can override that by exporting TMPDIR. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: check test names before start and support directoriesThomas Haller2023-09-071-0/+12
| | | | | | | | | | | Check for valid test names early. That's useful because we treat any unrecognized options as test names. We should detect a mistake early. While at it, also support specifying directory names instead of executable files. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: rework finding tests and add "--list-tests" optionThomas Haller2023-09-071-28/+30
| | | | | | | | | | | | | | Cleanup finding the test files. Also add a "--list-tests" option to see which tests are found and would run. Also get rid of the FIND="$(which find)" detection. Which system doesn't have a working find? Also, we can just fail when we try to use find, and don't need a check first. This is still after "unshare", which will be addressed next. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: rework command line parsing in "run-tests.sh"Thomas Haller2023-09-071-30/+68
| | | | | | | | | | | | Parse the arguments in a loop, so that their order does not matter. Also, soon more command line arguments will be added, and this way of parsing seems more maintainable and flexible. Currently this is still after the is-root check and after unshare. That will be addressed later. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests: shell: auto-run kmemleak if its availableFlorian Westphal2023-07-201-1/+55
| | | | | | | | | | | | | | On my test vm a full scan takes almost 5s. As this would slowdown the test runs too much, only run them every couple of tests. This allows to detect when there is a leak reported at the end of the script, and it allows to narrow down the test case/group that triggers the issue. Add new -K flag to force kmemleak runs after each test if its available, this can then be used to find the exact test case. Signed-off-by: Florian Westphal <fw@strlen.de>
* tests: shell: Introduce valgrind modePhil Sutter2023-07-041-0/+47
| | | | | | | | Pass flag '-V' to run-tests.sh to run all 'nft' invocations in valgrind leak checking environment. Code copied from iptables' shell-testsuite where it proved to be useful already. Signed-off-by: Phil Sutter <phil@nwl.cc>