summaryrefslogtreecommitdiffstats
path: root/tests/shell/run-tests.sh
Commit message (Collapse)AuthorAgeFilesLines
* tests: shell: Avoid escape chars when printing to non-terminalsPhil Sutter8 days1-2/+2
| | | | | | | | Print the 'EXECUTING' status line only if stdout is a terminal, the mandatory following escape sequence to delete it messes up log file contents. Signed-off-by: Phil Sutter <phil@nwl.cc>
* tests: shell: flush ruleset with -U after feature probingPablo Neira Ayuso2023-12-061-0/+3
| | | | | | | feature probe script leaves a ruleset in place, flush it once probing is complete. Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* tests/shell: workaround lack of $SRANDOM before bash 5.1Thomas Haller2023-11-281-0/+1
| | | | | | | $SRANDOM is only supported since bash 5.1. Add a fallback to $RANDOM. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: workaround lack of `wait -p` before bash 5.1Thomas Haller2023-11-281-7/+40
| | | | | | | | | | | | Before bash 5.1, `wait -p` is not supported. So we cannot know which child process completed. As workaround, explicitly wait for the next PID. That works, but it significantly reduces parallel execution, because a long running job blocks the queue. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: check and generate JSON dump filesThomas Haller2023-11-151-4/+7
| | | | | | | | | | | | | | | | | | | | | | The rules after a successful test are good opportunity to test `nft -j list ruleset` and `nft -j --check`. This quite possibly touches code paths that are not hit by other tests yet. The only downside is the increase of the test runtime (which seems negligible, given the benefits of increasing test coverage). Future commits will generate and commit those ".json-nft" dump files. "DUMPGEN=y" will, like before, regenerate only the existing "*.{nodump,nft,json-nft}" files (unless a test has none of the 3 files, in which case they are all generated and the user is suggested to commit the correct ones). Now also "DUMPGEN=all" is honored, that will generate all 3 files, regardless of whether they already existed. That is useful if you start out with a test that only has a .nft file, and then you want to generate a .json-nft file too. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: honor NFT_TEST_VERBOSE_TEST variable to debug tests via `bash -x`Thomas Haller2023-10-211-1/+14
| | | | | | | | | | | | | | | | | | | | | | | | | It can be cumbersome to debug why a test fails. Our tests are just shell scripts, which for the most part don't print much. That is good, but for debugging, it can be useful to run the test via `bash -x`. Previously, we would just patch the source file while debugging. Add an option "-x" and NFT_TEST_VERBOSE_TEST=y environment variable. If set, "test-wrapper.sh" will check whether the shebang is "#!/bin/bash" and add "-x" to the command line. While at it, let test-wrapper.sh also log a line like Command: $CMD With this, we see in the log the command that was run, and how NFT_TEST_VERBOSE_TEST may have affected it. This is anyway useful, because many tests don't print anything at all, and we end up with an empty "testout.log". Empty files are cumbersome, e.g. I like to use `grep -R ^` to show the content of all files, which does not show empty files. Ensuring that something is always written is desirable. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add "-S|--setup-host" option to set sysctl for rootless testsThomas Haller2023-10-101-0/+46
| | | | | | | | | | | | | | | | | | | | | | Most tests can run just fine without root. A few of them will fail if /proc/sys/net/core/{wmem_max,rmem_max} is too small (as it is by default on the host). The easy workaround is to bump those limits once. This has to be repeated after each reboot. Doing that manually (every time) is cumbersome. Add a "--setup-host" option for that. Usage: $ sudo ./tests/shell/run-tests.sh -S Setting up host for running as rootless (requires root). echo 4096000 > /proc/sys/net/core/rmem_max (previous value 100000) echo 4096000 > /proc/sys/net/core/wmem_max (previous value 100000) Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* tests/shell: preserve result directory with NFT_TEST_FAIL_ON_SKIPThomas Haller2023-10-101-2/+7
| | | | | | | | | | | On a successful run, the result directory will be deleted (unless run with "-k|--keep-logs" option or NFT_TEST_KEEP_LOGS=y). With NFT_TEST_FAIL_ON_SKIP=y, when there are no failures but skipped tests, also preserve the result. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
* tests/shell: honor NFT_TEST_FAIL_ON_SKIP variable to fail on any skipped testsThomas Haller2023-09-211-1/+7
| | | | | | | | | | | | | | | | | | The test suite should pass with various kernels and build configurations. Of course, that means, that some tests will be gracefully skipped, and we don't treat that as an overall failure. However, it should be possible to run a specific kernel (net-next?) and build configuration, where we expect that all tests pass. Add an option to fail the run, if any tests were skipped. This is to ensure that we don't have broken tests that never pass. This will make more sense with automated CI is running, to enable on a test system and ensure that at least on that system, all tests pass. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: run `nft --check` on persisted dump filesThomas Haller2023-09-191-1/+3
| | | | | | | | | | | | | | | | "nft --check" will trigger a rollback in kernel. The existing dump files might hit new code paths. Take the opportunity to call the command on the existing files. And alternative would be to write a separate tests, that iterates over all files. However, then we can only run all the commands sequentially (unless we do something smart). That might be slower than the opportunity to run the checks in parallel. More importantly, it would be nice if the check for the dump file is clearly tied to the file's test. So run it right after the test, from the test wrapper. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: colorize NFT_TEST_HAS_SOCKET_LIMITSThomas Haller2023-09-181-11/+18
| | | | | | | | | | | | | | NFT_TEST_HAS_SOCKET_LIMITS= is similar to NFT_TEST_HAVE_* variables and indicates a feature (or lack thereof), except that it's inverted. Maybe this should be consolidated, however, NFT_TEST_HAS_SOCKET_LIMITS= is detected in the root namespace, unlike the shell scripts from features. So it's unclear how to consolidate them best. Anyway. Still highlight a lack of the capability, as it can cause tests to be skipped and we should see that easily. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: don't show the exit status for failed testsThomas Haller2023-09-181-6/+3
| | | | | | | | | | | | | | | | | | | | Previously, for failed tests we would print the exit code W: [FAILED] 2/2 tests/shell/testcases/listing/0013objects_0: got 1 This doesn't seem very useful. For one, we have special exit codes like 0 (OK), 77 (SKIPPED), 124 (DUMP FAIL), 123 (TAINTED), 122 (VALGRIND). Any other exit code is just an arbitrary failure. We don't define any special codes, and printing them is not useful. Note that further exit codes (118 - 121) are reserved, and could be special purposed, when there is a use. You can find the real exit code from the test in the result data in the "rc-failed" file. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: set C locale in "run-tests.sh"Thomas Haller2023-09-181-2/+6
| | | | | | | | | | The tests should run always the same, regardless of the user's language settings. Set LANG=C and LC_ALL=C and unset LANGUAGE. If some part wants to test a different language, it would set it explicitly. They anyway wouldn't want to depend on something from the user's environment. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: implement NFT_TEST_HAVE_json feature detection as scriptThomas Haller2023-09-181-14/+4
| | | | | | | | No more need to special case the "run a script" approach for detecting the json feature. Use the new mechanism instead. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: skip reset tests if kernel lacks supportFlorian Westphal2023-09-181-4/+21
| | | | | | | | | | | | reset is implemented via flush + extra attribute, so older kernels perform a flush. This means .nft doesn't work, we need to check if the individual set contents/sets are still in place post-reset. Make this generic and permit use of feat.sh in addition to the simpler foo.nft feature files. Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: suggest 4Mb /proc/sys/net/core/{wmem_max,rmem_max} for rootlessThomas Haller2023-09-181-3/+3
| | | | | | | | | | | | | | | | 2Mb was not enough to pass "tests/shell/testcases/sets/0030add_many_elements_interval_0" in an unprivileged/rootless namespace. Instead, bump the suggestion to 4Mb, which lets the test pass. Note that the 4Mb are only the recommended value when running the test as rootless, and is used to autodetect NFT_TEST_HAS_SOCKET_LIMITS=y. You can set whatever values are suitable for your environment, and explicitly indicate whether the limits are appropriate or not via NFT_TEST_HAS_SOCKET_LIMITS=n|y. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: colorize NFT_TEST_SKIP_/NFT_TEST_HAVE_ in test outputThomas Haller2023-09-161-2/+10
| | | | | | | | | Having a "SKIP" option as "y" or a "HAVE" option as "n", is note worthy because tests may be skipped based on that. Colorize, to make it easier to see in the test output. Signed-off-by: Thomas Haller <thaller@redhat.com>
* tests/shell: add feature probing via "features/*.nft" filesThomas Haller2023-09-161-0/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Running selftests on older kernels makes some of them fail very early because some tests use features that are not available on older kernels, e.g. -stable releases. Known examples: - inner header matching - anonymous chains - elem delete from packet path Also, some test cases might fail because a feature isn't compiled in, such as netdev chains. This adds a feature-probing mechanism to shell tests. Simply drop a 'nft -f' compatible file with a .nft suffix into "tests/shell/features". "run-tests.sh" will load it via `nft --check` and will export NFT_TEST_HAVE_${feature}=y|n Here ${feature} is the basename of the .nft file without file extension. It must be all lower-case. This extends the existing NFT_TEST_HAVE_json= feature detection. Similarly, NFT_TEST_REQUIRES(NFT_TEST_HAVE_*) tags work to easily skip a test. The test script that cannot fully work without the feature should either skip the test entirely (NFT_TEST_REQUIRES(NFT_TEST_HAVE_*)), or run a reduced/modified test. If a modified test was run and passes, it is still a good idea to mark the overall result as skipped (exit 77) instead of claiming success to the modified test. We want to know when not the full test was running, while we want to test as much as we can. This patch is based on Florian's feature probing patch. Originally-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: honor CLICOLOR_FORCE to force coloring in run-tests.shThomas Haller2023-09-151-5/+10
| | | | | | | | | | | | | We honor NO_COLOR= to disable coloring, let's also honor CLICOLOR_FORCE= to enable it. The purpose will be for `make` calling the script and redirecting to a file, while enabling colors. See-also: https://bixense.com/clicolors/ Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: accept $NFT_TEST_TMPDIR_TAG for the result directoryThomas Haller2023-09-151-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We allow the user to set "$TMPDIR" to affect where the "nft-test.*" directory is created. However, we don't allow the user to specify the exact location, so the user doesn't really know which directory was created. One remedy is that the test will also create the symlink "$TMPDIR/nft-test.latest.$USER" to point to the last test result. However, if you run multiple tests in parallel, that is not reliable to find the test results. Accept $NFT_TEST_TMPDIR_TAG and use it as part of the generated filename. That way, the caller can set it to a unique tag, and find the directory later based on that. For example export TMPDIR=/tmp export NFT_TEST_TMPDIR_TAG=".$(uuidgen)" ./tests/shell/run-tests.sh ls -lad "$TMPDIR/nft-test."*"$NFT_TEST_TMPDIR_TAG"*/ will work reliably -- as long as the tag is chosen uniquely. The reason to not allow the user to specify the directory name directly, is because we want that tests results follow the well-known pattern "/tmp/nft-test*". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: exit 77 from "run-tests.sh" if all tests were skippedThomas Haller2023-09-151-1/+7
| | | | | | | | | | | | | | If there are multiple tests and some of them pass and some are skipped, the overall result should be success (zero). Because likely the user just selected a bunch of tests (or all of them). So skipping some tests does not mean that the entire run is not a success. However, if all tests are skipped, then mark the overall result as skipped too. The more common case is if you only run one single test, then we want to know, that the test didn't run. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: in find_tests() use C locale for sorting tests namesThomas Haller2023-09-151-1/+1
| | | | | | | | It makes more sense, that the sort order does not depend on the user's locale. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: special handle base path starting with "./"Thomas Haller2023-09-151-1/+3
| | | | | | | | | | | | | | | | | | When we auto detect the tests with `tests/shell/run-tests.sh -L`, then commonly the NFT_TEST_BASEDIR starts with a redundant "./". That's a bit ugly. Instead, special handle that case and remove the prefix. The effect is that `tests/shell/run-tests.sh -L` shows tests/shell/testcases/bitwise/0040mark_binop_0 instead of ./tests/shell/testcases/bitwise/0040mark_binop_0 Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add option to shuffle execution order of testsThomas Haller2023-09-151-0/+18
| | | | | | | | | | | | The user can set NFT_TEST_SHUFFLE_TESTS=y|n to have the tests shuffled randomly. The purpose of shuffling is to find tests that depend on each other, or would break when run in unexpected order. If unspecified, by default tests are shuffled if no tests are selected on the command line. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: export NFT_TEST_RANDOM_SEED variable for testsThomas Haller2023-09-151-0/+46
| | | | | | | | | | | | | | | | | | | | | | | | Let "run-tests.sh" export a NFT_TEST_RANDOM_SEED variable, set to a decimal, random integer (in the range of 0 to 0x7FFFFFFF). The purpose is to provide a seed to tests for randomization. Randomizing tests is very useful to increase the coverage while not testing all combinations (which might not be practical). The point of NFT_TEST_RANDOM_SEED is that the user can set the environment variable so that the same series of random events is used. That is useful for reproducing an issue, that is known to happen with a certain seed. - by default, if the user leaves NFT_TEST_RANDOM_SEED unset or empty, the script generates a number using $SRANDOM. - if the user sets NFT_TEST_RANDOM_SEED to an integer it is taken as is (modulo 0x80000000). - otherwise, calculate a number by hashing the value of $NFT_TEST_RANDOM_SEED. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: ensure vgdb-pipe files are deleted from "nft-valgrind-wrapper.sh"Thomas Haller2023-09-141-1/+9
| | | | | | | | | | | | | When the valgrind process gets killed, those files can be left over. They are located in the original $TMPDIR (usually /tmp). They should be cleaned up. I tried to cleanup the files from withing "nft-valgrind-wrapper.sh" itself via a `trap`, but it doesn't work. Instead, let "run-tests.sh" delete all files with a matching pattern. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: kill running child processes when aborting "run-tests.sh"Thomas Haller2023-09-141-5/+26
| | | | | | | | | | | | | | When aborting "run-tests.sh", child processes were left running. Kill them. It's surprisingly complicated to get this somewhat right. Do it by enabling monitor mode for each test call, so that they run in separate process groups and we can kill the entire group. Note that we cannot just `kill -- -$$`, because it's not clear who is in this process group. Also, we don't want to kill the `tee` process which handles our logging. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: add "--quick" option to skip slow tests (via NFT_TEST_SKIP_slow=y)Thomas Haller2023-09-091-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It's important to run (a part) of the tests in a timely manner. Add an option to skip long running tests. Thereby, add a more general NFT_TEST_SKIP_* mechanism. This is related and inverse from "NFT_TEST_HAVE_json", where a test can require [ "$NFT_TEST_HAVE_json" != n ] to run, but is skipped when [ "$NFT_TEST_SKIP_slow" = y ]. Currently only NFT_TEST_SKIP_slow is supported. The user can set such environment variables (or use the -Q|--quick command line option). The configuration is printed in the test info. Tests should check for [ "$NFT_TEST_SKIP_slow" = y ] so that the variable has to be explicitly set to opt-out. For convenience, tests can also add a # NFT_TEST_SKIP(NFT_TEST_SKIP_slow) tag, which is evaluated by test-wrapper.sh. Or they can run a quick, reduced part of the test, but then should still indicate to be skipped. Mark 8 tests are as slow, that take longer than 5 seconds on my machine. With this, a parallel wall time for the non-slow tests is only 7 seconds (on my machine). The ultimate point is to integrate a call to "tests/shell/run-tests.sh" in a `make check` target. For development, you can then export NFT_TEST_SKIP_slow=y and have a fast `make check`. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: skip tests if nft does not support JSON modeThomas Haller2023-09-091-2/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We can build nft without JSON support, and some tests will fail without it. Instead, they should be skipped. Also note, that the test accepts any nft binary via the "NFT" environment variable. So it's not enough to make the skipping dependent on build configuration, but on the currently used $NFT variable. Let "run-test.sh" detect and export a "NFT_TEST_HAVE_json=y|n" variable. This is heavily inspired by Florian's feature probing patches. Tests that require JSON can check that variable, and skip. Note that they check in the form of [ "$NFT_TEST_HAVE_json" != n ], so the test is only skipped, if we explicitly detect lack of support. That is, don't check via [ "$NFT_TEST_HAVE_json" = y ]. Some of the tests still run parts of the tests that don't require JSON. Only towards the end of such partial run, mark the test as skipped. Some tests require JSON support throughout. For those, add a mechanism where tests can add a tag (in their first 10 lines): # NFT_TEST_REQUIRES(NFT_TEST_HAVE_json) This will be checked by "test-wrapper.sh", which will skip the test. The purpose of this is to make it low-effort to skip a test and to print the reason in the text output as Test skipped due to NFT_TEST_HAVE_json=n (test has "NFT_TEST_REQUIRES(NFT_TEST_HAVE_json)" tag) This is intentionally not shortened to NFT_TEST_REQUIRES(json), so that we can grep for NFT_TEST_HAVE_json to find all relevant places. Note that while NFT_TEST_HAVE_json is autodetected, the caller can also force it by setting the environment variable. This allows to see what would happen to such a test. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print number of completed tests to show progressThomas Haller2023-09-091-7/+13
| | | | | | | | | | | | | | | | | | Especially with VALGRIND=y, a full test run can take a long time. When looking at the output, it's interesting to get a feel how far along we are. Print the number of completed jobs vs. the number of total jobs, in the line showing the test result. It gives a nice progress status. Example: I: [OK] 1/373 ./tests/shell/testcases/bitwise/0040mark_binop_1 I: [OK] 2/373 ./tests/shell/testcases/bitwise/0040mark_binop_0 ... Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: set valgrind's "--vgdb-prefix=" to orignal TMPDIRThomas Haller2023-09-091-0/+5
| | | | | | | | | | | | | | | | | | | | | | | "test-wrapper.sh" sets TMPDIR="$NFT_TEST_TESTTMPDIR". That is useful, so that temporary files of the tests are placed inside the test result data. Sometimes tests miss to delete those files, which would result in piling up /tmp/tmp.XXXXXXXXXX files. By setting $TMPDIR, those files are clearly related to the test run that created them, and can be deleted together. However, valgrind likes to create files like "vgdb-pipe-from-vgdb-to-68-by-thom-on-???" inside $TMPDIR. These are pipes, so if you run `grep -R ^ /tmp/nft-test.latest` while the test is still running (to inspect the results), then the process hands reading from the pipe. Instead, tell valgrind to put those files in the original TMPDIR. For that purpose, export NFT_TEST_TMPDIR_ORIG from "run-tests.sh". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: set NFT_TEST_JOBS based on $(nproc)Thomas Haller2023-09-081-3/+7
| | | | | | | | Choose 150% of $(nproc) for the default vlaue of NFT_TEST_JOBS (rounded up). The minimal value chosen by default is 2. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: record wall time of test run in result dataThomas Haller2023-09-081-0/+16
| | | | | | | | | | | | | | | | | | | | | When running tests, it's useful to see how long it took. Keep track if the timestamps in a "times" file. Try: ( \ for d in /tmp/nft-test.latest.*/test-*/ ; do \ printf '%10.2f %s\n' \ "$(sed '1!d' "$d/times")" \ "$(cat "$d/name")" ; \ done \ | sort -n \ | awk '{print $0; s+=$1} END{printf("%10.2f\n", s)}' ; \ printf '%10.2f wall time\n' "$(sed '1!d' /tmp/nft-test.latest.*/times)" \ ) Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: no longer enable verbose output when selecting a testThomas Haller2023-09-081-4/+1
| | | | | | | | | | | | | | | | | | | | Previously, when selecting a test on the command line, it would also enable verbose output (except if the "--" separator was used). This convenience feature seems not great because the output from the test badly clutters the "run-test.sh" output. Now that the test results are all on disk, you can search them after the run with great flexibility (grep). Additionally, in previous versions, command line argument parsing was more restrictive, requiring that "-v" always be placed first. Now, the order does not matter, so it's easy to edit the command prompt and append a "-v", if that is what you want. Or if you really like verbose output, then `export VERBOSE=y`. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print "kernel is tainted" separate from test resultThomas Haller2023-09-081-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Once the kernel is tainted, it stays until reboot. It would not be useful to fail the entire test run based on that (and we don't do that). But then, it seems odd to print this in the same style as the test results, because a [FAILED] of a test counts as an overall failure. Instead, print this warning in a different style. Previously: $ ./tests/shell/run-tests.sh -- /usr/bin/true ... W: [FAILED] kernel is tainted I: [OK] /usr/bin/true I: results: [OK] 1 [SKIPPED] 0 [FAILED] 0 [TOTAL] 1 Now: $ ./tests/shell/run-tests.sh -- /usr/bin/true ... W: kernel is tainted I: [OK] /usr/bin/true I: results: [OK] 1 [SKIPPED] 0 [FAILED] 0 [TOTAL] 1 Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: redirect output of test script to file tooThomas Haller2023-09-081-0/+2
| | | | | | | | | | | It's useful to keep around for later. Redirect to the temporary directory. Note that the file content may be colorized too. `less -R` helps with that. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: don't redirect error/warning messages to stderrThomas Haller2023-09-081-2/+2
| | | | | | | | | | | | | | Writing some messages to stderr and some to stdout is not helpful. Once they are written to separate streams, it's hard to be sure about their relative order. Use grep to filter messages. Also, next we will redirect the entire output also to a file. There the output is also not split in two files. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: print the NFT setting with the VALGRIND=y wrapperThomas Haller2023-09-081-4/+5
| | | | | | | | | With this we see in the info output I: info: NFT=./tests/shell/helpers/nft-valgrind-wrapper.sh Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: fix handling failures with VALGRIND=yThomas Haller2023-09-081-1/+3
| | | | | | | | | | | | | | | | | | With VALGRIND=y, on memleaks the tests did not fail. Fix that by passing "--error-exitcode=122" to valgrind. But just returning 122 from $NFT command may not correctly fail the test. Instead, ensure to write a "rc-failed-valrind" file, which is picked up by "test-wrapper.sh" to properly handle the valgrind failure (and fail with error code 122 itself). Also, accept NFT_TEST_VALGRIND_OPTS variable to a pass additional arguments to valgrind. For example a "--suppressions" file. Also show the special error code [VALGRIND] in "run-test.sh". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: colorize terminal output with test resultThomas Haller2023-09-081-10/+61
| | | | | | | | | | Colors help to see what is important. It honors the common NO_COLOR=<anything> to disable coloring. It also does not colorize, if [ -t 1 ] indicates that stdout is not a terminal. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: cleanup print_test_result() and show TAINTED error codeThomas Haller2023-09-081-8/+10
| | | | | | | | | | | We will add more special error codes (122 for VALGRIND). Minor refactor of print_test_result() to make it easier to extend for that. Also, we will soon colorize the output. This preparation patch makes that easier too. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: skip test in rootless that hit socket buffer size limitThomas Haller2023-09-071-0/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The socket buffer limits like /proc/sys/net/core/{rmem_max,wmem_max} can cause tests to fail, when running rootless. That's because real-root can override those limits, rootless cannot. Add an environment variable NFT_TEST_HAS_SOCKET_LIMITS=*|n which is automatically set by "run-tests.sh". Certain tests will check for [ "$NFT_TEST_HAS_SOCKET_LIMITS" = y ] and skip the test. The user may manually bump those limits (requires root), and set NFT_TEST_HAS_SOCKET_LIMITS=n to get the tests to pass even as rootless. For example, the test passes with root: sudo ./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0 Without root, it would fail. Skip it instead: ./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0 ... I: [SKIPPED] tests/shell/testcases/sets/automerge_0 Or bump the limit: $ echo 3000000 | sudo tee /proc/sys/net/core/wmem_max $ NFT_TEST_HAS_SOCKET_LIMITS=n ./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0 ... I: [OK] tests/shell/testcases/sets/automerge_0 Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: bind mount private /var/run/netns in test containerThomas Haller2023-09-071-4/+28
| | | | | | | | | | | | | | | | | | | | | | Some tests want to run `ip netns add`, which requires write permissions to /var/run/netns. Also, /var/run/netns would be a systemwide mount path, and shared between the tests. We would want to isolate that. Fix that by bind mount a tmpfs inside the test wrapper, if we appear to have a private mount namespace. Fixes $ ./tests/shell/run-tests.sh -- tests/shell/testcases/netns/0001nft-f_0 Optimally, `ip netns add` would allow to specify a private location for those bind mounts. It seems that iproute2 is build with /var/run/netns, instead the more common /run/netns. Hence, handle /var/run instead of /run. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: support running tests in parallelThomas Haller2023-09-071-13/+73
| | | | | | | | | | | | | | | Add option to enable running jobs in parallel. The purpose is to speed up the run time of the tests. The global cleanup (removal of kernel modules) interferes with parallel jobs (or even with, unrelated jobs on the system). By setting NFT_TEST_JOBS= to a positive number, that cleanup is skipped. This option is too good to miss. Hence parallel execution is enabled by default, and you have to opt-out from it. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: move valgrind wrapper script to separate scriptThomas Haller2023-09-071-34/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, in valgrind mode we would generate one script, which had "$NFT" variable and the temp directory hard coded. Soon, we will run jobs in parallel, so they would need at least different temp directories. Also, we want to put the valgrind results are inside "$NFT_TEST_TESTTMPDIR", along the test data. Extract the wrapper script to a separate script. It does not need to be generated ad-hoc, instead it uses the environment variables "$NFT_REAL" and "$NFT_TEST_TESTTMPDIR", as "run-tests.sh" prepares them. Also, add a "$NFT_REAL" variable for the actual NFT binary. We wrap the "$NFT" variable with VALGRIND=y or the user may pass "NFT='valgrind nft'". We should have access to the real binary. That might be useful for example to call `ldd "$NFT_REAL" | grep libjansson` to check for JSON support. Also, we use libtool. So quite possible the nft binary is actually a shell script. Calling valgrind on that script results in a lot of leak reports from shell (and slows down the command). Instead, use `libtool --mode=execute`. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: move taint check to "test-wrapper.sh"Thomas Haller2023-09-071-12/+4
| | | | | | | | | We will run tests in parallel. That means, we have multiple tests data and results in fly. That becomes simpler, if we move more result data to the test-wrapper and out of "run-tests.sh". Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: rework printing of test resultsThomas Haller2023-09-071-39/+80
| | | | | | | | | | | | | | | | | | | | - "test-wrapper.sh" no longer will print the test output to its stdout. Instead, it only writes the testout.log file. - rework the loop "run-tests.sh" for printing the test results. It no longer captures the output of the test, as the wrapper is expected to be silent. Instead, they get the output from the result directory. The benefit is, that there is no duplication in what we print and the captured data in the result directory. The verbose mode is only for convenience, to safe looking at the test data. It's not essential otherwise. - also move the evaluation of the test result (and printing of the information) to a separate function. Later we want to run tests in parallel, so the steps need to be clearly separated. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: move the dump diff handling inside "test-wrapper.sh"Thomas Haller2023-09-071-28/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | This fits there better. At this point, we are still inside the unshared namespace and right after the test. The test-wrapper.sh should compare (and generate) the dumps. Also change behavior for DUMPGEN=y. - Previously it would only rewrite the dump if the dumpfile didn't exist yet. Now instead, always rewrite the file with DUMPGEN=y. The mode of operation is anyway, that the developer afterwards checks `git diff|status` to pick up the changes. There should be no changes to existing files (as existing tests are supposed to pass). So a diff there either means something went wrong (and we should see it) or it just means the dumps correctly should be regenerated. - also, only generate the file if the "dumps/" directory exists. This allows to write tests that don't have a dump file and don't get it automatically generated. The test wrapper will return a special error code 124 to indicate that the test passed, but the dumps file differed. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: support --keep-logs option (NFT_TEST_KEEP_LOGS=y) to preserve ↵Thomas Haller2023-09-071-3/+19
| | | | | | | | | | | test output The test output is now all collected in the temporary directory. On success, that directory is deleted. Add an option to always preserve that directory. Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
* tests/shell: interpret an exit code of 77 from scripts as "skipped"Thomas Haller2023-09-071-1/+10
| | | | | | | | | | | | Allow scripts to indicate that a test could not run by exiting 77. "77" is chosen as exit code from automake's testsuites ([1]). Compare to git-bisect which chooses 125 to indicate skipped. [1] https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>