| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Let "run-tests.sh" export a NFT_TEST_RANDOM_SEED variable, set to
a decimal, random integer (in the range of 0 to 0x7FFFFFFF).
The purpose is to provide a seed to tests for randomization.
Randomizing tests is very useful to increase the coverage while not
testing all combinations (which might not be practical).
The point of NFT_TEST_RANDOM_SEED is that the user can set the
environment variable so that the same series of random events is used.
That is useful for reproducing an issue, that is known to happen with a
certain seed.
- by default, if the user leaves NFT_TEST_RANDOM_SEED unset or empty,
the script generates a number using $SRANDOM.
- if the user sets NFT_TEST_RANDOM_SEED to an integer it is taken
as is (modulo 0x80000000).
- otherwise, calculate a number by hashing the value of
$NFT_TEST_RANDOM_SEED.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When the valgrind process gets killed, those files can be left over.
They are located in the original $TMPDIR (usually /tmp). They should be
cleaned up.
I tried to cleanup the files from withing "nft-valgrind-wrapper.sh"
itself via a `trap`, but it doesn't work. Instead, let "run-tests.sh"
delete all files with a matching pattern.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When aborting "run-tests.sh", child processes were left running. Kill
them. It's surprisingly complicated to get this somewhat right. Do it by
enabling monitor mode for each test call, so that they run in separate
process groups and we can kill the entire group.
Note that we cannot just `kill -- -$$`, because it's not clear who is in
this process group. Also, we don't want to kill the `tee` process which
handles our logging.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
| |
@ih fails on kernels where payload expression doesn't support the 'inner'
base offset.
This test isn't about inner headers, so just use @nh which is
universally available.
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
| |
This test fails on kernels that lack
05abe4456fa3 ("netfilter: nf_tables: allow to register flowtable with no devices")
v5.8-rc1~165^2~27^2~1
Just add lo as dummy device.
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It's important to run (a part) of the tests in a timely manner.
Add an option to skip long running tests.
Thereby, add a more general NFT_TEST_SKIP_* mechanism.
This is related and inverse from "NFT_TEST_HAVE_json", where a test
can require [ "$NFT_TEST_HAVE_json" != n ] to run, but is skipped when
[ "$NFT_TEST_SKIP_slow" = y ].
Currently only NFT_TEST_SKIP_slow is supported. The user can set such
environment variables (or use the -Q|--quick command line option). The
configuration is printed in the test info.
Tests should check for [ "$NFT_TEST_SKIP_slow" = y ] so that the
variable has to be explicitly set to opt-out. For convenience, tests can
also add a
# NFT_TEST_SKIP(NFT_TEST_SKIP_slow)
tag, which is evaluated by test-wrapper.sh. Or they can run a quick, reduced
part of the test, but then should still indicate to be skipped.
Mark 8 tests are as slow, that take longer than 5 seconds on my machine.
With this, a parallel wall time for the non-slow tests is only 7 seconds
(on my machine).
The ultimate point is to integrate a call to "tests/shell/run-tests.sh"
in a `make check` target. For development, you can then export
NFT_TEST_SKIP_slow=y and have a fast `make check`.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We can build nft without JSON support, and some tests will fail without
it. Instead, they should be skipped. Also note, that the test accepts any
nft binary via the "NFT" environment variable. So it's not enough to
make the skipping dependent on build configuration, but on the currently
used $NFT variable.
Let "run-test.sh" detect and export a "NFT_TEST_HAVE_json=y|n" variable. This
is heavily inspired by Florian's feature probing patches.
Tests that require JSON can check that variable, and skip. Note that
they check in the form of [ "$NFT_TEST_HAVE_json" != n ], so the test is
only skipped, if we explicitly detect lack of support. That is, don't
check via [ "$NFT_TEST_HAVE_json" = y ].
Some of the tests still run parts of the tests that don't require JSON.
Only towards the end of such partial run, mark the test as skipped.
Some tests require JSON support throughout. For those, add a mechanism
where tests can add a tag (in their first 10 lines):
# NFT_TEST_REQUIRES(NFT_TEST_HAVE_json)
This will be checked by "test-wrapper.sh", which will skip the test.
The purpose of this is to make it low-effort to skip a test and to print
the reason in the text output as
Test skipped due to NFT_TEST_HAVE_json=n (test has "NFT_TEST_REQUIRES(NFT_TEST_HAVE_json)" tag)
This is intentionally not shortened to NFT_TEST_REQUIRES(json), so that
we can grep for NFT_TEST_HAVE_json to find all relevant places.
Note that while NFT_TEST_HAVE_json is autodetected, the caller can also
force it by setting the environment variable. This allows to see what
would happen to such a test.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Especially with VALGRIND=y, a full test run can take a long time. When
looking at the output, it's interesting to get a feel how far along we
are.
Print the number of completed jobs vs. the number of total jobs, in the
line showing the test result. It gives a nice progress status.
Example:
I: [OK] 1/373 ./tests/shell/testcases/bitwise/0040mark_binop_1
I: [OK] 2/373 ./tests/shell/testcases/bitwise/0040mark_binop_0
...
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
"test-wrapper.sh" sets TMPDIR="$NFT_TEST_TESTTMPDIR". That is useful, so
that temporary files of the tests are placed inside the test result
data.
Sometimes tests miss to delete those files, which would result in piling
up /tmp/tmp.XXXXXXXXXX files. By setting $TMPDIR, those files are
clearly related to the test run that created them, and can be deleted
together.
However, valgrind likes to create files like
"vgdb-pipe-from-vgdb-to-68-by-thom-on-???" inside $TMPDIR. These are
pipes, so if you run `grep -R ^ /tmp/nft-test.latest` while
the test is still running (to inspect the results), then the process
hands reading from the pipe.
Instead, tell valgrind to put those files in the original TMPDIR. For
that purpose, export NFT_TEST_TMPDIR_ORIG from "run-tests.sh".
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
DUMPGEN=y mode skips tests that don't have a corresponding "dumps/"
directory.
Add the "dumps/" directory for tests that lacked it, and generate ".nft"
files by running `./tests/shell/run-tests.sh -g`.
Yes, they are all empty. Not very exciting, but why not check for that
too?
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
| |
These files are generated by running `./tests/shell/run-tests.sh -g`.
Commit the .nodump files to git.
The point is to explicitly make it known that no dump file should be
there. This prevents `./tests/shell/run-tests.sh -g` from generating
the files and proposing (over and over) to add them to git.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Several tests didn't have a ".nft" dump file committed. Generate one and
commit it to git.
While not all tests have a stable ruleset to compare, many have. Commit
the .nft files for the tests where the output appears to be stable.
This was generated by running `./tests/shell/run-tests.sh -g` twice, and
commit the files that were identical both times. Note that 7 tests on my
machine fail, so those are skipped.
Also skip the files
tests/shell/testcases/maps/dumps/0004interval_map_create_once_0.nft
tests/shell/testcases/nft-f/dumps/0011manydefines_0.nft
tests/shell/testcases/sets/dumps/0011add_many_elements_0.nft
tests/shell/testcases/sets/dumps/0030add_many_elements_interval_0.nft
tests/shell/testcases/sets/dumps/0068interval_stack_overflow_0.nft
Those files are larger than 100KB, and I don't think we want to blow up
the git repository this way. Even if they are only text files and
compress well.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For some tests, the dump is not stable or useful to test. For example,
if they have an "expires" timestamps. Those tests don't have a .nft file
in the dumps directory, and don't have it checked.
DUMPGEN=y generates a new dump file, if the "dumps/" directory exists.
Omitting that directory is a way to prevent the generation of the file.
However, many such tests share their directory with tests that do have dumps.
When running tests with DUMPGEN=y, new files for old tests are generated.
Those files are not meant to be compared or committed to git because
it's known to not work.
Whether a test has a dump file, is part of the test. The absence of the
dump file should also be recorded and committed to git.
Add a way to opt-out from such generating such dumps by having .nodump
files instead of the .nft dump.
Later we should add unit tests that checks that no test has both a .nft
and a .nodump file in git, that the .nodump file is always empty, and
that every .nft/.nodump file has a corresponding test committed to git.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
| |
Choose 150% of $(nproc) for the default vlaue of NFT_TEST_JOBS
(rounded up). The minimal value chosen by default is 2.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When running tests, it's useful to see how long it took. Keep track if
the timestamps in a "times" file.
Try:
( \
for d in /tmp/nft-test.latest.*/test-*/ ; do \
printf '%10.2f %s\n' \
"$(sed '1!d' "$d/times")" \
"$(cat "$d/name")" ; \
done \
| sort -n \
| awk '{print $0; s+=$1} END{printf("%10.2f\n", s)}' ; \
printf '%10.2f wall time\n' "$(sed '1!d' /tmp/nft-test.latest.*/times)" \
)
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, when selecting a test on the command line, it would also
enable verbose output (except if the "--" separator was used).
This convenience feature seems not great because the output from the
test badly clutters the "run-test.sh" output.
Now that the test results are all on disk, you can search them after the
run with great flexibility (grep).
Additionally, in previous versions, command line argument parsing was
more restrictive, requiring that "-v" always be placed first. Now, the
order does not matter, so it's easy to edit the command prompt and
append a "-v", if that is what you want. Or if you really like verbose
output, then `export VERBOSE=y`.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Once the kernel is tainted, it stays until reboot. It would not be
useful to fail the entire test run based on that (and we don't do that).
But then, it seems odd to print this in the same style as the test
results, because a [FAILED] of a test counts as an overall failure.
Instead, print this warning in a different style.
Previously:
$ ./tests/shell/run-tests.sh -- /usr/bin/true
...
W: [FAILED] kernel is tainted
I: [OK] /usr/bin/true
I: results: [OK] 1 [SKIPPED] 0 [FAILED] 0 [TOTAL] 1
Now:
$ ./tests/shell/run-tests.sh -- /usr/bin/true
...
W: kernel is tainted
I: [OK] /usr/bin/true
I: results: [OK] 1 [SKIPPED] 0 [FAILED] 0 [TOTAL] 1
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
| |
It's useful to keep around for later. Redirect to the temporary
directory.
Note that the file content may be colorized too. `less -R` helps with
that.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Writing some messages to stderr and some to stdout is not helpful.
Once they are written to separate streams, it's hard to be sure about
their relative order.
Use grep to filter messages.
Also, next we will redirect the entire output also to a file. There the
output is also not split in two files.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
| |
With this we see in the info output
I: info: NFT=./tests/shell/helpers/nft-valgrind-wrapper.sh
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With VALGRIND=y, on memleaks the tests did not fail. Fix that by passing
"--error-exitcode=122" to valgrind.
But just returning 122 from $NFT command may not correctly fail the test.
Instead, ensure to write a "rc-failed-valrind" file, which is picked up
by "test-wrapper.sh" to properly handle the valgrind failure (and fail
with error code 122 itself).
Also, accept NFT_TEST_VALGRIND_OPTS variable to a pass additional
arguments to valgrind. For example a "--suppressions" file.
Also show the special error code [VALGRIND] in "run-test.sh".
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
| |
Colors help to see what is important.
It honors the common NO_COLOR=<anything> to disable coloring. It also
does not colorize, if [ -t 1 ] indicates that stdout is not a terminal.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
| |
We will add more special error codes (122 for VALGRIND). Minor refactor
of print_test_result() to make it easier to extend for that.
Also, we will soon colorize the output. This preparation patch makes
that easier too.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The previous code was mostly correct, but hard to understand.
Rework it.
Also, on failure now always write "rc-failed-exit", which is the exit
code that "test-wrapper.sh" reports to "run-test.sh". Note that this
error code may not be the same as the one returned by the TEST binary.
The latter you can find in one of the files "rc-{ok,skipped,failed}".
In general, you can search the directory with test results for those
"rc-*" files. If you find a "rc-failed" file, it was counted as failure.
There might be other "rc-failed-*" files, depending on whether the diff
failed or kernel got tainted.
Also, reserve all the error codes 118 - 124 for the "test-wrapper.sh".
For example, 124 means a dump difference and 123 means kernel got
tainted. In the future, 122 will mean a valgrind error. Other numbers
are not reserved. If a test command fails with such an reserved code,
"test-wrapper.sh" modifies it to 125, so that "run-test.sh" does not get
the wrong idea about the failure reason. This is not new in this patch,
except that the reserved range was extended for future additions.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There are some existing tests, that skip operation when they fail to
create a dummy interface. Use the new exit code 77 to indicate
"SKIPPED".
I wonder why creating a dummy device would ever fail and why we don't
just fail the test altogether in that case. But the patch does not
change that.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Various tests create additional temporary files. They really should just
use "$NFT_TEST_TESTTMPDIR" for that. However, they mostly use `mktemp`.
The scripts are supposed to cleanup those files afterwards. However,
often that does not work correctly and /tmp gets full of left over
temporary files.
Export "TMPDIR" so that they use the test-specific temporary directory.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
| |
We are going to set $TMPDIR to another location. The previous code made
assumptions that the generated path would always be in /tmp. Fix that.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Runtimes are important. Add a way to find out how long tests took.
$ ./tests/shell/run-tests.sh
...
$ for d in /tmp/nft-test.latest.*/test-*/ ; do \
printf '%10.2f %s\n' \
"$(sed '1!d' "$d/times")" \
"$(cat "$d/name")" ; \
done \
| sort -n \
| awk '{print $0; s+=$1} END{printf("%10.2f\n", s)}'
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The socket buffer limits like /proc/sys/net/core/{rmem_max,wmem_max}
can cause tests to fail, when running rootless. That's because real-root
can override those limits, rootless cannot.
Add an environment variable NFT_TEST_HAS_SOCKET_LIMITS=*|n which is
automatically set by "run-tests.sh".
Certain tests will check for [ "$NFT_TEST_HAS_SOCKET_LIMITS" = y ] and
skip the test.
The user may manually bump those limits (requires root), and set
NFT_TEST_HAS_SOCKET_LIMITS=n to get the tests to pass even as rootless.
For example, the test passes with root:
sudo ./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0
Without root, it would fail. Skip it instead:
./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0
...
I: [SKIPPED] tests/shell/testcases/sets/automerge_0
Or bump the limit:
$ echo 3000000 | sudo tee /proc/sys/net/core/wmem_max
$ NFT_TEST_HAS_SOCKET_LIMITS=n ./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0
...
I: [OK] tests/shell/testcases/sets/automerge_0
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some tests want to run `ip netns add`, which requires write permissions
to /var/run/netns. Also, /var/run/netns would be a systemwide mount
path, and shared between the tests. We would want to isolate that.
Fix that by bind mount a tmpfs inside the test wrapper, if we appear to
have a private mount namespace.
Fixes
$ ./tests/shell/run-tests.sh -- tests/shell/testcases/netns/0001nft-f_0
Optimally, `ip netns add` would allow to specify a private
location for those bind mounts.
It seems that iproute2 is build with /var/run/netns, instead the more
common /run/netns. Hence, handle /var/run instead of /run.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add option to enable running jobs in parallel. The purpose is to speed
up the run time of the tests.
The global cleanup (removal of kernel modules) interferes with parallel
jobs (or even with, unrelated jobs on the system). By setting
NFT_TEST_JOBS= to a positive number, that cleanup is skipped.
This option is too good to miss. Hence parallel execution is enabled by
default, and you have to opt-out from it.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, in valgrind mode we would generate one script, which had
"$NFT" variable and the temp directory hard coded.
Soon, we will run jobs in parallel, so they would need at least
different temp directories. Also, we want to put the valgrind results
are inside "$NFT_TEST_TESTTMPDIR", along the test data.
Extract the wrapper script to a separate script. It does not need to be
generated ad-hoc, instead it uses the environment variables "$NFT_REAL" and
"$NFT_TEST_TESTTMPDIR", as "run-tests.sh" prepares them.
Also, add a "$NFT_REAL" variable for the actual NFT binary. We wrap the
"$NFT" variable with VALGRIND=y or the user may pass "NFT='valgrind
nft'". We should have access to the real binary. That might be useful
for example to call `ldd "$NFT_REAL" | grep libjansson` to check for
JSON support.
Also, we use libtool. So quite possible the nft binary is actually a
shell script. Calling valgrind on that script results in a lot of leak
reports from shell (and slows down the command). Instead, use `libtool
--mode=execute`.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
| |
We will run tests in parallel. That means, we have multiple tests data and results
in fly. That becomes simpler, if we move more result data to the
test-wrapper and out of "run-tests.sh".
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- "test-wrapper.sh" no longer will print the test output to its stdout.
Instead, it only writes the testout.log file.
- rework the loop "run-tests.sh" for printing the test results. It no
longer captures the output of the test, as the wrapper is expected to
be silent. Instead, they get the output from the result directory.
The benefit is, that there is no duplication in what we print and the
captured data in the result directory. The verbose mode is only for
convenience, to safe looking at the test data. It's not essential
otherwise.
- also move the evaluation of the test result (and printing of the
information) to a separate function. Later we want to run tests in
parallel, so the steps need to be clearly separated.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This fits there better. At this point, we are still inside the unshared
namespace and right after the test. The test-wrapper.sh should compare
(and generate) the dumps.
Also change behavior for DUMPGEN=y.
- Previously it would only rewrite the dump if the dumpfile didn't
exist yet. Now instead, always rewrite the file with DUMPGEN=y.
The mode of operation is anyway, that the developer afterwards
checks `git diff|status` to pick up the changes. There should be
no changes to existing files (as existing tests are supposed to
pass). So a diff there either means something went wrong (and we
should see it) or it just means the dumps correctly should be
regenerated.
- also, only generate the file if the "dumps/" directory exists. This
allows to write tests that don't have a dump file and don't get it
automatically generated.
The test wrapper will return a special error code 124 to indicate that
the test passed, but the dumps file differed.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
| |
test output
The test output is now all collected in the temporary directory. On
success, that directory is deleted. Add an option to always preserve
that directory.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Allow scripts to indicate that a test could not run by exiting 77.
"77" is chosen as exit code from automake's testsuites ([1]). Compare to
git-bisect which chooses 125 to indicate skipped.
[1] https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Don't unshare the entire shell script. Instead, call unshare each test
separately. That means, all tests use now a different sandbox and will
also allow (with further changes) to run them in parallel.
Also, allow to run rootless/unprivileged.
The script first tries to run a separate PID+USER+NET namespace. If that
fails, it downgrades to USER+NET. If that fails, it downgrades to a
separate NET namespace. If unshare still fails, the script fails
entirely. That differs from before, where the script would proceed
without sandboxing. The script will now always require that unsharing
works, unless the user opts-out.
If the user cannot unshare, they can set NFT_TEST_UNSHARE_CMD to the
command used for unsharing. It may be empty for no unshare. The command
line arguments -U/--no-unshare are a shortcut for setting
NFT_TEST_UNSHARE_CMD="".
If we are able to create a separate USER namespace, then this mode
allows to run the test as rootless/unprivileged. We no longer require
[ `id -u` = 0 ]. Some tests may not work as rootless. For example, the
socket buffers is limited by /proc/sys/net/core/{wmem_max,rmem_max}
which real-root can override, but rootless tests cannot. Such tests
should check for [ "$NFT_TEST_HAS_REALROOT" != y ] and skip gracefully.
Usually, the user doesn't need to tell the script whether they have
real-root. The script will autodetect it via [ `id -u` = 0 ]. But that
won't work when run inside a rootless container already. In that case,
the user would want to tell the script that there is no real-root. They
can do so via the -R/--without-root option or NFT_TEST_HAS_REALROOT=n.
If tests wish, the can know whether they run inside "unshare"
environment by checking for [ "$NFT_TEST_HAS_UNSHARED" = y ].
When setting NFT_TEST_UNSHARE_CMD to override the unshare command, users
may want to also set NFT_TEST_HAS_UNSHARED= and NFT_TEST_HAS_REALROOT=
correctly.
As we run each test in a separate unshare environment, we need a wrapper
"tests/shell/helpers/test-wrapper.sh" around the test, which executes
inside the tested environment. Also, each test gets its own temp
directory prepared in NFT_TEST_TESTTMPDIR. This is also the place, where
test artifacts and results will be collected.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
| |
As the script can be configured via environment variables or command
line option, it's useful to show the environment variables that we
received or set during the test setup.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, we would honor "y" as opt-in, and all other values meant
false.
- accept alternatives to "y", like "1" or "true".
- normalize the value, to either be "y" or "n".
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Let the test wrapper prepare and export two environment variables for
the test:
- "$NFT_TEST_BASEDIR" is just the top directory where the test scripts
lie.
- "$NFT_TEST_TMPDIR" is a `mktemp` directory created by "run-tests.sh"
and removed at the end. Tests may use that to leave data there.
This directory will be used for various things, like the "nft" wrapper
in valgrind mode, the results of the tests and possibly as cache for
feature detection.
The "$NFT_TEST_TMPDIR" was already used before with the "VALGRIND=y"
mode. It's only renamed and got an extended purpose.
Also drop the unnecessary first detection of "$DIFF" and the "$SRC_NFT"
variable.
Also, note that the mktemp creates the temporary directory under /tmp.
Which is commonly a tempfs. The user can override that by exporting
TMPDIR.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
| |
Check for valid test names early. That's useful because we treat any
unrecognized options as test names. We should detect a mistake early.
While at it, also support specifying directory names instead of
executable files.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Cleanup finding the test files. Also add a "--list-tests" option to see
which tests are found and would run.
Also get rid of the FIND="$(which find)" detection. Which system doesn't
have a working find? Also, we can just fail when we try to use find, and
don't need a check first.
This is still after "unshare", which will be addressed next.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Parse the arguments in a loop, so that their order does not matter.
Also, soon more command line arguments will be added, and this way of
parsing seems more maintainable and flexible.
Currently this is still after the is-root check and after unshare. That
will be addressed later.
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
| |
This script suppressed a few tests when ran via run-tests.sh,
don't do that, it would have caught the previous 'get' bug
years ago.
Signed-off-by: Florian Westphal <fw@strlen.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The numgen extension generates numbers in little-endian.
This can be very tricky when trying to combine it with IP addresses, which use big endian.
This change adds a new byteorder operation to convert data type endianness.
Before this patch:
$ sudo nft -d netlink add rule nat snat_chain snat to numgen inc mod 7 offset 0x0a000001
ip nat snat_chain
[ numgen reg 1 = inc mod 7 offset 167772161 ]
[ nat snat ip addr_min reg 1 ]
After this patch:
$ sudo nft -d netlink add rule nat snat_chain snat to numgen inc mod 7 offset 0x0a000001
ip nat snat_chain
[ numgen reg 1 = inc mod 7 offset 167772161 ]
[ byteorder reg 1 = hton(reg 1, 4, 4) ]
[ nat snat ip addr_min reg 1 ]
Regression tests have been modified to include these new cases.
Signed-off-by: Jorge Ortiz Escribano <jorge.ortiz.escribano@gmail.com>
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
|
|
|
|
|
|
| |
This bytecode output file contains many duplicated entries, remove them.
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
|
|
|
|
|
|
| |
Cover matching on DF and MF bits and fragments.
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The ip frag-off field in the protocol definition is 16-bits long
and it contains the DF (0x2000) and MF (0x4000) bits too.
iptables-translate also suggests:
ip frag-off & 0x1ffff != 0
to match on fragments. Use hexadecimal for listing this header field.
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
|
|
|
|
|
|
|
|
|
|
| |
Let the last few batches also push an update that contains
elements twice.
This is expected to cause the batch to be aborted,
which increases code coverage on kernel side.
Signed-off-by: Florian Westphal <fw@strlen.de>
|