tests/shell: add "--quick" option to skip slow tests (via NFT_TEST_SKIP_slow=y)

It's important to run (a part) of the tests in a timely manner.
Add an option to skip long running tests.

Thereby, add a more general NFT_TEST_SKIP_* mechanism.

This is related and inverse from "NFT_TEST_HAVE_json", where a test
can require [ "$NFT_TEST_HAVE_json" != n ] to run, but is skipped when
[ "$NFT_TEST_SKIP_slow" = y ].

Currently only NFT_TEST_SKIP_slow is supported. The user can set such
environment variables (or use the -Q|--quick command line option). The
configuration is printed in the test info.

Tests should check for [ "$NFT_TEST_SKIP_slow" = y ] so that the
variable has to be explicitly set to opt-out. For convenience, tests can
also add a

    # NFT_TEST_SKIP(NFT_TEST_SKIP_slow)

tag, which is evaluated by test-wrapper.sh. Or they can run a quick, reduced
part of the test, but then should still indicate to be skipped.

Mark 8 tests are as slow, that take longer than 5 seconds on my machine.
With this, a parallel wall time for the non-slow tests is only 7 seconds
(on my machine).

The ultimate point is to integrate a call to "tests/shell/run-tests.sh"
in a `make check` target. For development, you can then export
NFT_TEST_SKIP_slow=y and have a fast `make check`.

Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>

8 files changed, 22 insertions, 0 deletions

diff --git a/tests/shell/testcases/maps/0004interval_map_create_once_0 b/tests/shell/testcases/maps/0004interval_map_create_once_0
index 3de0c9de..64f434ad 100755
--- a/tests/shell/testcases/maps/0004interval_map_create_once_0
+++ b/tests/shell/testcases/maps/0004interval_map_create_once_0
@@ -5,6 +5,10 @@
 
 HOWMANY=63
 
+if [ "$NFT_TEST_SKIP_slow" = y ] ; then
+	HOWMANY=5
+fi
+
 tmpfile=$(mktemp)
 if [ ! -w $tmpfile ] ; then
 	echo "Failed to create tmp file" >&2
@@ -64,3 +68,7 @@ if [ "$EXPECTED" != "$GET" ] ; then
 	exit 1
 fi
 
+if [ "$HOWMANY" != 63 ] ; then
+	echo "Run a partial test due to NFT_TEST_SKIP_slow=y. Skip"
+	exit 77
+fi
diff --git a/tests/shell/testcases/maps/0018map_leak_timeout_0 b/tests/shell/testcases/maps/0018map_leak_timeout_0
index 5a07ec74..09db315a 100755
--- a/tests/shell/testcases/maps/0018map_leak_timeout_0
+++ b/tests/shell/testcases/maps/0018map_leak_timeout_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_SKIP(NFT_TEST_SKIP_slow)
+
 set -e
 
 RULESET="table ip t {
diff --git a/tests/shell/testcases/maps/vmap_timeout b/tests/shell/testcases/maps/vmap_timeout
index e59d37ab..43d03197 100755
--- a/tests/shell/testcases/maps/vmap_timeout
+++ b/tests/shell/testcases/maps/vmap_timeout
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_SKIP(NFT_TEST_SKIP_slow)
+
 set -e
 
 dumpfile=$(dirname $0)/dumps/$(basename $0).nft
diff --git a/tests/shell/testcases/sets/0043concatenated_ranges_0 b/tests/shell/testcases/sets/0043concatenated_ranges_0
index 90ee6a82..4165b2f5 100755
--- a/tests/shell/testcases/sets/0043concatenated_ranges_0
+++ b/tests/shell/testcases/sets/0043concatenated_ranges_0
@@ -1,5 +1,7 @@
 #!/bin/sh -e
 #
+# NFT_TEST_SKIP(NFT_TEST_SKIP_slow)
+#
 # 0043concatenated_ranges_0 - Add, get, list, timeout for concatenated ranges
 #
 # Cycle over supported data types, forming concatenations of three fields, for
diff --git a/tests/shell/testcases/sets/0044interval_overlap_0 b/tests/shell/testcases/sets/0044interval_overlap_0
index face90f2..19aa6f5e 100755
--- a/tests/shell/testcases/sets/0044interval_overlap_0
+++ b/tests/shell/testcases/sets/0044interval_overlap_0
@@ -1,5 +1,7 @@
 #!/bin/sh -e
 #
+# NFT_TEST_SKIP(NFT_TEST_SKIP_slow)
+#
 # 0044interval_overlap_0 - Add overlapping and non-overlapping intervals
 #
 # Check that adding overlapping intervals to a set returns an error, unless:
diff --git a/tests/shell/testcases/sets/0044interval_overlap_1 b/tests/shell/testcases/sets/0044interval_overlap_1
index eeea1943..905e6d5a 100755
--- a/tests/shell/testcases/sets/0044interval_overlap_1
+++ b/tests/shell/testcases/sets/0044interval_overlap_1
@@ -1,5 +1,7 @@
 #!/bin/sh -e
 #
+# NFT_TEST_SKIP(NFT_TEST_SKIP_slow)
+#
 # 0044interval_overlap_1 - Single-sized intervals can never overlap partially
 #
 # Check that inserting, deleting, and inserting single-sized intervals again
diff --git a/tests/shell/testcases/sets/automerge_0 b/tests/shell/testcases/sets/automerge_0
index fc34f886..170c3865 100755
--- a/tests/shell/testcases/sets/automerge_0
+++ b/tests/shell/testcases/sets/automerge_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_SKIP(NFT_TEST_SKIP_slow)
+
 set -e
 
 RULESET="table inet x {
diff --git a/tests/shell/testcases/transactions/30s-stress b/tests/shell/testcases/transactions/30s-stress
index 757e7639..4d5d1d8b 100755
--- a/tests/shell/testcases/transactions/30s-stress
+++ b/tests/shell/testcases/transactions/30s-stress
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_SKIP(NFT_TEST_SKIP_slow)
+
 runtime=30
 
 # allow stand-alone execution as well, e.g. '$0 3600'