Author: Ana Rey <anarey@gmail.com>
Date: 18/Sept/2014

Here, the automated regression testing for nftables and some test
files.

This script checks that the rule input and output of nft matches.
More details here below.

A) What is this testing?

This script tests two different paths:

* The rule input from the command-line. This checks the different steps
  from the command line to the kernel. This includes the parsing,
  evaluation and netlink generation steps.

* The output listing that is obtained from the kernel. This checks the
  different steps from the kernel to the command line: The netlink
  message parsing, postprocess and textify steps to display the rule
  listing.

As a final step, this script compares that the rule that is added can
be listed by nft.

B) What options are available?

The script offers the following options:

* Execute test files:

./nft-test.py                           # Run all test files
./nft-test.py path/file.t               # Run this test file

If there is a problem, it shows the differences between the rule that
is added and the rule that is listed by nft.

In case you hit an error, the script doesn't keep testing for more
families. Unless you specify the --force-family option.

* Execute broken tests:

./nft-test.py -e

This runs tests for rules that need a fix: This mode runs the lines that
that start with a "-" symbol.

* Debugging:

./nft-test.py -d

This shows all the commands that the script executes, so you can watch
its internal behaviour.

* Keep testing all families on error.

./nft-test.py -f

Don't stop testing for more families in case of error.

C) What is the structure of the test file?

A test file contains a set of rules that are added in the system.

Here, an example of a test file:

   :input;type filter hook input priority 0    # line 1

   *ip;test-ipv4;input                         # line 2
   *ip6;test-ipv6;input                        # line 3
   *inet;test-inet;input                       # line 4

   ah hdrlength != 11-23;ok;ah hdrlength < 11 ah hdrlength > 23   # line 5
   - tcp dport != {22-25}                                         # line 6

   !set1 ipv4_addr;ok                          # line 7
   ?set1 192.168.3.8 192.168.3.9;ok            # line 8
   # This is a commented-line.                 # line 9

Line 1 defines a chain. The name of this chain is "input". The type is "filter",
the hook is "input" and the priority is 0.

Line 2 defines a table. The name of the table is 'test-ipv4', the family is ip
and the chain to be added to it is 'input'. Lines 3 and 4 defines more tables for
different families so the rules in this test file are also tested there.

Line 5 defines the rule, the ";" character is used as separator of several
parts:

* Part 1: "ah hdrlength != 11-23" is the rule to check.
* Part 2: "ok" is the result expected with the execute of this rule.
* Part 3: "ah hdrlength < 11 ah hdrlength > 23". This is the expected
  output. You can leave this empty if the output is the same as the
  input.

Line 6 is a marked line. This means that this rule is tested if
'-e' is passed as argument to nft-test.py.

Line 7 adds a new set. The name of this set is "set1" and the type
of this set is "ipv4_addr".

Line 8 adds two elements into the 'set1' set: "192.168.3.8" and
"192.168.3.9". A whitespace separates the elements of the set.

Line 9 uses the "#" symbol that means that this line is commented out.

D) What is a payload file?

A payload file contains info about the netlink message exchanged to achieve the
transaction.

The output can be generated in two ways. Let's see an example via socket
matching.

   # generate an empty payload file
   $ touch inet/socket.t.payload

   $ ./nft-test.py inet/socket.t # this will generate inet/socket.t.payload.got

   $ mv inet/socket.t.payload.got inet/socket.t.payload

The other way is using nft --debug=netlink. This has a drawback over the former
option, as rules has to be run one by one and also a comment has to be added
before every rule in the payload file.

   $  nft --debug=netlink add rule ip sockip4 sockchain socket exists
   ip sockip4 sockchain
     [ match name socket rev 3 ]

E) The test folders

The test files are divided in several directories: ip, ip6, inet, arp,
bridge and any.

 * "ip" folder contains the test files that are executed in ip and inet
   table.

 * "ip6" folder contains the test files that are executed in ip6 and inet
   table.

 * "inet" folder contains the test files that are executed in the ip, ip6
    and inet table.

 * "arp" folder contains the test files that are executed in the arp
   table.

 * "bridge" folder: Here are the test files are executed in bridge
   tables.

 * "any" folder: Here are the test files are executed in ip, ip6, inet,
   arp and bridge tables.

F) Meaning of messages:

* A warning message means the rule input and output of nft mismatches.
* An error message means the nft-tool shows an error when we add it or
  the listing is broken after the rule is added.
* An info message means something that is not necessarily related to any test
  case and does not indicate faulty behaviour.

G) Acknowledgements

Thanks to the Outreach Program for Women (OPW) for sponsoring this test
infrastructure and my mentor Pablo Neira.

-EOF-