Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions doc/project/guide/change-a-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,14 @@ If investigation is still needed, keep the failing outputs as evidence. Investig
## 3. Check the scope of the movement

```bash
opp_run_fingerprint_tests --load $INET_ROOT -p inet -f '<affected-case-regex>'
opp_run_fingerprint_tests -f '<affected-case-regex>'
```

Start with an explicit filter for the configurations directly related to the changed contract and record the command, mode, configuration, run/seed, status and generated artifacts ([TR-FOCUSED-EVIDENCE](../rule/testing.md#tr-focused-evidence)). A broader integration run may then look for collateral movement, but it does not replace the focused evidence. A change that moves far more configurations than expected is telling you something. A serializer change that moves a mobility fingerprint is not a serializer change.

## 4. Regenerate

After approval, regenerate only the baselines the change actually moves. A blanket regeneration sweeps up unrelated drift and hides it under your reason. For fingerprints, run `opp_update_fingerprint_test_results --load $INET_ROOT -p inet -f '<affected-case-regex>'` with the filter of step 3, and read `git diff tests/fingerprint/store.json`.
After approval, regenerate only the baselines the change actually moves. A blanket regeneration sweeps up unrelated drift and hides it under your reason. For fingerprints, run `opp_update_fingerprint_test_results -f '<affected-case-regex>'` with the filter of step 3, and read `git diff tests/fingerprint/store.json`.

## 5. Commit it with the change that causes it

Expand Down
2 changes: 1 addition & 1 deletion doc/project/guide/run-the-gates.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Map each changed path, symbol or behavioral contract to the directly related cas

```bash
doc/project/enforcement/check-architecture.sh src/inet/<affected-subtree>
opp_run_fingerprint_tests --load $INET_ROOT -p inet -f '<affected-case-regex>'
opp_run_fingerprint_tests -f '<affected-case-regex>'
# Use the category runner's explicit selector for unit, module, protocol and other tests.
```

Expand Down
28 changes: 16 additions & 12 deletions doc/src/developers-guide/ch-testing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,19 @@ for each concrete config that can run without user input, and some more runs.

The fingerprint tests run with the :command:`opp_run_fingerprint_tests` command of
`opp_repl <https://github.com/omnetpp/opp_repl>`__. Set up the OMNeT++ and INET
environment first. Then run all fingerprint tests from anywhere in the INET tree:
environment first. Then run all fingerprint tests from the INET root:

.. code-block:: shell

$ opp_run_fingerprint_tests --load $INET_ROOT -p inet
$ cd $INET_ROOT
$ opp_run_fingerprint_tests

The ``--load $INET_ROOT`` option loads INET's project description, the file
:file:`inet.opp` in the INET root, and ``-p inet`` selects INET. From the INET root,
opp_repl loads the file without the option. The description sets the store, the
The opp_repl commands find INET's project description, the file :file:`inet.opp` in
the INET root, without an option: they look for ``*.opp`` files in the current folder
and then in its parent folders. So the commands of this chapter work in each folder
of the INET tree. In a subfolder, a command without a filter tests only that
subfolder. Outside the INET tree, add ``--load $INET_ROOT -p inet``. The description
sets the store, the
ingredients that CI checks (``tplx``, ``~tNl`` and ``~tND``) and a CPU time limit of
600s for each simulation. The command shows its options with ``-h``.

Expand All @@ -60,7 +64,7 @@ For example, the following command runs all wireless tests in release mode:

.. code-block:: shell

$ opp_run_fingerprint_tests --load $INET_ROOT -p inet -m release -f wireless
$ opp_run_fingerprint_tests -m release -f wireless

While the tests are running, the command prints a line for each simulation. The
test result of each fingerprint is PASS, FAIL or ERROR, and a FAIL line shows the
Expand All @@ -78,13 +82,13 @@ To accept new fingerprints, run the update command with the same filter:

.. code-block:: shell

$ opp_update_fingerprint_test_results --load $INET_ROOT -p inet -f wireless
$ opp_update_fingerprint_test_results -f wireless

The update runs each stored run at its stored limit, and writes the new values into
the store. It reports KEEP, UPDATE or INSERT for each simulation, and
:command:`git diff tests/fingerprint/store.json` shows the changed values. A new
config gets its run-0 fingerprints with ``--no-stored-only``, for example
``opp_update_fingerprint_test_results --load $INET_ROOT -p inet --no-stored-only -c '^MyConfig$' -r 0``.
``opp_update_fingerprint_test_results --no-stored-only -c '^MyConfig$' -r 0``.
The new entries take the ``sim-time-limit`` of the ini file.

Usually, when a simulation model is being developed, the fingerprint tests should
Expand Down Expand Up @@ -126,8 +130,8 @@ the affected runs. After the change, check them:

.. code-block:: shell

$ opp_update_fingerprint_test_results --load $INET_ROOT -p inet --no-stored-only -f <filter> -r 0 --ingredients plx
$ opp_run_fingerprint_tests --load $INET_ROOT -p inet -f <filter> --ingredients plx
$ opp_update_fingerprint_test_results --no-stored-only -f <filter> -r 0 --ingredients plx
$ opp_run_fingerprint_tests -f <filter> --ingredients plx

Another common example is when a submodule is renamed. Since the default fingerprint
ingredients contain the full module path of all events, this change will break
Expand All @@ -147,8 +151,8 @@ and the test after the change. ``--append-args`` passes the rest of the line to
.. code-block:: shell

# please note the quoting uses both ' and " in the right order
$ opp_update_fingerprint_test_results --load $INET_ROOT -p inet -f <filter> --append-args --fingerprint-modules='"not(fullPath(**.wlan[*].mac.**))"'
$ opp_run_fingerprint_tests --load $INET_ROOT -p inet -f <filter> --append-args --fingerprint-modules='"not(fullPath(**.wlan[*].mac.**))"'
$ opp_update_fingerprint_test_results -f <filter> --append-args --fingerprint-modules='"not(fullPath(**.wlan[*].mac.**))"'
$ opp_run_fingerprint_tests -f <filter> --append-args --fingerprint-modules='"not(fullPath(**.wlan[*].mac.**))"'

INET uses continuous integration on Github. Fingerprint and other tests are run
automatically for changes on the master and integration branches. Moreover, all
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,13 @@ INET no longer has its own fingerprint test tool. The baseline is the JSON
store ``tests/fingerprint/store.json``, and the commands of opp_repl run and
update it. Install opp_repl as the GitHub workflow does, with
``pip install "opp_repl @ git+https://github.com/omnetpp/opp_repl.git@github_ci"``, set up the OMNeT++
and INET environment, and run the commands from anywhere in the INET tree:
and INET environment, and run the commands in the INET tree, where they find INET's
:file:`inet.opp` by themselves:

.. code-block:: bash

opp_run_fingerprint_tests --load $INET_ROOT -p inet -m release -f wireless
opp_update_fingerprint_test_results --load $INET_ROOT -p inet -m release -f wireless
opp_run_fingerprint_tests -m release -f wireless
opp_update_fingerprint_test_results -m release -f wireless

The second command replaces the ``.UPDATED`` files of ``fingerprinttest``: it
writes the new values into the store, and ``git diff`` shows them. The command
Expand Down
4 changes: 2 additions & 2 deletions inet.opp
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ SimulationProject(
msg_folders=["src"],
# INET's description for opp_repl. It lives in the INET repository, so that a change of INET's
# test setup lands with the INET change; opp_repl bundles an older copy as a fallback for INET
# versions without this file. Load it with --load $INET_ROOT (or from the INET root, where
# opp_repl loads every *.opp file of the current folder).
# versions without this file. An opp_repl command finds this file without an option in the
# INET root and in each folder below it; elsewhere, load it with --load $INET_ROOT.
#
# Per-kind test configuration. Only INET-specific aspects are declared here;
# opp_repl's generic defaults (run everything, mode=debug, smoke/sanitizer
Expand Down
6 changes: 3 additions & 3 deletions tests/fingerprint/README
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ The ethernet-*.ini files of this folder are simulations that exist
only for the fingerprint test.

Run the tests and accept new values with the opp_repl commands, after
the OMNeT++ and INET setenv:
the OMNeT++ and INET setenv, anywhere in the INET tree:

opp_run_fingerprint_tests --load $INET_ROOT -p inet [-f <regex>]
opp_update_fingerprint_test_results --load $INET_ROOT -p inet [-f <regex>]
opp_run_fingerprint_tests [-f <regex>]
opp_update_fingerprint_test_results [-f <regex>]

doc/src/developers-guide/ch-testing.rst describes the options.

Expand Down
4 changes: 2 additions & 2 deletions tutorials/fingerprint/doc/accepting.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ First, run the fingerprint tests with the ``tplx`` ingredients:

.. code-block:: bash

$ opp_run_fingerprint_tests --load $INET_ROOT -p inet -w tutorials/fingerprint --ingredients tplx -c '^(Ethernet|Wifi)$'
$ opp_run_fingerprint_tests -w tutorials/fingerprint --ingredients tplx -c '^(Ethernet|Wifi)$'

.. code-block:: fp

Expand All @@ -25,7 +25,7 @@ The model is validated, so the new values are correct. Record them in the store:

.. code-block:: bash

$ opp_update_fingerprint_test_results --load $INET_ROOT -p inet -w tutorials/fingerprint --ingredients tplx -c '^(Ethernet|Wifi)$'
$ opp_update_fingerprint_test_results -w tutorials/fingerprint --ingredients tplx -c '^(Ethernet|Wifi)$'

.. code-block:: fp

Expand Down
4 changes: 2 additions & 2 deletions tutorials/fingerprint/doc/first.rst
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ Always give the project and the working directory filter. Use ``tplx`` as the in

.. code-block:: bash

$ opp_run_fingerprint_tests --load $INET_ROOT -p inet -w tutorials/fingerprint --ingredients tplx
$ opp_run_fingerprint_tests -w tutorials/fingerprint --ingredients tplx

.. code-block:: fp

Expand All @@ -205,7 +205,7 @@ For example, the `Changing a Timer` step needs the ``Wifi`` configuration only:

.. code-block:: bash

$ opp_run_fingerprint_tests --load $INET_ROOT -p inet -w tutorials/fingerprint --ingredients tplx -c '^Wifi$'
$ opp_run_fingerprint_tests -w tutorials/fingerprint --ingredients tplx -c '^Wifi$'

.. code-block:: fp

Expand Down
2 changes: 1 addition & 1 deletion tutorials/fingerprint/doc/refactoring.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,6 @@ The refactoring doesn't change the fingerprint because the code is functionally

.. code-block:: fp

$ opp_run_fingerprint_tests --load $INET_ROOT -p inet -w tutorials/fingerprint --ingredients tplx -c '^(Ethernet|Wifi)$'
$ opp_run_fingerprint_tests -w tutorials/fingerprint --ingredients tplx -c '^(Ethernet|Wifi)$'
tutorials/fingerprint -c Ethernet for 0.2s PASS
tutorials/fingerprint -c Wifi for 5s PASS
6 changes: 3 additions & 3 deletions tutorials/fingerprint/doc/renaming_module_parameter.rst
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ and the router doesn't forward packets. This change causes the fingerprint tests

.. code-block:: bash

$ opp_run_fingerprint_tests --load $INET_ROOT -p inet -w tutorials/fingerprint --ingredients tplx -c '^Ethernet$'
$ opp_run_fingerprint_tests -w tutorials/fingerprint --ingredients tplx -c '^Ethernet$'

.. code-block:: fp

Expand All @@ -58,7 +58,7 @@ To correct the model, the renaming needs to be followed everywhere. When we chan

.. code-block:: bash

$ opp_run_fingerprint_tests --load $INET_ROOT -p inet -w tutorials/fingerprint --ingredients tplx -c '^Ethernet$'
$ opp_run_fingerprint_tests -w tutorials/fingerprint --ingredients tplx -c '^Ethernet$'

.. code-block:: fp

Expand All @@ -77,7 +77,7 @@ When run the ``tplx`` fingerprint tests, they result in ERROR:

.. code-block:: bash

$ opp_run_fingerprint_tests --load $INET_ROOT -p inet -w tutorials/fingerprint --ingredients tplx -c '^Ethernet$'
$ opp_run_fingerprint_tests -w tutorials/fingerprint --ingredients tplx -c '^Ethernet$'

.. code-block:: fp

Expand Down
Loading