path: root/travis
diff options
authorCyrill Gorcunov <gorcunov@gmail.com>2018-12-19 18:57:05 +0300
committerCyrill Gorcunov <gorcunov@gmail.com>2018-12-20 14:28:31 +0300
commit3c0b924c9d031cee8f3f9da7bb2405bf65ef30bb (patch)
tree4373b607e2f0eb11d8f6f2e29a426f3758bb6f28 /travis
parent296515468447185af1f310cb2d3a7a8b06996b61 (diff)
test: nasm-t -- Add more details into README
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
Diffstat (limited to 'travis')
1 files changed, 69 insertions, 7 deletions
diff --git a/travis/README.md b/travis/README.md
index d5aee054..d8f1c4dd 100644
--- a/travis/README.md
+++ b/travis/README.md
@@ -26,14 +26,50 @@ A typical test case processed by the following steps:
Use `nasm-t.py -h` command to get the detailed description of every option.
+### Test unit structure
+Each test consists at least of three files:
+ - a test descriptor in with `*.json` extension;
+ - a source file to compile;
+ - a target file to compare result with, it is assumed to have
+ the same name as output generated during the pass file but with `*.t`
+ extension; thus if a test generates `*.bin` file the appropriate target
+ should have `*.bin.t` name.
+Running tests
+To run all currently available tests simply type the following
+python3 travis/nasm-t.py run
+By default the `nasm-t.py` scans `test` subdirectory for `*.json` files and
+consider each as a test descriptor. Then every test is executed sequentially.
+If the descriptor can not be parsed it silently ignored.
+To run a particular test provide the test name, for example
+python3 travis/nasm-t.py list
+./travis/test/utf Test __utf__ helpers
+./travis/test/utf Test errors in __utf__ helpers
+python3 travis/nasm-t.py run -t ./travis/test/utf
+Test name duplicates in the listing above means that the descriptor
+carries several tests with same name but different options.
Test descriptor file
A descriptor file should provide enough information how to run the NASM
itself and which output files or streams to compare with predefined ones.
-We use `JSON` format with the following fields:
+We use *JSON* format with the following fields:
- `description`: a short description of a test which is shown to
- a user when tests are listed;
+ a user when tests are being listed;
- `id`: descriptor internal name to use with `ref` field;
- `ref`: a reference to `id` from where settings should be
copied, it is convenient when say only `option` is different
@@ -49,14 +85,14 @@ We use `JSON` format with the following fields:
- `stderr`: a file containing *stderr* stream output to check;
- `stdout`: a file containing *stdout* stream output to check;
- `output`: a file containing compiled result to check, in other
- words it is a name passed as `-o` option to the compiler.
+ words it is a name passed as `-o` option to the compiler;
+ - `error`: an error handler, can be either *over* to ignore any
+ error happened, or *expected* to make sure the test is failing.
+### Examples
A simple test where no additional options are used, simply compile
`absolute.asm` file with `bin` format for output, then compare
-produced `absolute.bin` file with precompiled `absolute.bin.dest`.
+produced `absolute.bin` file with precompiled `absolute.bin.t`.
@@ -69,6 +105,10 @@ produced `absolute.bin` file with precompiled `absolute.bin.dest`.
+Note the `output` target is named as *absolute.bin* where *absolute.bin.t*
+should be already precompiled (we will talk about it in `update` action)
+and present on disk.
A slightly complex example: compile one source file with different optimization
options and all results must be the same. To not write three descriptors
we assign `id` to the first one and use `ref` term to copy settings.
@@ -102,3 +142,25 @@ warnings to compare.
+Updating tests
+If during development some of the targets are expected to change
+the tests will start to fail so the should be updated. Thus new
+precompiled results will be treated as templates to compare with.
+To update all tests in one pass run
+python3 travis/nasm-t.py update
+=== Updating ./travis/test/xcrypt ===
+ Processing ./travis/test/xcrypt
+ Executing ./nasm -f bin -o ./travis/test/xcrypt.bin ./travis/test/xcrypt.asm
+ Moving ./travis/test/xcrypt.bin to ./travis/test/xcrypt.bin.t
+=== Test ./travis/test/xcrypt UPDATED ===
+and commit the results. To update a particular test provide its name
+with `-t` option.