diff options
Diffstat (limited to 'travis/README.md')
| -rw-r--r-- | travis/README.md | 104 |
1 files changed, 104 insertions, 0 deletions
diff --git a/travis/README.md b/travis/README.md new file mode 100644 index 00000000..d5aee054 --- /dev/null +++ b/travis/README.md @@ -0,0 +1,104 @@ +Testing NASM +============ +We use [Travis CI](https://travis-ci.org/) service to execute NASM tests, +which basically prepares the environment and runs our `nasm-t.py` script. + +The script scans a testing directory for `*.json` test descriptor files +and runs test by descriptor content. + +Test engine +----------- +`nasm-t.py` script is a simple test engine written by Python3 language +which allows either execute a single test or run them all in a sequence. + +A typical test case processed by the following steps: + + - a test descriptor get parsed to figure out which arguments + are to be provided into the NASM command line; + - invoke the NASM with arguments; + - compare generated files with precompiled templates. + +`nasm-t.py` supports the following commands: + + - `list`: to list all test cases + - `run`: to run test cases + - `update`: to update precompiled templates + +Use `nasm-t.py -h` command to get the detailed description of every option. + +Test descriptor file +-------------------- +A descriptor file should provide enough information how to run the NASM +itself and which output files or streams to compare with predefined ones. +We use `JSON` format with the following fields: + + - `description`: a short description of a test which is shown to + a user when tests are listed; + - `id`: descriptor internal name to use with `ref` field; + - `ref`: a reference to `id` from where settings should be + copied, it is convenient when say only `option` is different + while the rest of the fields are the same; + - `format`: NASM output format to use (`bin`,`elf` and etc); + - `source`: is a source file name to compile, this file must + be shipped together with descriptor file itself; + - `option`: an additional option passed to the command line; + - `update`: a trigger to skip updating targets when running + an update procedure; + - `target`: an array of targets which the test engine should + check once compilation finished: + - `stderr`: a file containing *stderr* stream output to check; + - `stdout`: a file containing *stdout* stream output to check; + - `output`: a file containing compiled result to check, in other + words it is a name passed as `-o` option to the compiler. + +Examples +-------- + +A simple test where no additional options are used, simply compile +`absolute.asm` file with `bin` format for output, then compare +produced `absolute.bin` file with precompiled `absolute.bin.dest`. + +```json +{ + "description": "Check absolute addressing", + "format": "bin", + "source": "absolute.asm", + "target": [ + { "output": "absolute.bin" } + ] +} +``` + +A slightly complex example: compile one source file with different optimization +options and all results must be the same. To not write three descriptors +we assign `id` to the first one and use `ref` term to copy settings. +Also, it is expected that `stderr` stream will not be empty but carry some +warnings to compare. + +```json +[ + { + "description": "Check 64-bit addressing (-Ox)", + "id": "addr64x", + "format": "bin", + "source": "addr64x.asm", + "option": "-Ox", + "target": [ + { "output": "addr64x.bin" }, + { "stderr": "addr64x.stderr" } + ] + }, + { + "description": "Check 64-bit addressing (-O1)", + "ref": "addr64x", + "option": "-O1", + "update": "false" + }, + { + "description": "Check 64-bit addressing (-O0)", + "ref": "addr64x", + "option": "-O0", + "update": "false" + } +] +``` |