This document provides an overview of the test runner and how to integrate it with other stacks. So far we have it working with BoringSSL and some incomplete integrations with NSS and OpenSSL.
Note that supporting non-BoringSSL implementations is a work in progress and interfaces may change in the future. Consumers should pin to a particular revision rather than using BoringSSL’s master branch directly. As we gain experience with other implementations, we hope to make further improvements to portability, so please contact davidben@google.com and ekr@rtfm.com if implementing a new shim.
The test runner integrates with the TLS stack under test through a “shim”: a command line program which encapsulates the stack. By default, the shim points to the BoringSSL shim in the same source tree, but any program can be supplied via the -shim-path flag. The runner opens up a server socket and provides the shim with a -port argument that points to that socket. The shim always connects to the runner as a TCP client even when acting as a TLS server. For DTLS, there is a small framing layer that gives packet boundaries over TCP. The shim can also pass a variety of command line arguments which are used to configure the stack under test. These can be found at test_config.cc.
The shim reports success by exiting with a 0 error code and failure by reporting a non-zero error code and generally sending a textual error value to stderr. Many of the tests expect specific error string (such as NO_SHARED_CIPHER) that indicates what went wrong.
There are a number of situations in which the runner might succeed with some tests and not others:
We have implemented several features which allow implementations to ease these compatibility issues.
The runner can be supplied with a JSON configuration file which is intended to allow for a per-stack mapping. This file currently takes two directives:
DisabledTests: A JSON map consisting of the pattern matching the tests to be disabled as the key and some sort of reason why it was disabled as the value. The key is used as a match against the test name. The value is ignored and is just used for documentation purposes so you can remember why you disabled a test. -include-disabled overrides this filter.
ErrorMap: A JSON map from the internal errors the runner expects to the error strings that your implementation spits out. Generally you’ll need to map every error, but if you also provide the -loose-errors flag, then every un-mapped error just gets mapped to the empty string and treated as if it matched every error the runner expects.
The -shim-config flag is used to provide the config file.
If the shim encounters some request from the runner that it knows it can’t fulfill (e.g., a command line flag that it doesn’t recognize), then it can exit with the special code 89. Shims are recommended to use this exit code on unknown command-line arguments.
The test runner interprets this as “unimplemented” and skips the test. If run normally, this will cause the test runner to report that the entire test suite failed. The -allow-unimplemented flag suppresses this behavior and causes the test runner to ignore these tests for the purpose of evaluating the success or failure of the test suite.
The test runner can also be used to stress malloc failure codepaths. If passed -malloc-test=0, the runner will run each test repeatedly with an incrementing MALLOC_NUMBER_TO_FAIL environment variable. The shim should then replace the malloc implementation with one which fails at the specified number of calls. If there are not enough calls to reach the number, the shim should fail with exit code 88. This signals to the runner that the test has completed.
See crypto/test/malloc.cc for an example malloc implementation.
Note these tests are slow and will hit Go's test timeout. Pass -timeout 72h to avoid crashing after 10 minutes.
DYLD_LIBRARY_PATH=~/dev/nss-dev/nss-sandbox/dist/Darwin15.6.0_64_DBG.OBJ/lib go test -shim-path ~/dev/nss-dev/nss-sandbox/dist/Darwin15.6.0_64_DBG.OBJ/bin/nss_bogo_shim -loose-errors -allow-unimplemented -shim-config ~/dev/nss-dev/nss-sandbox/nss/external_tests/nss_bogo_shim/config.json