ssl/test/PORTING.md

# Porting to Other Implementations

## Introduction

This document provides an overview of the test runner and how to
integrate it with other stacks.  So far we have it working with
BoringSSL and some incomplete integrations with NSS and OpenSSL.

Note that supporting non-BoringSSL implementations is a work in
progress and interfaces may change in the future. Consumers should pin
to a particular revision rather than using BoringSSL’s master branch
directly. As we gain experience with other implementations, we hope to
make further improvements to portability, so please contact
davidben@google.com and ekr@rtfm.com if implementing a new shim.


## Integration Architecture

The test runner integrates with the TLS stack under test through a
“shim”: a command line program which encapsulates the stack. By
default, the shim points to the BoringSSL shim in the same source
tree, but any program can be supplied via the `-shim-path` flag. The
runner opens up a server socket and provides the shim with a `-port`
argument that points to that socket. The shim always connects to the
runner as a TCP client even when acting as a TLS server. For DTLS,
there is a small framing layer that gives packet boundaries over
TCP. The shim can also pass a variety of command line arguments which
are used to configure the stack under test. These can be found at
`test_config.cc`.


The shim reports success by exiting with a `0` error code and failure by
reporting a non-zero error code and generally sending a textual error
value to stderr. Many of the tests expect specific error string (such
as `NO_SHARED_CIPHER`) that indicates what went wrong.


## Compatibility Issues

There are a number of situations in which the runner might succeed
with some tests and not others:

* Defects in the stack under test
* Features which haven’t yet been implemented
* Failure to implement one or more of the command line flags the runner uses with the shim
* Disagreement about the right behavior/interpretation of the spec


We have implemented several features which allow implementations to ease these compatibility issues.

### Configuration File

The runner can be supplied with a JSON configuration file which is
intended to allow for a per-stack mapping. This file currently takes
two directives:


* `DisabledTests`: A JSON map consisting of the pattern matching the
  tests to be disabled as the key and some sort of reason why it was
  disabled as the value. The key is used as a match against the test
  name. The value is ignored and is just used for documentation
  purposes so you can remember why you disabled a
  test. `-include-disabled` overrides this filter.

* `ErrorMap`: A JSON map from the internal errors the runner expects to
  the error strings that your implementation spits out. Generally
  you’ll need to map every error, but if you also provide the
 ` -loose-errors` flag, then every un-mapped error just gets mapped to
  the empty string and treated as if it matched every error the runner
  expects.


The `-shim-config` flag is used to provide the config file.


### Unimplemented Features
If the shim encounters some request from the runner that it knows it
can’t fulfill (e.g., a command line flag that it doesn’t recognize),
then it can exit with the special code `89`. Shims are recommended to
use this exit code on unknown command-line arguments.

The test runner interprets this as “unimplemented” and skips the
test. If run normally, this will cause the test runner to report that
the entire test suite failed. The `-allow-unimplemented` flag suppresses
this behavior and causes the test runner to ignore these tests for the
purpose of evaluating the success or failure of the test suite.


### Malloc Tests

The test runner can also be used to stress malloc failure
codepaths. If passed `-malloc-test=0`, the runner will run each test
repeatedly with an incrementing `MALLOC_NUMBER_TO_FAIL` environment
variable. The shim should then replace the malloc implementation with
one which fails at the specified number of calls. If there are not
enough calls to reach the number, the shim should fail with exit code
`88`. This signals to the runner that the test has completed.

See `crypto/test/malloc.cc` for an example malloc implementation.

Note these tests are slow and will hit Go's test timeout. Pass `-timeout 72h` to
avoid crashing after 10 minutes.


## Example: Running Against NSS

```
DYLD_LIBRARY_PATH=~/dev/nss-dev/nss-sandbox/dist/Darwin15.6.0_64_DBG.OBJ/lib go test -shim-path ~/dev/nss-dev/nss-sandbox/dist/Darwin15.6.0_64_DBG.OBJ/bin/nss_bogo_shim -loose-errors -allow-unimplemented -shim-config ~/dev/nss-dev/nss-sandbox/nss/external_tests/nss_bogo_shim/config.json
```