Add a script to refresh fuzzer corpora.
This is getting to be a nuisance to do by hand.
Change-Id: If3aa7c666c4cc0bf97615564f258ff740a561766
Reviewed-on: https://boringssl-review.googlesource.com/12281
Reviewed-by: David Benjamin <davidben@google.com>
diff --git a/FUZZING.md b/FUZZING.md
index 7f83a7d..f004c27 100644
--- a/FUZZING.md
+++ b/FUZZING.md
@@ -71,25 +71,13 @@
## TLS transcripts
-The `client` and `server` corpora are seeded from the test suite. The test suite has a `-fuzzer` flag which mirrors the fuzzer mode changes above and a `-deterministic` flag which removes all non-determinism on the Go side. Not all tests pass, so `ssl/test/runner/fuzzer_mode.json` contains the necessary suppressions. To run the tests against a fuzzer-mode `bssl_shim`, run:
+The `client` and `server` corpora are seeded from the test suite. The test suite has a `-fuzzer` flag which mirrors the fuzzer mode changes above and a `-deterministic` flag which removes all non-determinism on the Go side. Not all tests pass, so `ssl/test/runner/fuzzer_mode.json` contains the necessary suppressions. The `run_tests` target will pass appropriate command-line flags.
+
+There are separate corpora, `client_corpus_no_fuzzer_mode` and `server_corpus_no_fuzzer_mode`. These are transcripts for fuzzers with only `BORINGSSL_UNSAFE_DETERMINISTIC_MODE` defined. To build in this mode, pass `-DNO_FUZZER_MODE=1` into CMake. This configuration is run in the same way but without `-fuzzer` and `-shim-path` flags.
+
+If both sets of tests pass, refresh the fuzzer corpora with `refresh_ssl_corpora.sh`:
```
-cd ssl/test/runner
-go test -fuzzer -deterministic -shim-config fuzzer_mode.json
+cd fuzz
+./refresh_fuzzer_corpora.sh /path/to/fuzzer/mode/build /path/to/non/fuzzer/mode/build
```
-
-For a different build directory from `build/`, pass the appropriate `-shim-path` flag. If those tests pass, record a set of transcripts with:
-
-```
-go test -fuzzer -deterministic -transcript-dir /tmp/transcripts/
-```
-
-Note the suppressions file is ignored so disabled tests record transcripts too. Then merge into the existing corpora:
-
-```
-cd build/
-./fuzz/client -max_len=50000 -merge=1 ../fuzz/client_corpus /tmp/transcripts/tls/client
-./fuzz/server -max_len=50000 -merge=1 ../fuzz/server_corpus /tmp/transcripts/tls/server
-```
-
-There are separate corpora, `fuzz/client_corpus_no_fuzzer_mode` and `fuzz/server_corpus_no_fuzzer_mode`. These are transcripts for fuzzers with only `BORINGSSL_UNSAFE_DETERMINISTIC_MODE` defined. To build in this mode, pass `-DNO_FUZZER_MODE=1` into CMake. These corpora are updated the same way, but without the `-fuzzer` and `-shim-config` flags.
diff --git a/fuzz/refresh_ssl_corpora.sh b/fuzz/refresh_ssl_corpora.sh
new file mode 100755
index 0000000..bded442
--- /dev/null
+++ b/fuzz/refresh_ssl_corpora.sh
@@ -0,0 +1,112 @@
+#!/bin/bash
+# Copyright (c) 2016, Google Inc.
+#
+# Permission to use, copy, modify, and/or distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
+# SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
+# OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+set -ex
+
+if [[ $# -ne 2 ]]; then
+ echo "Usage: $0 fuzzer_mode_build_dir no_fuzzer_mode_build_dir"
+ exit 1
+fi
+
+fuzzer_mode_build_dir=$1
+no_fuzzer_mode_build_dir=$2
+
+
+# Sanity-check the build directories.
+
+if ! grep -q '^FUZZ:' "$fuzzer_mode_build_dir/CMakeCache.txt"; then
+ echo "$fuzzer_mode_build_dir was not built with -DFUZZ=1"
+ exit 1
+fi
+
+if grep -q '^NO_FUZZER_MODE:' "$fuzzer_mode_build_dir/CMakeCache.txt"; then
+ echo "$fuzzer_mode_build_dir was built with -DNO_FUZZER_MODE=1"
+ exit 1
+fi
+
+if ! grep -q '^FUZZ:' "$no_fuzzer_mode_build_dir/CMakeCache.txt"; then
+ echo "$no_fuzzer_mode_build_dir was not built with -DFUZZ=1"
+ exit 1
+fi
+
+if ! grep -q '^NO_FUZZER_MODE:' "$no_fuzzer_mode_build_dir/CMakeCache.txt"; then
+ echo "$no_fuzzer_mode_build_dir was not built with -DNO_FUZZER_MODE=1"
+ exit 1
+fi
+
+
+# Sanity-check the current working directory.
+
+assert_directory() {
+ if [[ ! -d $1 ]]; then
+ echo "$1 not found."
+ exit 1
+ fi
+}
+
+assert_directory client_corpus
+assert_directory client_corpus_no_fuzzer_mode
+assert_directory server_corpus
+assert_directory server_corpus_no_fuzzer_mode
+
+
+# Gather new transcripts. Ignore errors in running the tests.
+
+fuzzer_mode_shim=$(readlink -f "$fuzzer_mode_build_dir/ssl/test/bssl_shim")
+no_fuzzer_mode_shim=$(readlink -f \
+ "$no_fuzzer_mode_build_dir/ssl/test/bssl_shim")
+
+fuzzer_mode_transcripts=$(mktemp -d '/tmp/boringssl-transcript.XXXXXX')
+no_fuzzer_mode_transcripts=$(mktemp -d '/tmp/boringssl-transcript.XXXXXX')
+
+echo Recording fuzzer-mode transcripts
+(cd ../ssl/test/runner/ && go test \
+ -shim-path "$fuzzer_mode_shim" \
+ -transcript-dir "$fuzzer_mode_transcripts" \
+ -fuzzer \
+ -deterministic) || true
+
+echo Recording non-fuzzer-mode transcripts
+(cd ../ssl/test/runner/ && go test \
+ -shim-path "$no_fuzzer_mode_shim" \
+ -transcript-dir "$no_fuzzer_mode_transcripts" \
+ -deterministic) || true
+
+
+# Minimize the existing corpora.
+
+minimize_corpus() {
+ local fuzzer="$1"
+ local corpus="$2"
+
+ echo "Minimizing ${corpus}"
+ mv "$corpus" "${corpus}_old"
+ mkdir "$corpus"
+ "$fuzzer" -max_len=50000 -merge=1 "$corpus" "${corpus}_old"
+ rm -Rf "${corpus}_old"
+}
+
+minimize_corpus "$fuzzer_mode_build_dir/fuzz/client" client_corpus
+minimize_corpus "$fuzzer_mode_build_dir/fuzz/server" server_corpus
+minimize_corpus "$no_fuzzer_mode_build_dir/fuzz/client" client_corpus_no_fuzzer_mode
+minimize_corpus "$no_fuzzer_mode_build_dir/fuzz/server" server_corpus_no_fuzzer_mode
+
+
+# Incorporate the new transcripts.
+
+"$fuzzer_mode_build_dir/fuzz/client" -max_len=50000 -merge=1 client_corpus "${fuzzer_mode_transcripts}/tls/client"
+"$fuzzer_mode_build_dir/fuzz/server" -max_len=50000 -merge=1 server_corpus "${fuzzer_mode_transcripts}/tls/server"
+"$no_fuzzer_mode_build_dir/fuzz/client" -max_len=50000 -merge=1 client_corpus_no_fuzzer_mode "${no_fuzzer_mode_transcripts}/tls/client"
+"$no_fuzzer_mode_build_dir/fuzz/server" -max_len=50000 -merge=1 server_corpus_no_fuzzer_mode "${no_fuzzer_mode_transcripts}/tls/server"
