tree 4bada9568d3ef750e5f63fc0c92316524e695957
parent 9d48902108d2ed0b04ea862d7d3aa8bf14e17f47
author David Benjamin <davidben@google.com> 1686097018 -0400
committer Boringssl LUCI CQ <boringssl-scoped@luci-project-accounts.iam.gserviceaccount.com> 1686159566 +0000

Give up on qsort for sk_FOO_sort

OpenSSL's API constraints are such that sk_FOO_sort must take a
comparison function of type
int (*cmp)(const FOO *const *a, const FOO *const *b)

However qsort expects a comparison function of type
int (*cmp)(const void *a, const void *b)

In C, it is UB to cast a function pointer to a different type and call
it, even if the underlying calling conventions are the same. Moreover,
as qsort doesn't take a context parameter on its comparisons, we cannot
do the usual convention with closures in C.

qsort_r and qsort_s would avoid this, but they are unusable. Too many
libcs don't have them, and those that do define them inconsistently. See
https://stackoverflow.com/a/39561369

It seems this UB has finally hit a sanitizer in fxbug.dev/128274.
Irritating as it is to not even have a working sort function, I think
the easiest option is to just give up on qsort. As we did with bsearch
in https://boringssl-review.googlesource.com/c/boringssl/+/35304, just
implement an in-place heap sort ourselves.

Bug: fxbug.dev/128274
Change-Id: I9de6b4018bf635da0d0c5a680bd7811d297b0bb3
Reviewed-on: https://boringssl-review.googlesource.com/c/boringssl/+/60507
Commit-Queue: David Benjamin <davidben@google.com>
Reviewed-by: Bob Beck <bbe@google.com>