Fix 20-year-old typo in BN_mask_bits.
This clearly was supposed to be a return 1. See
https://github.com/openssl/openssl/issues/5537 for details.
(Additionally, now that our BIGNUMs may be non-minimal, this function
violates the rule that BIGNUM functions should not depend on widths. We
should use w >= bn_minimal_width(a) to retain the original behavior. But
the original behavior is nuts, so let's just fix it.)
Update-Note: BN_mask_bits no longer reports failure in some cases. These
cases were platform-dependent and not useful, and code search confirms
nothing was relying on it.
Change-Id: I31b1c2de6c5de9432c17ec3c714a5626594ee03c
Reviewed-on: https://boringssl-review.googlesource.com/26464
Commit-Queue: Steven Valdez <svaldez@google.com>
Reviewed-by: Steven Valdez <svaldez@google.com>
CQ-Verified: CQ bot account: commit-bot@chromium.org <commit-bot@chromium.org>
diff --git a/crypto/fipsmodule/bn/shift.c b/crypto/fipsmodule/bn/shift.c
index 7a49028..d8dfe5f 100644
--- a/crypto/fipsmodule/bn/shift.c
+++ b/crypto/fipsmodule/bn/shift.c
@@ -292,7 +292,7 @@
int w = n / BN_BITS2;
int b = n % BN_BITS2;
if (w >= a->width) {
- return 0;
+ return 1;
}
if (b == 0) {
a->width = w;
diff --git a/include/openssl/bn.h b/include/openssl/bn.h
index 14ea79e..0a844ed 100644
--- a/include/openssl/bn.h
+++ b/include/openssl/bn.h
@@ -494,7 +494,12 @@
OPENSSL_EXPORT int BN_is_bit_set(const BIGNUM *a, int n);
// BN_mask_bits truncates |a| so that it is only |n| bits long. It returns one
-// on success or zero if |n| is greater than the length of |a| already.
+// on success or zero if |n| is negative.
+//
+// This differs from OpenSSL which additionally returns zero if |a|'s word
+// length is less than or equal to |n|, rounded down to a number of words. Note
+// word size is platform-dependent, so this behavior is also difficult to rely
+// on in OpenSSL and not very useful.
OPENSSL_EXPORT int BN_mask_bits(BIGNUM *a, int n);
// BN_count_low_zero_bits returns the number of low-order zero bits in |bn|, or
