Skip to content

Handle null context and algorithm names without crashing the JVM - #37

Merged
dstebila merged 1 commit into
masterfrom
ds-fix-null-arguments
Nov 3, 2026
Merged

dstebila merged 1 commit into
masterfrom
ds-fix-null-arguments

Conversation

@dstebila

@dstebila dstebila commented Nov 2, 2026

Copy link
Copy Markdown
Member

Passing null to several public APIs crashed the JVM with a SIGSEGV. Java exception handlers can't catch the crash, so it ends the whole process.

  • Null context. Signature.sign(message, ctx) and Signature.verify(message, signature, ctx, public_key) already treat a null ctx as length 0, but the JNI code passed the null array to GetByteArrayElements. This affects every release since context support was added in 0.2.0.
  • Null algorithm name. new Signature(null), new KeyEncapsulation(null), Sigs.is_sig_enabled(null), KEMs.is_KEM_enabled(null), Rand.randombytes_switch_algorithm(null), Signature.create_sig_new(null) and KeyEncapsulation.create_KEM_new(null) passed the null string to GetStringUTFChars. This affects all releases.

This PR also fixes a related problem when signing fails. The signature length was uninitialized, so a failure such as an ML-DSA context longer than 255 bytes surfaced as an ArrayIndexOutOfBoundsException with a garbage length, instead of RuntimeException("Cannot sign message").

Changes

  • A null context is passed to liboqs as NULL with length 0, which means no context. A signature made with a null context verifies with the plain verify. The Javadoc now says that null means no context. Empty arrays behave as before.
  • A null algorithm name is treated as not enabled. The constructors throw MechanismNotSupportedError, Rand.randombytes_switch_algorithm throws its usual RuntimeException, and create_sig_new(null) and create_KEM_new(null) leave the existing mechanism in place.
  • Both signing functions initialize the signature length to 0, so a failed signing operation throws "Cannot sign message".

Testing

There are 7 new tests in SigTest, KEMTest and a new RandTest. Before the fix, 6 of them crashed the JVM and the seventh failed on the exception message.

Tested locally with mvn package on macOS (arm64, JDK 27) against liboqs main (8c41c85), where all the new tests pass.

Until #36 is merged, three existing testAllSigs cases fail on both this branch and master. They are ML-DSA-44-extmu, ML-DSA-65-extmu and ML-DSA-87-extmu, which liboqs signs only when the message is the 64-byte μ, while the test signs a 27-byte message. On master that failure shows up as the ArrayIndexOutOfBoundsException with a garbage length described above. With this PR it becomes RuntimeException: Cannot sign message. Merged with #36, mvn package passes with no errors.

Reported by Zhang Jianhui (Jack), @mingmingzhizhongcz.

🤖 Generated with Claude Code

Passing null as the context to Signature.sign(message, ctx) or
Signature.verify(message, signature, ctx, public_key) crashed the JVM,
because the JNI code passed the null array to GetByteArrayElements. A
null algorithm name crashed the same way in GetStringUTFChars. Pass a
null context to liboqs as NULL, and treat a null algorithm name as not
enabled.

Also initialize the signature length, so that a failed signing
operation throws "Cannot sign message" instead of an exception built
from an uninitialized length.

Signed-off-by: Douglas Stebila <dstebila@uwaterloo.ca>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jimouris jimouris left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - nice!

Looking at the JNI C code reminded me that we wrote this pre AI :)

You mentioned it in the description but are all CI failures fixed by #36? (Might be best to stack these)

@dstebila
dstebila merged commit 008baa6 into master Nov 3, 2026
6 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants