Skip to content

testing/libc/arch_libc: Test every overridable string function. - #3706

Open
Fishwaldo wants to merge 1 commit into
apache:masterfrom
Fishwaldo:upstream-arch-libc-string
Open

testing/libc/arch_libc: Test every overridable string function.#3706
Fishwaldo wants to merge 1 commit into
apache:masterfrom
Fishwaldo:upstream-arch-libc-string

Conversation

@Fishwaldo

Copy link
Copy Markdown
Contributor

Summary

  • testing/libc/arch_libc covered strcpy. A machine directory can
    override fourteen string and memory functions, and the mistakes such
    overrides actually make are specific: the word loop that works aligned
    and corrupts otherwise, the copy that runs past a guard byte, the overlap
    memmove must survive in both directions, the terminator a length cap
    was supposed to beat, the sign of a comparison whose bytes wrap.
  • The new suite drives every function through every combination of source
    and destination offset within a register, twenty one sizes chosen to
    straddle every internal boundary, overlap at every distance up to a
    register width each way, terminators probed for by name, caps placed
    before, at and past differences, and guard bytes around every
    destination. A speed report follows, one line per scenario, measured
    through CLOCK_MONOTONIC.
  • Two ways such a benchmark can quietly measure nothing are designed out,
    both learned by measuring nothing: the compiler will hoist a pure call
    whose arguments never change, so the pointers are laundered through an
    asm that claims to change them without disturbing the alignment under
    test; and an earlier scenario's writes can leave the compare buffers
    unequal, so equality is verified before the compare timings and the one
    scenario that needs shifted content prepares it explicitly after every
    clean reader has run.
  • The suite is a separate source file behind its own option, on by default,
    and the program's exit status now says whether anything failed. The
    existing strcpy test is untouched, including its perf_gettime
    dependency, which configurations without hardware performance counters
    will want switched off.
  • It is arch-neutral: it exercises whatever the machine directory provides,
    and passes identically against the generic C library when nothing is
    overridden.
  • Why now: this was written to validate a RISC-V machine-directory
    optimisation series that adds ten functions and reworks two. That series
    is a separate PR against apache/nuttx, which I will link here in a
    comment as soon as it is open. This suite does not depend on it and
    stands on its own; the RISC-V work depends on this suite for its evidence,
    which is why it comes first.

Impact

  • Is new feature added? Is existing feature changed? NEW, additive.
    A new test source behind CONFIG_TESTING_ARCH_LIBC_STRING, default on.
    The existing strcpy test is unchanged.
  • Impact on user? NO, beyond a larger test binary when the option is
    enabled. The program's exit status now reflects failure, which is a
    change only for anyone scripting the old always-zero behaviour.
  • Impact on build? NO. One new object when the option is on.
  • Impact on hardware? NO. Portable C; no arch-specific code.
  • Impact on documentation? NO.
  • Impact on security? NO.
  • Impact on compatibility? NO.
  • Anything else? The suite is deliberately unkind. It found real faults in
    the RISC-V work it was written for, and it detects deliberately injected
    corruption, shown below.

Testing

I confirm that changes are verified on local setup and works as intended:

  • Build Host: macOS 26.5.1, arm64 (Apple Silicon), xPack riscv-none-elf-gcc
    15.2.0
  • Targets: RISC-V rv64 on real hardware (ESWIN EIC7700X EVB, 1.4 GHz,
    downstream board port not yet upstream) and RISC-V rv32 under QEMU

Run against the generic C library, with no machine-directory overrides, on
rv64 hardware:

########## ROUND 1 final: BASELINE (EVB rv64 generic) ##########
== correctness ==
  memcpy     correctness: ok (0 bad)
  memmove    correctness: ok (0 bad)
  mv-overlap correctness: ok (0 bad)
  memset     correctness: ok (0 bad)
  memcmp     correctness: ok (0 bad)
  str-scan   correctness: ok (0 bad)
  strcmp-fam correctness: ok (0 bad)
== fails: 0 ==

Run against a fully overridden machine directory on the same rv64 hardware,
with the speed report and its repetition counts:

########## ROUND 2 (final harness): OPTIMIZED rv64 ##########
== correctness ==
  memcpy     correctness: ok (0 bad)
  memmove    correctness: ok (0 bad)
  mv-overlap correctness: ok (0 bad)
  memset     correctness: ok (0 bad)
  memcmp     correctness: ok (0 bad)
  str-scan   correctness: ok (0 bad)
  strcmp-fam correctness: ok (0 bad)
  [compare buffers equal]
  memcmp 64K aligned             342.6 MB/s  (1376 reps, 0.251s)
  memcmp 64K s+1/d+1             337.3 MB/s  (1360 reps, 0.252s)
  strncmp 4K aligned             244.8 MB/s  (15664 reps, 0.250s)
  strncmp 4K both+2              241.0 MB/s  (15424 reps, 0.250s)
  strlen 4K aligned             3274.3 MB/s  (209552 reps, 0.250s)
  strlen 4K +3                  3240.5 MB/s  (207392 reps, 0.250s)
  memchr 4K aligned             2496.8 MB/s  (159792 reps, 0.250s)
  strchr 4K absent              2281.0 MB/s  (145984 reps, 0.250s)
  strchr 4K absent +5           2285.0 MB/s  (146240 reps, 0.250s)
  strrchr 4K                     700.3 MB/s  (44816 reps, 0.250s)
  memcmp 64K s+1/d+2              41.7 MB/s  (176 reps, 0.264s)
  memcpy 64K aligned            3020.0 MB/s  (12080 reps, 0.250s)
  memcpy 64K src+1              2584.0 MB/s  (10336 reps, 0.250s)
  memmove-bk 64K                3464.0 MB/s  (13856 reps, 0.250s)
  strcpy 4K aligned             1975.3 MB/s  (126416 reps, 0.250s)
  strcpy 4K both+1              1822.5 MB/s  (116640 reps, 0.250s)
== fails: 0 ==

The same suite on rv32 under QEMU, including strlcpy:

rv32 built
== string correctness ==
  memcpy     correctness: ok (0 bad)
  memmove    correctness: ok (0 bad)
  mv-overlap correctness: ok (0 bad)
  memset     correctness: ok (0 bad)
  memcmp     correctness: ok (0 bad)
  str-scan   correctness: ok (0 bad)
  strcmp-fam correctness: ok (0 bad)
  strlcpy    correctness: ok (0 bad)
== string fails: 0 ==

A passing suite proves little on its own, so the detection was verified by
injecting a single-bit fault into an optimized memcpy (xori t1, t1, 1
added to its inner loop) and confirming the suite catches it:

	or		t1, t1, t2
	xori		t1, t1, 1
== correctness ==
  memcpy  correctness: FAIL (576 bad)
  memset  correctness: ok (0 bad)
  strcmp  correctness: ok (0 bad)
== fails: 576 ==

576 detections from one flipped bit, with the neighbouring functions
correctly reporting clean. That last capture is from an earlier revision of
the harness, which covered memcpy, memset and strcmp before the suite
was widened to the full set; it is included as evidence that the checks
detect corruption rather than as a run of the final code.

PR verification Self-Check

  • This PR introduces only one functional change.
  • I have updated all required description fields above.
  • My PR adheres to Contributing Guidelines and Documentation.
  • My PR is still work in progress (not ready for review).
  • My PR is ready for review and can be safely merged into a codebase.

Claude (claude-opus-5) assisted with authoring this test suite, its code
comments and this PR description. The commit carries an Assisted-by: tag
per CONTRIBUTING.md §1.5.

@Fishwaldo

Copy link
Copy Markdown
Contributor Author

The RISC-V series this suite was written for is now open as
apache/nuttx#19735. Every correctness claim in that PR is this suite's output,
on rv64 hardware and on rv32 under QEMU.

This PR remains independent of it: the suite is arch-neutral, passes against
the generic C library when nothing is overridden, and is useful to any machine
directory that overrides these functions.

This test covered strcpy.  A machine directory can override fourteen
string and memory functions, and the mistakes such overrides actually
make are specific: the word loop that works aligned and corrupts
otherwise, the copy that runs past a guard byte, the overlap memmove
must survive in both directions, the terminator that a length cap was
supposed to beat, the sign of a comparison whose bytes wrap.

So the new suite drives every function through every combination of
source and destination offset within a register, twenty one sizes
chosen to straddle every internal boundary, overlap at every distance
up to a register width each way, terminators probed for by name, caps
placed before, at and past differences, and guard bytes around every
destination.  A speed report follows, one line per scenario, measured
through CLOCK_MONOTONIC.

Two ways such a benchmark can quietly measure nothing are designed
out, both learned by measuring nothing: the compiler will hoist a pure
call whose arguments never change, so the pointers are laundered
through an asm that claims to change them without disturbing the
alignment under test; and an earlier scenario's writes can leave the
compare buffers unequal, so equality is verified before the compare
timings and the one scenario that needs shifted content prepares it
explicitly after every clean reader has run.

The suite is a separate source file behind its own option, on by
default, and the program's exit status now says whether anything
failed.  The existing strcpy test is untouched, including its
perf_gettime dependency, which configurations without hardware perf
counters will want switched off.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Justin Hammond <justin@dynam.ac>
@Fishwaldo
Fishwaldo force-pushed the upstream-arch-libc-string branch from 542c585 to 555e43f Compare August 9, 2026 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant