perf(lu): factor small square row-major matrices through the full-pivot dual (#314)

PATCHSET.md

@@ -121,7 +121,7 @@ gate has changed and two kinds of adoption are now in the ledger.
 | Empty complex tridiagonal EVD with eigenvectors | `tridiagonal_self_adjoint_evd` had no empty-input return, unlike `self_adjoint_evd`. For a complex type with eigenvectors requested, its phase loop wrote `scale[0]` into an empty buffer and panicked; real types and the values-only path did not reach that write. It now returns `Ok(())` for `n == 0`. The function is upstream faer code (sarah-quiñones); nothing was reported upstream. | KEEP | tensor4all/faer#31 | Correctness only. Found by the #31 LAPACK validation harness (`lapack_tridiagonal_evd`, 36 panicking calls per complex type). Test `empty_tridiagonal_with_vectors` (f64/c64/c32) fails before the change and passes after. Upstream report: [`upstream-reports/faer-021-tridiag-evd-empty-complex`](upstream-reports/faer-021-tridiag-evd-empty-complex/). |
 | Row-major QR layout adaptation | `qr_in_place` factors a matrix whose columns are not the contiguous direction through a column-major copy: the panel is not layout-aware (`make_householder_in_place` reads the pivot column and scales it, and the rank-one update beside it walked the block with `z!`), so a row-major matrix cost one cache line per element -- at `4096x256` f64 the panel was 25x its column-major cost and the whole deficit, and the factorization did not scale (47.8 / 42.9 / 43.0 ms at 1T / 4T / 8T). `qr_in_place` now copies into a column-major buffer, factors that, and copies the factor back, which is what LAPACK's row-major interface pays internally; the arithmetic is the copy's, so the factor is the column-major factor of the same matrix bit for bit. Tiled copy; with threads it is split along the destination's major dimension in whole 16-element tiles (columns into the buffer, rows on the write-back, so a thread writes whole lines of its own: tensor4all/faer#92), and the work buffer is allocated uninitialised and initialised by the threads that copy into it, so its pages are first touched in parallel rather than by a serial `vec![zero; m * n]` on the caller (`linalg::column_major_copy`, shared by the four row-major adaptations). One `m * n` heap buffer per call, because `qr_in_place_scratch` cannot know the strides. The copy has no cost gate yet (tensor4all/faer#92 part 2, after #91's dispatch measurements). Paired, best of 21, CPUs 4 / 4-7 / 4-11, fork `bfe4041` -> `044f947`, C order ms (before -> after, F order after): `4096x256` f64 47.83 / 42.85 / 42.96 -> 19.44 / 10.22 / 8.39 (15.77 / 8.37 / 7.23), `1024x64` f64 1.090 / 2.687 / 2.919 -> 0.587 / 0.613 / 0.678, `512x512` f64 9.44 / 9.56 / 12.73 -> 5.24 / 3.11 / 3.24, `256x256` f64 1.455 / 1.596 / 2.315 -> 0.921 / 0.833 / 0.904. Against the campaign's MKL C-order column (27.73 / 8.66 / 8.20) `4096x256` f64 goes from 1.72x / 4.95x / 5.24x behind to 0.70x / 1.18x / 1.02x. Test: `test_row_major_factor_is_the_column_major_one` -- the factor, `tau` and rank bit for bit at f64 and c64, tall/wide/square, sequential and threaded, backwards strides, and a zero leading column. A one-row matrix is left on the direct path (it eliminates at most one column). The `perf-warn` that told row-major callers to transpose is gone. | KEEP | tlinalg-rs#46 | Heap allocation inside a public call, the same boundary the fork's two row splits already cross. Own code: no OpenBLAS or MKL source is used or adapted. Exactly the adapted shapes (tall or square) are the column-major factorisation bit for bit; the guards' exclusions are not. Why, and what lost: `docs/worklogs/2026-10-15-qr-row-major-layout.md`. |
 | Row-major col-pivoting QR | `qr::col_pivoting::factor::qr_in_place` computes the norm of every one of the `n` columns before it starts and re-reduces them per step, and its incremental-norm SIMD path is gated on `A.row_stride() == 1` and unwraps a column-major view, so a row-major matrix pays one cache line per element on every shape. It now factors a column-major copy and writes the factor back, wide included. Paired, best of 21, CPUs 4 / 4-7 / 4-11, `f64`, C order before -> after (F order after): `256x256` 58.96 / 59.63 / 59.61 -> 1.52 / 1.40 / 1.19 (1.45 / 1.33 / 1.54), `1024x64` 22.23 / 22.29 / 22.45 -> 0.726 / 0.520 / 0.509 (0.472 / 0.522 / 0.536); `c64` `256x256` one thread 64.40 -> 3.29, `1024x64` 24.69 -> 1.64. So the C/F ratio at `256x256` one thread is 38.8x -> 1.05 and the C order is *faster* than the F order at eight threads. Nothing in the campaign records a column-pivoted QR, so this is the record's own driver. Test: `linalg::tests::test_row_major_adaptation_matches_the_column_major_one`. | KEEP | tlinalg-rs#46 | Heap allocation inside a public call, as for the no-pivoting row. Why, and what lost: `docs/worklogs/2026-10-15-row-major-adaptation-lu-colpiv.md`. |
-| Row-major LU, partial pivoting | `lu::partial_pivoting::factor::lu_in_place` walks each pivot column twice per step (search, then scaling) and swaps elements *inside* a column, which is a cache line per element when the columns are not contiguous. It now factors a column-major copy and writes the factor and its permutations back. Paired, best of 21, `f64`, C order before -> after (F order after): `512x512` 12.14 / 7.02 / 6.34 -> 3.37 / 2.01 / 1.89 (3.04 / 1.88 / 1.81) at 1T / 4T / 8T, `1024x1024` 59.80 / 27.62 / 22.93 -> 22.91 / 10.04 / 8.21 (19.89 / 8.19 / 6.65), `256x256` one thread 1.530 -> 0.486; `c64` `512x512` one thread 18.08 -> 9.50. Test: `linalg::tests::test_row_major_adaptation_matches_the_column_major_one` -- factor, permutations and transposition count bit for bit. The campaign's `lu` cell takes the composite, so `to_owned` below is what moves it. | KEEP | tlinalg-rs#46 | The full-pivoting flavour is deliberately **not** adapted: it already factors a row-major input through `mat.transpose_mut()`, and the copy measured a no-op there (512x512 f64 one thread 13.758 against 13.712 C order). Why: `docs/worklogs/2026-10-15-row-major-adaptation-lu-colpiv.md`. |
+| Row-major LU, partial pivoting | `lu::partial_pivoting::factor::lu_in_place` walks each pivot column twice per step (search, then scaling) and swaps elements *inside* a column, which is a cache line per element when the columns are not contiguous. It now factors a column-major copy and writes the factor and its permutations back. Paired, best of 21, `f64`, C order before -> after (F order after): `512x512` 12.14 / 7.02 / 6.34 -> 3.37 / 2.01 / 1.89 (3.04 / 1.88 / 1.81) at 1T / 4T / 8T, `1024x1024` 59.80 / 27.62 / 22.93 -> 22.91 / 10.04 / 8.21 (19.89 / 8.19 / 6.65), `256x256` one thread 1.530 -> 0.486; `c64` `512x512` one thread 18.08 -> 9.50. Test: `linalg::tests::test_row_major_adaptation_matches_the_column_major_one` -- factor, permutations and transposition count bit for bit. The campaign's `lu` cell takes the composite, so `to_owned` below is what moves it. | KEEP | tlinalg-rs#46 | The full-pivoting flavour is deliberately **not** adapted: it already factors a row-major input through `mat.transpose_mut()`, and the copy measured a no-op there (512x512 f64 one thread 13.758 against 13.712 C order); `10ee1c9` adapted it after all, behind the same `row_major_copy_pays` floor. Since tensor4all/faer#314 the square row-major matrices below 128 with contiguous rows (`col_stride == 1`) skip that copy and go through the transpose dual; generic strides keep the copy. Timed at 64, 80, 96, 112, 120 and 127 on Apple M5 Max (f64/c64, 1T/4T, `32c615f` against `b16d6b8`, per-repetition median of three: dual 0.89-1.02x a column-major input, the copy 0.98-1.34x) and at 64 and 96 on Zen 5 (0.94-1.04x); the sizes in between were not timed. From 128 up and on every tall shape the copy stays, because the dual's strided row scaling (and, tall, its strided long swap and wide trailing block) costs more than the copy. On a magnitude tie a row-major input that is not copied (this exclusion, but also wide and small ones) is searched transposed and takes the smallest row first, a column-major or copied one the smallest column, so tied pivots differ by layout (documented on `lu_in_place`, "Pivot ties"). The SIMD search's accumulator merge now keeps the smaller row on a tie within a column, for every layout (it kept whichever accumulator merged first; inherited from upstream). Guard `full_pivoting::factor::row_major_takes_the_copy`; tests `faer/tests/full_piv_lu_row_major_no_copy.rs` (no `m * n` allocation on the dual shapes, the copy still seen on 128x128, 256x32 and generic-stride 96x96; the tie cases) and `pivot_search_keeps_the_smaller_row_across_accumulators` and the full-pivoting arm of `test_row_major_adaptation_matches_the_column_major_one` (127x127 added). Why: `docs/worklogs/2026-11-09-full-piv-lu-row-major-dual.md`. Why: `docs/worklogs/2026-10-15-row-major-adaptation-lu-colpiv.md`. |
 | Row-major `to_owned` | `MatRef::to_owned` builds its result with `Mat::from_fn`, which fills column by column, so a row-major source is read with a stride of `ncols` and every element costs its own cache line: `512x512` f64 1.01 ms against 0.057, `1024x1024` 6.36 against 0.36, `4096x256` 5.08 against 0.37, `256x4096` 4.30 against 0.36. It now stages a tile when the source's columns are not contiguous and the source is already canonical. After: 0.275 / 2.007 / 2.045 / 1.774 ms. This is what the campaign's `lu` and `qr` cells pay -- a probe at `n = 512` f64 one thread C order reads `lu` 6.1417 -> 4.156 and 4.5329 -> 3.033 at eight threads, `qr` 12.5143 -> 11.185 -- and `full_piv_lu` 12.95 -> 12.63, `cholesky`, `eigh` and `svd_values` unchanged, because those three copy through their own routes rather than `to_owned`. | KEEP | tlinalg-rs#46 | The tile adds a `Mat::zeros` pass and is serial (`to_owned` has no `par`). Gated on `T::IS_CANONICAL`: `canonical()` reinterprets memory and does not apply `Conj::apply`, so a conjugated source (`A.adjoint().to_owned()`, which the sparse Cholesky tests use) keeps the elementwise path -- the ungated attempt broke five of them. The attempted generic `cloned` tile is **dropped**: initializing every destination entry by cloning the first source element amplifies heap-owned payloads by `m * n` before overwrite. `cloned` retains the original elementwise implementation; `test_cloned_does_not_replicate_the_first_heap_value` checks that each source is cloned once. The `Option<T>` tile still moves each cloned numeric value into `to_owned`/factorization destinations without a second clone. Why: `docs/worklogs/2026-10-15-row-major-adaptation-lu-colpiv.md`. |
 | Row-major `copy_from_triangular_lower` | `MatMut::copy_from_triangular_lower` (what `Llt::new` and `Ldlt::new` copy their input with) ran the elementwise zip, which for a row-major source into a column-major destination reads with a stride of the source's row length: `1024x1024` f64 one thread 4.16 ms against 0.30 ms from a column-major source. It now copies each 16-column band as its diagonal tile plus the rectangle under it through `copy_across_layouts_block`, the tile `to_owned` uses: 0.65 ms. The campaign's `cholesky` cell, 1024 f64 8T, three processes per arm against MKL in the same process: row-major 5.445 -> 2.236 ms (MKL 3.33 / 3.39), column-major 1.674 -> 1.649 (MKL 2.65 / 2.67). Tests: `linalg::tests::tiled_triangular_copy_writes_what_the_elementwise_one_does` and `linalg::tests::llt_and_partial_piv_lu_factor_both_orders_bit_for_bit` (the factor of the same matrix through the two orders, bit for bit). | KEEP | faer-benchmark slide re-take (1024, 8T) | Serial, like `to_owned`: the method has no `par`. A conjugated source keeps the elementwise path. `copy_from` (the full-matrix copy) still runs elementwise: 8.6 ms row-major against 0.47 ms at `1024x1024`; it is not on the `cholesky` or `lu` path. Why: `docs/worklogs/2026-11-06-row-major-triangular-copy.md`. |
 | Threaded rank-one Householder apply | The `N == 1` arm of `apply_block_householder_on_the_left_in_place_generic` is a sequential rank-one update per target column and it never read `par`. A tall blocked QR's panel is made almost entirely of those updates, so the panel was the serial part of it: at 4096x256 c64 the panel is 5.96 ms at 1T and 6.31 ms at 8T, against OpenBLAS's `?gemv`/`?geru` panel at 5.97 -> 2.58 ms on the same eight cores. When the caller asks for threads and the update is at least 8192 elements, the arm now falls through to the generic path, which partitions the target's columns and threads its two products. Through tlinalg's compact factor (`householder_factor`), pool row, 4T / 8T: 4096x256 f64 6.73 -> 5.77 and 6.67 -> 5.83 ms, 4096x256 c64 21.42 -> 18.75 and 19.50 -> 17.44 ms (11-14%), with 512x512, 128x128, `qr`, `svd_thin` and every 1T row unchanged within noise. Tests: `threaded_rank_one_matches_the_sequential_one` (f64 and c64; 2048x8, 16384x1, 64x8) and `rank_one_negates_its_own_reflector_column`. | KEEP | tlinalg-rs#46 | Own code: no OpenBLAS or MKL source is used or adapted, their timings are only the target. |