[kriboe90]

Re-cut of #157 onto current master, which it had drifted eleven conflicting files away from. The finished half of that branch is already in master as #158, so this holds only the open half, with dark_img_indices and the reader without a caller gone for good.

What is in here

1. The RAW data are read without the clipping at zero as well (3fff826)

The channel data are clipped at zero after the black level subtraction, which turns the negative half of the read noise into a positive offset of the integrated intensity of every LED. The offset stays while the LED dims in smoke, so it saturates the extinction coefficients in dense smoke, where it is 0.3 to 0.7 % of I0 in the Warsaw tests.

_read_img_array_and_optical_black_from_raw_file returns the usual data plus a version without the clipping, in which every Bayer color keeps the black level of its own optical black pixels, and the read noise of every single pixel. _read_raw_image, _clipped_data and _unclipped_data split what was one function, so the two readers share how the data are scaled instead of repeating it, and the file is still read once. The orientation of #156 reaches all four arrays.

2. Step 3 writes the unclipped intensity and its noise (35fc09d)

ob_sum_col_value and ob_noise_sum_col_value, after the side band columns, so the prefix of the files released with v0.11.0 keeps its place and its values. The layouts of -s3_fast and -s3 now have 12 and 26 names, which collide with none of 6, 7, 10, 21 and 24. Both columns are NaN for JPG and PNG and for sensors without an optical black margin.

read_channel_data_and_mask_from_img becomes read_channel_data_mask_and_optical_black_from_img and returns the two further arrays, so step 3 still reads every image once. Both carry 0 outside the color channel, so the sum over a search area and the variance of that sum need the mask no further.

3. The linear solver weights the LEDs by their signal to noise ratio (552b2fe)

snr_full_weight weights every LED with min(1, SNR / snr_full_weight) — the inverse standard deviation of the optical depth, capped so that well measured LEDs keep the weight 1 and the meaning of lambda_reg does not change. LEDs at or below zero signal get the weight 0, an image whose weights do not add up to one full LED yields NaN, and layers whose path length weighted mean LED weight stays below min_layer_weight become NaN because their coefficients would rest on LEDs near the noise level.

The weighting needs a reference property with a noise column, which only ob_sum_col_val has. Asking for it with any other property is refused by name, and the loader stops if the noise column is missing from the binary data.

How num_moving_avg_images relates to n_summarize

While #157 sat unmerged, #165 made the existing average_images configurable through n_summarize. The two are different and both are kept:

	what it does	time resolution	rows
average_images + n_summarize	summarizes N images into one and numbers the result from 1 again	N times coarser	N times fewer
num_moving_avg_images	centred moving average of the reference property, every image keeps its own time	N times coarser	unchanged

The old branch called the second one num_avg_images, one letter away from the first and impossible to tell apart in a config file. It is renamed, and combining the two is refused with a message that says which one to pick, since the result would be smoothed and decimated at once.

Tests

Unit tests pass, 337. New files: test_led_weighting.py for the weights, the discarded layers, the two refusals and the smoothing; test_black_level.py grew the unclipped reader and the two columns of an LED. The RAW test of the channel mask now mocks the one pass over the RAW file instead of a single reader, so every reader derived from it stays consistent.

The encoding check of the CI is clean locally.

Not covered: no RAW image is read by any unit test, so the optical black itself still rests on the comparison against BCL on the Warsaw tests V008 and V009, camera 2. The acceptance tests run in the CI here, which is also the first run under the pinned numpy 1.26.4 — this machine has 2.2.6.

Why this is a draft

The clipping is still in place for every column but ob_sum_col_val. Removing it from the default channel data is the point of the exercise and is not in here: it lets sum_col_val go negative, which changes a contract that max_col_value, the fit and the postprocessing rely on, and it needs the nonlinear solver and the stacked analysis to be covered first.

Two decisions follow from that and are open:

• Two policies for "the LED has vanished" would then coexist. subtract_pedestal clamps to max(S - N * B, background_floor * S) and its own docstring says values at the limit carry no information on the extinction, yet nothing marks them. The answer here is the opposite: keep the negative value and discard it by SNR. Combined, the floor must not clamp a sum based on the optical black. Today the two cannot meet, because the weighting is wired to ob_sum_col_val alone and that column never goes through subtract_pedestal.
• ob_noise_sum_col_value is the read noise of the box only. Subtracting a side band pedestal would add the variance of that estimate, and the side bands hold far fewer pixels than the box, so the weighting would be optimistic exactly in dense smoke.

Replaces #157.

🤖 Generated with Claude Code

[kriboe90]

Folded into #170, which already contains every commit of this branch.

#170 removes the clipping from the default channel data, and once sum_col_val is the unclipped sum the ob_sum_col_val introduced here is the same quantity under a second name. Merging this first would release a column and retire it in the next PR, so the two belong together.

What came from here is unchanged in #170: the read without the clipping, the per-pixel read noise, the SNR weighting with snr_full_weight and min_layer_weight, and num_moving_avg_images reconciled against the n_summarize that #165 added. ob_noise_sum_col_val is now noise_sum_col_val, the noise of sum_col_val, so the weighting applies to the default reference property.

The branch stays as it is.