[kriboe90]

Draft for the open half of the dark level correction. The finished half was split out into #158, which should merge first. This branch still contains it, so its diff will shrink to the remainder once #158 is in master.

What belongs here

• ob_sum_col_value and ob_noise_sum_col_value: the integrated intensity of an LED without the clipping at zero, with the black level of every color of the Bayer pattern taken individually, and the read noise of that sum from the optical black.
• snr_full_weight and min_layer_weight: the linear solver weights every LED with min(1, SNR / snr_full_weight), and layers whose path-length-weighted mean LED weight stays below min_layer_weight become NaN instead of resting on LEDs near the noise level.
• num_avg_images: centred moving average of the reference property over N images before the normalisation, with the reference intensities taken from the single images.
• Still to come: removing the clipping at zero from the default channel data, so that sum_col_val and the side band columns profit from it as well and not only an opt-in column.

What moved to #158

The merge of master, the optical black as the mandatory black level source with the fallback to the metadata value, the LEDSA version in the step 3 header, and the binary data guard. None of that has an open question, and it improves every reference property rather than an opt-in one.

What was dropped

• dark_img_indices, which subtracted a per-LED dark level taken from images with the LEDs and the ambient light switched off. ob_sum_col_val takes the same information from the optical black pixels of every image, per Bayer color and without extra exposures at the experiment, so the option was superseded by its own commit message. It spanned seven files, a solver attribute, subtract_dark_intensities and an option of config_analysis.ini.
• read_channel_data_and_mask_from_img, which has no caller left because step 3 reads through read_channel_data_and_optical_black_from_img. Its three unit tests now cover that reader instead, the first one driving it through a fake RAW image rather than mocking the function under test.

Why this stays a draft

• The clipping at zero is the point and is not in yet. Removing it lets the sums go negative, which changes a contract that max_col_value, the fit and the postprocessing rely on, and the nonlinear solver and the stacked analysis are not covered.
• Two policies for "the LED has vanished" would then coexist. subtract_pedestal clamps to max(S - N * B, background_floor * S) and its own docstring says values at the limit carry no information on the extinction, yet nothing marks them. The answer here is the opposite: keep the negative value and discard it by SNR. Combined, the floor must not clamp a sum based on the optical black. Today the two cannot meet, because NOISE_COLUMNS wires the SNR weighting to ob_sum_col_val alone and that column never goes through subtract_pedestal.
• ob_noise_sum_col_value is the read noise of the box only. Subtracting a side band pedestal adds the variance of that estimate, and the side bands hold far fewer pixels than the box, so the weighting would be optimistic exactly in dense smoke.
• Validation. The comparison against BCL covers the Warsaw tests V008 and V009, camera 2. No RAW image is read by any unit test, and the acceptance tests have not run on this branch.

Note on the merge state

This branch conflicts with master in four files, because #155 and #156 landed after the merge commit it carries. That is left as it is on purpose: merging master again now would duplicate the resolution that #158 already contains. Once #158 is merged, master comes in here and the conflicts reduce to the parts this branch really changes.

🤖 Generated with Claude Code

[ManuelOsburg]

Hi Kristian,

thanks for merging #145 and for the follow-up in #150.

One of the points you left open there is that values at background_floor are not flagged. #157 discusses the same question as two policies for an LED that has vanished. Our fork has a marking for this, and I'd like to know where you want it before I prepare anything.

What it does. python -m ledsa -rv writes analysis/ray_validity_channel<c>.csv with one row per ray, i.e. per image and LED. The extinction coefficients are not changed. Two criteria are evaluated, both based on the side band pedestal P = sidebg_pedestal:

• Raw reading (sum_col_val): a ray is saturated if S − P ≤ 0, or if b_raw = −ln(S/S0) falls short of the corrected optical depth b_corr = −ln((S − P)/(S0 − P0)) by more than a relative threshold, i.e. 1 − b_raw/b_corr > threshold.
• Side band reading (sidebg_sum_col_val): a ray is invalid in three cases:
  • the floor applies, S − P ≤ floor · S;
  • a relative error U_B of the pedestal shifts b_sidebg by more than the threshold relative to b_sidebg, with delta_b = U_B · (P/S_corr + P0/S0_corr), image and reference term added;
  • there is no pedestal (NaN).

S0, P0 and S0_corr are means over the reference images. The threshold (0.1), the minimum optical depth from which the relative criteria apply (0.1) and U_B (0.2) are set values, not derived ones.

On one camera of one of our Hasenheide runs, the marking agrees with that of our own evaluation scripts on all 216,384 rays, for thresholds from 0.05 to 0.2 and U_B from 0.1 to 0.3. That confirms the implementation, not the criterion itself.

Relation to #157. It addresses two points you name there:

• it marks the values clamped by the floor;
• it accounts for the uncertainty of the side band estimate, which ob_noise_sum_col_value leaves out. That uncertainty is the set relative error U_B, though, not a variance estimated from the band pixels.

It is a marking after the fact and does not weight the inversion.

Options I see:

1. A separate PR now. It would contain only the marking, adapted to background_floor, and be independent of Optical black intensity without clipping, SNR weighting and image averaging #157. If Optical black intensity without clipping, SNR weighting and image averaging #157 later weights the solver by SNR, the marking stays a diagnostic for sidebg_sum_col_val.
2. No separate PR; fold the criterion into Optical black intensity without clipping, SNR weighting and image averaging #157. sidebg_sum_col_val would get an entry in NOISE_COLUMNS whose noise includes the uncertainty of the pedestal, and a clamped value would get weight 0. That would leave one policy instead of two.
3. Neither: it stays in the fork.

Which would you prefer?

On the other open point, mean vs. trimmed mean in the side bands, I could compare both estimators on our Hasenheide data, if that helps.

Best,
Manuel

[kriboe90]

Replaced by #169, which re-cuts this work onto current master.

This branch had drifted eleven conflicting files away from master: the finished half merged as #158, and #161 through #167 landed after the merge commit it carries. Rebuilding the remainder on top of master was cleaner than resolving that, as it was for #158.

Carried over: the unclipped ob_sum_col_value and ob_noise_sum_col_value, the SNR weighting of the linear solver, and the moving average, now called num_moving_avg_images and reconciled with the n_summarize that #165 added in the meantime.

Dropped for good: dark_img_indices, superseded by ob_sum_col_val, and read_channel_data_and_mask_from_img, which has no caller left.

The branch stays as it is, so nothing is lost.