[kriboe90]

Supersedes #169, which is folded in here: this branch holds all of its commits. Together they finish the dark level correction whose first half merged as #158.

The problem

sum_col_val carried an additive offset that did not vanish when an LED dimmed in smoke, so the extinction coefficients saturated in dense smoke. The problem statement of #145 names three sources; #158 removed one of them by taking the black level from the optical black pixels instead of the RAW metadata. The one left is the clipping at zero after that subtraction, which turns the negative half of the read noise into a positive offset. In the Warsaw tests it was 0.3 to 0.7 % of I0.

The commits

3fff826 read the RAW data without the clipping at zero as well. _read_raw_image, _clipped_data and _unclipped_data split what was one function, so every version of the data shares how it is scaled and the file is still read once. The orientation of #156 reaches all of them.

35fc09d write the unclipped intensity and its noise in step 3. As two opt-in columns at that point, so the clipped and the unclipped sum could be compared in one run.

552b2fe weight the LEDs of the linear solver by their signal to noise ratio. snr_full_weight weights every LED with min(1, SNR / snr_full_weight) — the inverse standard deviation of the optical depth, capped so that well measured LEDs keep the weight 1 and the meaning of lambda_reg does not change. An image whose weights do not add up to one full LED yields NaN, and layers whose path length weighted mean LED weight stays below min_layer_weight become NaN, because their coefficients would rest on LEDs near the noise level.

39b30c4 stop clipping the channel data at zero. The channel data are now float32 without a lower limit, and every Bayer color is corrected with the black level of its own optical black pixels. That resolves the old TODO: channel is only relevant for black level, consider individually!: the channel no longer selects a black level at all.

Two further commits follow from the measurement below and have sections of their own: f32b64f turns the SNR weighting on by default and aa4cf07 takes the lower limit off the corrected sums.

Because the default is now the unclipped sum, the two columns of the second commit collapse into it: ob_sum_col_val is gone and ob_noise_sum_col_val became noise_sum_col_val, the noise of sum_col_val. The SNR weighting therefore applies to the default reference property rather than an opt-in one. The layouts of -s3_fast and -s3 have 11 and 25 names, which collide with none of 6, 7, 10, 21 and 24.

What the clipping was hiding

Audited before anything was changed. Four places relied on non-negative data:

• The binary data cast sum_col_val and max_col_val to int. The whole effect is a fraction of a count, so that cast would have discarded it. Both stay float.
• _estimate_local_background dropped every border pixel at or below zero, because foreign Bayer colors are masked with 0. It takes the channel mask instead: a channel pixel may now legitimately be 0 or negative, and dropping those raises the median. bgsub_sum_col_val therefore changes for Bayer images.
• bgsub_sum_col_val clipped pixel by pixel after subtracting the background, putting the offset straight back. It subtracts the median linearly now, as the side bands do. Its own bias test flips from bias['bgsub'] < -0.02 to abs(bias['bgsub']) < 0.02, which is the clearest single measurement of what this PR buys.
• The linear solver clamped a relative intensity at or below zero to 1e-10, an optical depth of about 23 that would have dominated the solve. Such an LED has no signal left and now gets the weight 0 whether or not the SNR weighting is on.

Checked and deliberately untouched: the LED fit, which minimises a least-squares cost and assumes nothing about the sign, the nonlinear solver, the stacked analysis and the postprocessing.

How num_moving_avg_images relates to n_summarize

While this work sat unmerged, #165 made the existing average_images configurable through n_summarize. The two are different and both are kept:

	what it does	time resolution	rows
average_images + n_summarize	summarizes N images into one and numbers the result from 1 again	N times coarser	N times fewer
num_moving_avg_images	centred moving average of the reference property, every image keeps its own time	N times coarser	unchanged

The earlier name, num_avg_images, was one letter from the other and indistinguishable in a config file. Combining the two is refused with a message saying which to pick, since the result would be smoothed and decimated at once.

The lower limit of the corrected sums is off by default

background_floor kept the corrected sums at max(S - N * B, floor * S) with 0.05, because the clipping made the pedestal of a vanished LED unavoidable and the optical depth ran away where the LED disappeared in it. With unclipped data the limit does the opposite of what is wanted: a corrected sum at or below zero is the statement that the signal has vanished, and the solver acts on it — weight 0 for the LED, NaN for a layer whose remaining LEDs sit in the read noise. The limit turns that statement into 5 % of the plain sum, which the solver cannot tell from a measurement. On V008 that is the regime at 1.24 and 1.64 1/m, where the honest answer is NaN.

The docstring of subtract_pedestal already said that values at the limit "carry no information on the extinction and should be treated as invalid"; nothing ever marked them. Now something does.

The default is None, in the code and in the config.ini template. A fraction is still accepted, so the behaviour of v0.11.0 is one option away, and configurations that name one keep it (aa4cf07).

Tests

Unit tests pass, 362. The ones that changed, changed because the contract did: test_masked_zero_pixels_are_ignored becomes test_foreign_pixels_are_excluded_by_the_mask_and_not_by_their_value plus a new test that channel pixels at or below zero are kept, and the two "values are unchanged" tests now assert that bgsub_sum_col_val differs with and without a channel mask.

New: test_black_level.py for the per-color black levels and scaling, the fallback to the metadata, values that stay negative, the float dtype that keeps the fraction of a count, the per-pixel read noise and the sums of an LED; test_led_weighting.py for the weights, the discarded layers, the two refusals and the smoothing.

Measured against BCL

Run on the Warsaw tests V008 and V009, camera 2, with 39b30c4, in three variants: n_sum is sum_col_val unclipped, n_sumw the same with snr_full_weight = 10 and min_layer_weight = 0.5, n_bgsub the new bgsub_sum_col_val. The remaining settings come from the existing config_analysis.ini. The numbers are the ratio LEDSA / BCL for the red LEDs of V008, so 1.00 is agreement.

fog stage (BCL k)	with clipping	n_sum	#169 ob column	n_sumw	n_bgsub	#131 bgsub
1 (0.41 1/m)	0.96	0.98	0.98	0.98	1.01	1.01
2 (0.81 1/m)	0.87	0.96	0.96	0.95	1.02	0.98
3 (1.24 1/m)	0.65	0.86	0.99	NaN	0.82	0.78
4 (1.64 1/m)	0.50	0.69	0.97	NaN	0.71	0.60

Where the measurement is valid, the clipping was the error. At stage 2 the ratio goes from 0.87 to 0.96. The residual in the dark frames falls from 0.68 % of I0 with clipping to −0.01 % without it, which is the offset this PR is about, measured. n_sum reproduces the ob_sum_col_val of #169 exactly, as it must, since they are the same quantity.

At stages 3 and 4 the red LEDs of this camera no longer measure anything. The transmission is 0.1 to 0.2 % of I0 against a read noise of about 0.17 %, so 10 to 32 % of the values are at or below zero. The step 3 data are identical to #169's; everything below comes from the solver.

Do not read the table as a regression at stages 3 and 4. The 0.97 and 0.99 of #169 were not a valid measurement either: that branch gave every LED at or below zero an optical depth of about 23, which happened to cancel the low bias of the LEDs that survived, and the scatter over strips and heights reached 1.0. The agreement was an artefact of two errors meeting. This PR replaces the fabricated 23 with weight 0, and n_sumw then returns NaN, which is the honest answer where the signal sits in the noise.

n_sum without the weighting is biased low there, and the solver cannot notice. Only the LEDs whose noise happened to be positive survive the weight-0 rule, so the surviving sample is biased bright and k comes out too low: 0.86 and 0.69. The guard weights.sum() < 1.0 is far too permissive to catch this, because 68 to 90 % of the LEDs still carry full weight. What catches it is the SNR weighting: with a signal-to-noise ratio near 1 the weights fall to about 0.1, the path weighted layer weight drops below min_layer_weight, and the layer becomes NaN.

So snr_full_weight is not a refinement once the transmission approaches the read noise — it is the only mechanism that distinguishes "no signal" from "small signal". It is therefore on by default (f32b64f), with DEFAULT_SNR_FULL_WEIGHT = 10, the value this comparison used. It costs nothing at stages 1 and 2 (0.98 and 0.95) and turns the saturation regime into NaN instead of a number that is 14 to 31 % too low.

A default may not stop an analysis of data that cannot carry it, so the weighting switches itself off with a note rather than raising in three cases: a reference property without a noise column, a binary file written before the noise column existed, and a noise column that is NaN for every LED — which is what JPG and PNG images and sensors without an optical black margin give. snr_full_weight = None still switches it off by hand, and the result header now reports Noise_property next to Snr_full_weight, so it says whether the weighting was used and not only whether it was asked for. The nonlinear solver has no weighting and passes None explicitly.

One related fix: an LED whose noise is unknown now keeps the full weight instead of weight 0. A missing noise estimate is no reason to drop a measured intensity, and without this a RAW image without an optical black margin would have produced NaN everywhere.

n_bgsub is unbiased at stages 1 and 2 (1.01, 1.02) but noticeably noisier than n_sum, and it over-subtracts by about 0.05 % of I0. In V009 it runs 0 to 4 points high and crosses above BCL from roughly 200 s at 2.17 m. That is the border ring sitting in the halo of the LED, which #145 already held against #131's estimator; the clipping used to mask it.

V009 shows no regression where the effect is small. The signal there is about 15 % of I0, so the clipping barely matters: n_sum and n_sumw lie 0.1 to 0.7 points above the stored results in the last minute before the flame ends, and at most 0.3 points from #169.

One side effect to record: step 3 now writes the image times with fractions of a second, 0 to 1 s later than the existing image_infos. The IDs and names are unchanged. It comes from the EXIF sub-second work, not from this PR, but it shifts the time axis slightly against older results and against a reference series.

What is still open

Nothing in the design. Three defaults changed here — the unclipped data, the SNR weighting, the missing lower limit — and they are three parts of one decision: where the signal of an LED has vanished, say so instead of producing a number. Each is one config option away from the old behaviour.

What remains is review, and a judgement that belongs to you: the BCL comparison above covers the red LEDs of V008 and V009 on camera 2. It is the clearest data available, and it is one camera on two tests.

🤖 Generated with Claude Code