[ManuelOsburg]

Problem

The search area of an LED contains an additive pedestal besides the LED itself: stray light, light emitted or scattered by the smoke, and, for raw images, noise clipped at 0 after the black level subtraction. Under dense smoke the LED dims while the pedestal stays, so sum_col_val overestimates the transmission. bgsub_sum_col_val (#131) takes the background from the 2 px border ring of the search area. For small search areas that ring lies in the halo of the LED itself, so it measures part of the LED as background.

Change

Step 3 writes three more columns after bgsub_sum_col_value. Like bgsub_sum_col_val, they are opt-in through reference_property:

Column	Background estimate
sidebg_sum_col_val	mean B over the channel pixels in side bands next to the LED strip, subtracted as N·B
sidebg_pedestal	the subtracted N·B
bgsub2_sum_col_val	mean b over the channel pixels (including 0) of the 2 px border ring, subtracted as N·b

N is the number of channel pixels in the search area.

• Side bands:

  • They run parallel to the strip, 25 to 60 px across it and ±R along it, with R = search_area_radius.
  • The strip direction is estimated for each LED from its nearest neighbours in the same LED array of step 2, so vertical, horizontal and oblique strips work. With merge_led_array_indices the arrays of step 2 are used, because a merged array may contain several strips.
  • Pixels within R + margin of any other search area are excluded, so neighbouring strips do not enter the estimate.
  • The geometry depends only on the LED positions and is cached per process.

• Means include zeros, subtraction is linear: clipped zeros are part of the box sum, so they are part of the mean as well. Subtracting N·b instead of clipping pixel by pixel keeps the correction linear in the pixel values.

• Channel mask: for raw images the channel pixels are taken from the Bayer pattern (raw_colors_visible), not from the pixel values, since channel pixels can be 0. For this, read_channel_data_and_mask_from_img is added next to read_channel_data_from_img. It goes through _get_reader and _read_or_raise, so unreadable images are skipped as in Develop skip corrupted images #137.

• Lower limit: both corrected sums are kept at least at sideband_floor, a fraction of sum_col_val (default 0.05). Without it the optical depth would be unbounded where the LED vanishes in the pedestal. Values at this limit carry no information on the extinction.

• Options: config.ini gets sideband_distances (25 60), sideband_floor (0.05) and sideband_exclusion_margin (None = R) in [analyse_photo]. Config files without the options use the defaults.

• Reading the results: _get_column_names knows the new layouts with 9 (-s3_fast) and 23 (-s3) columns. The earlier layouts with 5, 6 and 20 columns are still read.

• NaN: sidebg_sum_col_val is NaN for an LED whose strip direction cannot be estimated (an LED array with a single LED) or whose side bands contain no pixel, for example because the image ends closer to the strip. bgsub2_sum_col_val is NaN only if the search area is too small for a border ring (at most 4 px). The solvers do not handle NaN:

  • the linear one stops with ValueError: array must not contain infs or NaNs from nnls;
  • the nonlinear one writes zeros as extinction coefficients without a warning.

  ExtinctionCoefficients.load_img_data therefore stops with an error if the selected reference property contains NaN, and names the LEDs. sum_col_val and bgsub_sum_col_val contain no NaN, so this check does not affect them.

• Docs: README section "Background correction (optional)", the -s3_fast entry of the CLI docs and the API docs of the new module.

Default results are unchanged

• Test data: -s3_fast followed by --analysis (linear and nonlinear, sum_col_val and bgsub_sum_col_val) gives bit-identical extinction coefficients on master and on this branch. The step 3 files are identical in their first 6 columns.
• Real data: the same holds on three raw images of a fire experiment, in all three channels. There, the analysis with sidebg_sum_col_val also runs through.

Tests

• Unit tests:
  • test_sideband_background.py covers:
    • the strip direction and the band geometry, including excluded neighbouring strips and clipping at the image border;
    • the LED arrays used with merged arrays;
    • the pedestal estimate, including clipped zeros;
    • the corrected sum and the lower limit;
    • unchanged earlier values and the options of old and new config files;
    • the Bayer channel mask and read_channel_data_and_mask_from_img;
    • the error for NaN in the reference property. Without the check, this test fails.
  • test_ring_background_mean.py covers the ring mean and bgsub2_sum_col_val.
  • test_background_subtraction.py additionally checks the new column layouts.
• New suite 10_test_sideband_background.robot:
  • The results for sum_col_val and bgsub_sum_col_val are bit-identical with and without the three new columns.
  • On clean images sidebg_sum_col_val reproduces the ground truth: rmse 0.014 / 0.016 / 0.022 < 0.05.
  • With a constant background of 15, sum_col_val is biased (rmse 0.13 / 0.18 / 0.13 > 0.05), while sidebg_sum_col_val stays close (0.022 / 0.025 / 0.027 < 0.06), also below half the rmse of the sum.
  • The test images are only 50 px wide, so the suite sets sideband_distances = 12 20.
  • Suite 07 covers corrupted images with the new reader, since -s3_fast now reads through it.
• Runs:
  • Windows without UTF-8 mode (cp1252): the whole Robot suite passes 23/23 and the unit tests pass 71/71. The encoding check of the CI is clean.
  • Linux: I can only see the Linux leg in CI. Statically, the changed files parse with the Python 3.10 grammar, the blobs have LF line endings and git diff --check is clean.
• Together with Read the column names from the first existing step 3 result file #143: the branch merges with Read the column names from the first existing step 3 result file #143 without conflicts, and the unit tests of both pass, 77/77.