Skip to content

Add AxisArray.segment, a counter marking breaks in a stream - #277

Draft
cboulay wants to merge 3 commits into
fix/272-shm-grow-retained-viewsfrom
cboulay/axisarray-segment
Draft

cboulay wants to merge 3 commits into
fix/272-shm-grow-retained-viewsfrom
cboulay/axisarray-segment

Conversation

@cboulay

@cboulay cboulay commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Stacked on #273.

Why

Stateful processors assume each message continues the previous one. Nothing on a message says when that stops being true: after dropped packets, a pause, a new section of a file, or a source that was reconfigured. Today each consumer has to infer it, from timestamps (unreliable with jittery sources, and invisible once a stage has rewritten them) or from a hash of the message layout. See ezmsg-org/ezmsg-sigproc#247.

What

AxisArray.segment: int = 0. A producer changes it, normally by adding one, on the first message that does not continue the previous one. Consumers compare it with the last value they saw and start again when it differs.

  • It is a level and not an edge, so it survives a stage that drops or merges messages. A boolean "new segment" flag on one message would not.
  • Only a change means anything; 0 is where a stream that never breaks stays.
  • It is the last field with a default, so existing producers and positional construction are unaffected.
  • replace, pickling, wire elision, iter_over_axis and transpose already carry it; tests pin each.

Nothing in ezmsg core reads the field. ezmsg-baseproc will compare it in BaseStatefulProcessor.__call__, and sources will set it.

Equality

AxisArray.__eq__ now compares stream_dim and segment as well as key, attrs and axes. Before, two messages that differed only in stream_dim compared equal. This is a behaviour change for any code that compares a message declaring stream_dim with one that does not; the ezmsg, ezmsg-baseproc and ezmsg-sigproc test suites pass unchanged.

Not changed

  • AxisArray.concatenate keeps the first message's value, as it does for key and attrs.

A producer changes it on the first message that does not continue the
previous one: after a gap in time, or after the source is reconfigured.
Stateful consumers compare it with the last value they saw and start again
when it differs. Defaults to 0 and is the last field, so existing producers
and positional construction are unaffected.

This branch was previously deployed

1 inactive deployment
github-pages — 1c67d78c Deployed Nov 13, 2026 by cboulay via deploy #128
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant