Repository navigation
Conversation
A producer changes it on the first message that does not continue the previous one: after a gap in time, or after the source is reconfigured. Stateful consumers compare it with the last value they saw and start again when it differs. Defaults to 0 and is the last field, so existing producers and positional construction are unaffected.
… into cboulay/axisarray-segment
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #273.
Why
Stateful processors assume each message continues the previous one. Nothing on a message says when that stops being true: after dropped packets, a pause, a new section of a file, or a source that was reconfigured. Today each consumer has to infer it, from timestamps (unreliable with jittery sources, and invisible once a stage has rewritten them) or from a hash of the message layout. See ezmsg-org/ezmsg-sigproc#247.
What
AxisArray.segment: int = 0. A producer changes it, normally by adding one, on the first message that does not continue the previous one. Consumers compare it with the last value they saw and start again when it differs.0is where a stream that never breaks stays.replace, pickling, wire elision,iter_over_axisandtransposealready carry it; tests pin each.Nothing in ezmsg core reads the field. ezmsg-baseproc will compare it in
BaseStatefulProcessor.__call__, and sources will set it.Equality
AxisArray.__eq__now comparesstream_dimandsegmentas well askey,attrsandaxes. Before, two messages that differed only instream_dimcompared equal. This is a behaviour change for any code that compares a message declaringstream_dimwith one that does not; the ezmsg, ezmsg-baseproc and ezmsg-sigproc test suites pass unchanged.Not changed
AxisArray.concatenatekeeps the first message's value, as it does forkeyandattrs.