Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,15 @@
- **A row whose onset is `n/a` is validated wherever it is.** Such a row was skipped unless it was the last row of the file, so its assembled-row errors (for example a tag repeated across the `HED` column and the sidecar) went unreported. ([#1481](https://github.com/hed-standard/hed-python/issues/1481))
- **`n/a` onsets no longer cause `ONSETS_UNORDERED`**: only the numeric onsets must be in order. `EventManager` still requires every onset to be numeric.
- **The HED string context of a `TEMPORAL_TAG_ERROR` for an unconvertible `Delay`** (`(Delay/3 month, (Red))`) keeps the `Delay` group; it was removed after the issue was made.
- **Errors that were missed are now reported** (hed-tests cases added in hed-tests #99):
- Misnested parentheses with equal counts, such as `(Red)), ((Blue)` and `)(Red)(` (`PARENTHESES_MISMATCH`).
- A `#` that is not the whole value of its tag, such as `Label/x#`, `Label/#x` or `Age/##`, in a sidecar or a definition (`PLACEHOLDER_INVALID`, specification Appendix B reason f). Units after one blank (`Frequency/# Hz`) are not part of the value. A definition containing `Age/##` now gets this code instead of `DEFINITION_INVALID`.
- A curly-brace reference whose name has characters other than letters, digits, hyphen and underscore, such as `{my.col}`: it was ignored, so an unknown column went unreported and a known one was never substituted (`SIDECAR_BRACES_INVALID` when unknown). A name with a comma or parenthesis is still not a reference, since a reference must parse as one tag. A `#` in a referenced name (`{col#1}`) is no longer filled with the row value when the HED is assembled.
- A definition beside other tags in a sidecar entry, categorical or value, such as `(Definition/MyDef, (Red)), Label/#` (`DEFINITION_INVALID`). An entry that holds only definitions is still valid.
- A `HED` key below the second level of an entry that also has a valid one, such as in `Levels` (`SIDECAR_INVALID`).
- Units in a `Def` value checked against the definition's placeholder tag with its units: `Def/Freq/3 kHz` with `(Definition/Freq/#, (Frequency/# Hz))` gives `Frequency/3 kHz Hz` (`UNITS_INVALID`).
- **A sidecar string that references `{HED}` is no longer checked for repeated tags and group rules in the sidecar**, since what the `HED` column contributes is known only per row (specification 3.2.9.3 rule 10). `(Red, {HED}), (Red)` was reported as `TAG_EXPRESSION_REPEATED` although a row with `HED=Blue` is valid; such errors are now found when each row is assembled.
- **`annotation_util.df_to_hed` with `description_tag=True` no longer adds a `Description` tag beside definitions**, which made the sidecar invalid; the description stays in the `Levels` or `Description` text.
- In a file without an `onset` column, the `char_index` of an assembled-row issue now counts in the assembled string with `", "` between cells, as for files with onsets (one more per earlier cell than before).

## Specification files
Expand Down
4 changes: 3 additions & 1 deletion hed/errors/error_messages.py
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,9 @@ def val_error_reserved_tag_splice(tag, group):


@hed_error(ValidationErrors.PARENTHESES_MISMATCH)
def val_error_parentheses(opening_parentheses_count, closing_parentheses_count):
def val_error_parentheses(opening_parentheses_count, closing_parentheses_count, unmatched_index=None):
if unmatched_index is not None:
return f"Closing parenthesis at index {unmatched_index} has no opening parenthesis before it."
return (
f"Number of opening and closing parentheses are unequal. "
f"{opening_parentheses_count} opening parentheses. {closing_parentheses_count} "
Expand Down
11 changes: 10 additions & 1 deletion hed/models/column_mapper.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"""

import copy
import re
import warnings
from collections import Counter
from functools import partial
Expand All @@ -12,6 +13,10 @@
from hed.models.column_metadata import ColumnMetadata, ColumnType
from hed.models.column_source import is_blank_name
from hed.models.definition_dict import DefinitionDict
from hed.models.model_constants import COLUMN_REF_PATTERN

# A curly-brace reference, matched so it is kept whole, or a placeholder outside one.
_PLACEHOLDER_OUTSIDE_REFS = re.compile(COLUMN_REF_PATTERN.pattern + "|#")

NO_WARN_COLUMNS = ["onset", "duration"]

Expand Down Expand Up @@ -459,4 +464,8 @@ def _value_handler(value_str, x):
if x == "n/a":
return "n/a"

return value_str.replace("#", str(x))
# A column name may hold "#" ({col#1}), so placeholders inside curly-brace references are left alone.
value = str(x)
return _PLACEHOLDER_OUTSIDE_REFS.sub(
lambda match: value if match.group(0) == "#" else match.group(0), value_str
)
3 changes: 2 additions & 1 deletion hed/models/df_util.py
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,8 @@ def _remover(match):
# c1/c2 contain the comma(and possibly spaces) separating this ref from other tags
# p1/p2 contain the parentheses directly surrounding the tag
# All four groups can have spaces.
pattern = r"(?P<c1>[\s,]*)(?P<p1>[(\s]*)" + old_value + r"(?P<p2>[\s)]*)(?P<c2>[\s,]*)"
# The reference is escaped: a column name may hold regex characters ({my.col}, {col+1}).
pattern = r"(?P<c1>[\s,]*)(?P<p1>[(\s]*)" + re.escape(old_value) + r"(?P<p2>[\s)]*)(?P<c2>[\s,]*)"
return re.sub(pattern, _remover, text)


Expand Down
7 changes: 7 additions & 0 deletions hed/models/model_constants.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,14 @@
"""Defined constants for definitions, def labels, and expanded labels."""

import re
from enum import IntEnum

# A curly-brace column reference in a sidecar annotation, ``{column_name}`` or ``{HED}``. A column name may hold
# any character but a brace, comma or parenthesis (``{my.col}``): a reference must parse as a single tag
# (specification 3.2.9.3, rule 8), so ``{a,b}`` is braces around HED, not a reference. Nested and unmatched braces
# are reported before this is applied.
COLUMN_REF_PATTERN = re.compile(r"\{([^{},()]+)\}")

# The whitespace that may not begin or end a column name or a top-level sidecar key (specification 3.2.10):
# blank, tab, line feed and carriage return only. A non-breaking space is not padding.
NAME_WHITESPACE = " \t\n\r"
Expand Down
4 changes: 2 additions & 2 deletions hed/models/sidecar.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
"""Contents of a JSON file or merged JSON files."""

import json
import re

from hed.errors import ErrorHandler
from hed.errors.error_types import ErrorContext
from hed.errors.exceptions import HedExceptions, HedFileError
from hed.models.column_metadata import ColumnMetadata, ColumnType
from hed.models.definition_dict import DefinitionDict
from hed.models.hed_string import HedString
from hed.models.model_constants import COLUMN_REF_PATTERN


class Sidecar:
Expand Down Expand Up @@ -240,7 +240,7 @@ def get_column_refs(self) -> list[str]:
if column_data.column_type == ColumnType.Ignore:
continue
hed_strings = column_data.get_hed_strings()
matches = hed_strings.str.findall(r"\{([a-z_\-0-9]+)\}", re.IGNORECASE)
matches = hed_strings.str.findall(COLUMN_REF_PATTERN)
Comment thread
VisLab marked this conversation as resolved.
Resolved
u_vals = [match for sublist in matches for match in sublist]

found_vals.update(u_vals)
Expand Down
12 changes: 11 additions & 1 deletion hed/tools/analysis/annotation_util.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@
from hed.models.sidecar import Sidecar
from hed.models.tabular_input import TabularInput

# A Definition tag at the start of a tag: short, long or intermediate form, with an optional library prefix
# ("tl:Definition/MyDef"). Not "Def/MyDefinition/3" or "Description/See organizational-property/definition/x".
_DEFINITION_TAG_PATTERN = re.compile(
r"(?:^|[,(])\s*(?:[a-z]+:)?(?:(?:property/)?organizational-property/)?definition/", re.IGNORECASE
)


def check_df_columns(df, required_cols=("column_name", "column_value", "description", "HED")) -> list[str]:
"""Return a list of the specified columns that are missing from a dataframe.
Expand Down Expand Up @@ -388,7 +394,11 @@ def _get_value_entry(hed_entry, description_entry, description_tag=True):
tags = hed_entry
if description_entry and description_entry != "n/a":
value_dict["Description"] = description_entry
if description_tag and tags:
if _DEFINITION_TAG_PATTERN.search(tags):
# An entry that holds definitions may hold nothing else (specification 3.2.9.1), so the description
# is kept only in the Description or Levels text, not as a Description tag beside the definitions.
pass
elif description_tag and tags:
tags = tags + ", Description/" + description_entry
elif description_tag and not tags:
tags = "Description/" + description_entry
Expand Down
13 changes: 12 additions & 1 deletion hed/validator/def_validator.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,19 @@ def validate_def_value_units(self, def_tag, hed_validator, allow_placeholders=Fa
def_contents = def_entry.get_definition(def_tag, placeholder_value=placeholder, return_copy_of_tag=True)
if def_contents and def_entry.takes_value and hed_validator:
placeholder_tag = def_contents.get_first_group().find_placeholder_tag()
# When the definition writes units after its placeholder (Frequency/# Hz), the substituted tag is
# what must be valid: Def/Freq/3 kHz gives Frequency/3 kHz Hz, which is UNITS_INVALID. The value
# starts the extension in both, so the index offset is the same.
validate_text = placeholder
template_tag = def_entry.contents.find_placeholder_tag() if def_entry.contents else None
if template_tag is not None and template_tag.extension != "#":
validate_text = placeholder_tag.extension
def_issues += hed_validator.validate_units(
placeholder_tag, placeholder, report_as=def_tag, error_code=error_code, index_offset=len(tag_label) + 1
placeholder_tag,
validate_text,
report_as=def_tag,
error_code=error_code,
index_offset=len(tag_label) + 1,
)

return def_issues
Expand Down
42 changes: 42 additions & 0 deletions hed/validator/hed_validator.py
Original file line number Diff line number Diff line change
Expand Up @@ -281,6 +281,8 @@ def _validate_individual_tags_in_hed_string(self, hed_string_obj, allow_placehol
)
elif hed_tag.short_base_tag == DefTagNames.DEFINITION_KEY:
validation_issues += self._validate_definition_value(hed_tag, allow_placeholders)
elif allow_placeholders and self._placeholder_not_whole_value(hed_tag):
validation_issues += self._report_placeholder_not_whole_value(hed_tag)
elif allow_placeholders and hed_tag.is_unit_class_tag() and hed_tag.extension.startswith("# "):
# If placeholder is followed by units (e.g., "# m-per-s^2"), validate the units
validation_issues += self.validate_units(hed_tag, allow_placeholders=allow_placeholders)
Expand All @@ -289,6 +291,46 @@ def _validate_individual_tags_in_hed_string(self, hed_string_obj, allow_placehol

return validation_issues

@staticmethod
def _placeholder_not_whole_value(hed_tag) -> bool:
"""Return True if a takes-value tag has a ``#`` that is not the whole of its value.

Parameters:
hed_tag (HedTag): A tag that is not Definition, Def or Def-expand.

Returns:
bool: True for ``Label/x#`` or ``Label/#x``; False for ``Label/#``, ``Frequency/# Hz`` and for a
value without ``#``.

Notes:
- PLACEHOLDER_INVALID reason f of the specification (Appendix B). Units after one blank are not part
of the value.
"""
extension = hed_tag.extension
if "#" not in extension or extension == "#" or not hed_tag.is_takes_value_tag():
return False
return not (hed_tag.is_unit_class_tag() and extension.startswith("# "))

@staticmethod
def _report_placeholder_not_whole_value(hed_tag) -> list[dict]:
"""Report PLACEHOLDER_INVALID for a tag whose value has a ``#`` that is not the whole value.

Parameters:
hed_tag (HedTag): A tag for which ``_placeholder_not_whole_value`` is True.

Returns:
list[dict]: One PLACEHOLDER_INVALID issue, at the first ``#``. The value-class check is skipped, so
the code is PLACEHOLDER_INVALID rather than CHARACTER_INVALID.
"""
index = len(hed_tag.org_base_tag) + 1 + hed_tag.extension.index("#")
return error_reporter.ErrorHandler.format_error(
ValidationErrors.INVALID_TAG_CHARACTER,
tag=hed_tag,
index_in_tag=index,
index_in_tag_end=index + 1,
actual_error=ValidationErrors.PLACEHOLDER_INVALID,
)

def _validate_definition_value(self, definition_tag, allow_placeholders) -> list[dict]:
"""Validate the two-part value of a Definition tag: a name, then optionally ``/#``.

Expand Down
Loading
Loading