Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 20 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,13 +235,30 @@ testSpecs`.
characters is SIDECAR_INVALID (`paddedSidecarCategoricalKey`, hed-tests
`sidecar-invalid-value-key-whitespace`). Before, `go ` was trimmed to `go`
and annotated silently. A HED-column cell such as `n/a ` or a blanks-only cell
is still a missing value (`isMissingTsvValue`). A data line that holds a tab,
such as ` \t`, is a row and its cells are checked; an empty line or a line of
blanks without a tab is skipped, as hed-python reads it. Onsets are grouped
is still a missing value (`isMissingTsvValue`). A data line is a row and its
cells are checked, whether or not it holds a tab: ` \t` is a row, and so is a
line of one blank (a one-column table has no wrong cell count). An empty line
is reported, not skipped (see the line rules below). Onsets are grouped
without their surrounding whitespace when duplicate tags across rows are
checked, so `19` with or without blanks around it is one onset. A categorical
value with no key in a column whose annotation holds a `{column}` splice is
SIDECAR_KEY_MISSING, as in other columns; it was an INTERNAL_ERROR.
- The line rules of a tabular file (specification 3.2.10, as in hed-typescript,
hed-tests `tabular-invalid-blank-column-name`, `tabular-invalid-empty-line`
and `tabular-invalid-cell-count`): the header is always line 1, so an empty
first line is one blank column name (`blankTsvColumn`) instead of being
skipped. One line break at the end of the contents is not a line; any other
empty line is TABULAR_INVALID (`emptyTsvLine`), and a line whose number of
tab-separated cells differs from line 1 is TABULAR_INVALID (`tsvCellCount`).
Each offending line is reported with its `tsvLine`, in line order, after any
header problem and before the sidecar is checked, and the rows are not
validated further. `{headers, rows}` input is checked too (`rows[0]` is the
header row; a row with no cells is an empty line); `Map` input has no rows to
check. `BidsTsvFile` has the new fields `emptyLines` and `badCellCounts`.
These checks run only on a file that carries HED: a column or sidecar entry
named `HED`, or a cell of the first non-empty line of text data that is `HED`
once trimmed, so an empty first line does not hide the HED column. A blank
column name alone does not.
- A partnered library loaded from its unmerged file now gives the same schema
as its merged file:
- its elements carry `inLibrary`, so redeclaring a tag of the standard partner
Expand Down
127 changes: 93 additions & 34 deletions src/bids/tsvParser.js
Original file line number Diff line number Diff line change
Expand Up @@ -17,34 +17,40 @@ import { isPaddedName, stripNameWhitespace } from '../utils/string'

const stripBOM = (str) => str.replace(/^\uFEFF/, '')
const normalizeEOL = (str) => str.replace(/\r\n/g, '\n').replace(/\r/g, '\n')
const isContentfulRow = (row) => row && !/^\s*$/.test(row)
// A line with a tab has at least two columns, so it is a row even when every cell is blank or padded. An empty line
// or a line of blanks without a tab is skipped, as pandas reads it in hed-python (skip_blank_lines).
const isTableRow = (row) => row.includes('\t') || isContentfulRow(row)

/**
* Split TSV contents into cells: the first row, which is the header, then one array per row after it.
* Split TSV contents into lines (specification 3.2.10).
*
* The header cells are the column names and the other cells the values, all exactly as written (specification
* 3.2.9.1 and 3.2.10): nothing is trimmed, as for Map and {headers, rows} input. Header and data rows are found by
* the same rule, so a data line such as ` \t` is a row and its padded value is checked.
* Line breaks are \r\n, \r or \n. One line break at the end of the contents ends the last line and is not itself a
* line; any other empty line is kept, so that it can be reported. Line 1 is always the first line, even when empty.
*
* @param {string} contents The contents of a TSV file.
* @returns {string[][]} The rows of the file.
* @returns {string[]} The lines of the file; empty when the contents are empty.
*/
const splitRows = (contents) => {
const lines = stripBOM(normalizeEOL(contents)).split('\n')
const headerIndex = lines.findIndex(isTableRow)
if (headerIndex < 0) {
const splitLines = (contents) => {
const text = stripBOM(normalizeEOL(contents))
if (text.length === 0) {
return []
}
const dataRows = lines
.slice(headerIndex + 1)
.filter(isTableRow)
.map((str) => str.split('\t'))
return [lines[headerIndex].split('\t'), ...dataRows]
const lines = text.split('\n')
if (lines[lines.length - 1] === '') {
lines.pop()
}
return lines
}

/**
* Split TSV contents into cells: the first line, which is the header, then one array per line after it.
*
* The header cells are the column names and the other cells the values, all exactly as written (specification
* 3.2.9.1 and 3.2.10): nothing is trimmed, as for Map and {headers, rows} input. Every line is kept, so that row
* index i is file line i + 1; an empty line becomes the row `['']` (see {@link findRowProblems}).
*
* @param {string} contents The contents of a TSV file.
* @returns {string[][]} The rows of the file.
*/
const splitRows = (contents) => splitLines(contents).map((line) => line.split('\t'))

/**
* Determine whether a TSV cell is a missing value: empty or `n/a` once any leading or trailing blank, tab, line feed
* or carriage return is removed.
Expand Down Expand Up @@ -155,31 +161,84 @@ export function findDuplicateHeaders(contents) {
}

/**
* Return the cells of the first header line of TSV contents (see {@link splitRows}), without splitting the other lines.
* Return the cells of the first line of TSV contents, the header (see {@link splitRows}), without splitting the other
* lines.
*
* Line breaks are \r\n, \r or \n, as in {@link splitRows}, so the result equals the header row that
* {@link parseTSV} uses. The cells are not trimmed.
* {@link parseTSV} uses. The cells are not trimmed. An empty first line is one blank column name, `['']`.
*
* @param {string} contents The contents of a TSV file.
* @returns {string[]} The header cells; empty when the contents have no header line.
* @returns {string[]} The header cells; empty when the contents are empty.
*/
export function parseTSVHeader(contents) {
const text = stripBOM(contents)
const lineBreak = /\r\n|\r|\n/g
let start = 0
while (start <= text.length) {
lineBreak.lastIndex = start
const match = lineBreak.exec(text)
const line = text.slice(start, match ? match.index : text.length)
if (isTableRow(line)) {
return line.split('\t')
}
if (!match) {
break
if (text.length === 0) {
return []
}
const match = /\r\n|\r|\n/.exec(text)
return text.slice(0, match ? match.index : text.length).split('\t')
}

/**
* Determine whether a cell of the first non-empty line of TSV contents is `HED` once its leading and trailing
* whitespace is removed.
*
* Such a file carries HED even when a malformed first line hides its HED column (specification 3.2.10), so it is
* validated and the malformed line reported.
*
* @param {string} contents The contents of a TSV file.
* @returns {boolean} Whether the first non-empty line names a HED column.
*/
export function firstNonEmptyLineHasHed(contents) {
const line = splitLines(contents).find((text) => text !== '')
return line !== undefined && line.split('\t').some((cell) => stripNameWhitespace(cell) === 'HED')
}

/**
* The lines of a tabular file that break the line rules of specification 3.2.10.
* @typedef {Object} TsvRowProblems
* @property {number[]} emptyLines The 1-based numbers of the empty lines other than the first.
* @property {{line: number, cells: number, expected: number}[]} badCellCounts Each other line whose number of
* tab-separated cells differs from the first line's.
*/

/**
* Find the lines other than the first that are empty or have a different number of cells than the first line
* (specification 3.2.10, TABULAR_INVALID).
*
* An empty line is reported as an empty line only, not also as a wrong cell count. For text contents, an empty line
* is a line with no characters; for {headers, rows} data, a row with no cells, since a row `['']` is one empty
* cell. In {headers, rows} data, rows[0] is the header row and the expected count is the number of headers.
*
* @param {string|OldParsedTSV} data The contents of a TSV file, or parsed TSV data in the old BIDS format.
* @returns {TsvRowProblems} The empty lines and the lines with a wrong cell count, by 1-based file line number.
*/
export function findRowProblems(data) {
const problems = { emptyLines: [], badCellCounts: [] }
let expected
let rows
if (typeof data === 'string') {
const lines = splitLines(data)
if (lines.length === 0) {
return problems
}
start = lineBreak.lastIndex
expected = lines[0].split('\t').length
rows = lines.map((line) => (line === '' ? null : line.split('\t')))
} else {
expected = data.headers.length
rows = data.rows.map((row) => (row.length === 0 ? null : row))
}
return []
rows.forEach((row, index) => {
if (index === 0) {
return
}
if (row === null) {
problems.emptyLines.push(index + 1)
} else if (row.length !== expected) {
problems.badCellCounts.push({ line: index + 1, cells: row.length, expected })
}
})
return problems
}

/**
Expand Down
40 changes: 38 additions & 2 deletions src/bids/types/tsv.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,15 @@
import isPlainObject from 'lodash/isPlainObject'

import { BidsFile } from './file'
import { convertParsedTSVData, findHeaderProblems, isMissingTsvValue, parseTSV, parseTSVHeader } from '../tsvParser'
import {
convertParsedTSVData,
findHeaderProblems,
findRowProblems,
firstNonEmptyLineHasHed,
isMissingTsvValue,
parseTSV,
parseTSVHeader,
} from '../tsvParser'
import { BidsSidecar } from './json'
import BidsHedTsvValidator from '../validator/tsvValidator'
import { IssueError } from '../../issues/issues'
Expand Down Expand Up @@ -40,6 +48,26 @@ export class BidsTsvFile extends BidsFile {
*/
blankColumns

/**
* The 1-based numbers of the empty lines other than the first (TABULAR_INVALID).
* @type {number[]}
*/
emptyLines

/**
* The lines other than the first whose number of cells differs from the first line's (TABULAR_INVALID).
* @type {{line: number, cells: number, expected: number}[]}
*/
badCellCounts

/**
* Whether a cell of the first non-empty line of text TSV data is `HED` once trimmed, so the file carries HED even
* when its first line is malformed (specification 3.2.10).
* @type {boolean}
* @private
*/
_hedInFirstNonEmptyLine

/**
* HED strings in the "HED" column of the TSV data.
* @type {string[]}
Expand Down Expand Up @@ -70,6 +98,12 @@ export class BidsTsvFile extends BidsFile {
this.paddedColumns = padded
this.blankColumns = blank
this.duplicateColumns = duplicate
// A Map has no row structure, so it has no empty lines or wrong cell counts to report.
const { emptyLines, badCellCounts } =
tsvData instanceof Map ? { emptyLines: [], badCellCounts: [] } : findRowProblems(tsvData)
this.emptyLines = emptyLines
this.badCellCounts = badCellCounts
Comment thread
VisLab marked this conversation as resolved.
Resolved
this._hedInFirstNonEmptyLine = typeof tsvData === 'string' && firstNonEmptyLineHasHed(tsvData)
this.hedColumnHedStrings = this._parseHedColumn()
this.mergedSidecar = new BidsSidecar(name, this.file, mergedDictionary, defManager)
}
Expand Down Expand Up @@ -127,10 +161,12 @@ export class BidsTsvFile extends BidsFile {
* @returns {boolean}
*/
get hasHedData() {
// A padded "HED" header is still the HED column, so the file is checked and its header reported.
// A padded "HED" header is still the HED column, so the file is checked and its header reported. So is a "HED"
// cell in the first non-empty line of text data, which an empty first line would otherwise hide (3.2.10).
return (
this.parsedTsv.has('HED') ||
this.paddedColumns.some((column) => stripNameWhitespace(column) === 'HED') ||
this._hedInFirstNonEmptyLine ||
this.mergedSidecar.hasHedData
)
}
Expand Down
15 changes: 15 additions & 0 deletions src/bids/validator/tsvValidator.js
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,21 @@ export class BidsHedTsvValidator extends BidsValidator {
return
}

// An empty line or a wrong cell count: report every offending line and do not validate the rows (3.2.10).
const rowIssues = [
...this.tsvFile.emptyLines.map((line) => ({ line, issue: generateIssue('emptyTsvLine', { line }) })),
...this.tsvFile.badCellCounts.map(({ line, cells, expected }) => ({
line,
issue: generateIssue('tsvCellCount', { line, cells, expected }),
})),
].sort((a, b) => a.line - b.line)
if (rowIssues.length > 0) {
for (const { line, issue } of rowIssues) {
this.errors.push(BidsHedIssue.fromHedIssue(issue, this.tsvFile.file, { tsvLine: line }))
}
return
}

// Validate the BIDS sidecar if it exists and return if there are errors
if (this.tsvFile.mergedSidecar) {
const issues = this.tsvFile.mergedSidecar.validate(this.hedSchemas)
Expand Down
10 changes: 10 additions & 0 deletions src/issues/data.js
Original file line number Diff line number Diff line change
Expand Up @@ -513,6 +513,16 @@ const issueData = {
level: 'error',
message: issueMessageTemplate`Column ${'position'} in the header of the tabular file has a blank name.`,
},
emptyTsvLine: {
hedCode: 'TABULAR_INVALID',
level: 'error',
message: issueMessageTemplate`Line ${'line'} of the tabular file is empty; only the first line may be empty.`,
},
tsvCellCount: {
hedCode: 'TABULAR_INVALID',
level: 'error',
message: issueMessageTemplate`Line ${'line'} of the tabular file has a tab-separated cell count of ${'cells'}, but the first line has a count of ${'expected'}.`,
},
paddedTsvValue: {
hedCode: 'TABULAR_INVALID',
level: 'error',
Expand Down
Loading
Loading