Skip to content

Fix Windows path filtering for FTP files - #374

Open
jay23606 wants to merge 1 commit into
AlertaDengue:mainfrom
jay23606:fix/windows-ftp-path-filter
Open

jay23606 wants to merge 1 commit into
AlertaDengue:mainfrom
jay23606:fix/windows-ftp-path-filter

Conversation

@jay23606

Copy link
Copy Markdown

Summary

Normalize Windows path separators before filtering files by client in PySUS.query, so FTP files are not dropped on Windows.

Fixes #372

Validation

  • python -m pytest pysus/tests/api/test_client.py -q — 51 passed.
  • Regression gate — the focused test failed on both clean-base worktrees and passed on both fix worktrees.
  • Black, isort, Flake8, and git diff --check passed.

@ArthurFachel

Copy link
Copy Markdown

Thanks for this. I was looking at #372 too and found the same pattern in a second place that this PR doesn't touch yet: _fetch_origin_direct in pysus/api/_impl/source.py (around line 266), which is the source="origin" path.

files = [f for f in files if str(f.path).startswith(prefix)]

With only the client.py change, PySUS.query(client=...) keeps the Windows paths, but _fetch_origin_direct then filters them out again, so the empty result remains on that path.

A regression test in the same style, for TestFetchOriginDirect in pysus/tests/api/test_source.py. It fails on master and passes once that line normalizes the path:

def test_prefix_filter_windows_path(self):
    """Regression for #372: backslash paths must still match prefix."""
    from pathlib import PureWindowsPath

    from pysus.api._impl.source import _fetch_origin_direct

    matching = self._file(
        PureWindowsPath("public/data/ftp/sinasc/DNSE2010.parquet")
    )
    other = self._file(PureWindowsPath("elsewhere/b.parquet"))
    pysus = self._make_pysus([matching, other])
    result = self._run(
        _fetch_origin_direct(
            pysus, "sinasc", None, "SE", 2010, None, "FTP", None,
            False, False, download=False, _bag=True,
        )
    )
    assert result == [matching]

Feel free to fold it into this PR. If it helps, a small shared helper (e.g. as_posix() for PurePath, str() otherwise) would keep both filters consistent. Happy to open a follow-up PR instead if you'd rather keep this one focused.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pysus.ftp.* returns an empty FileBag on Windows (origin filter uses str(path))

2 participants