Skip to content

feat: recognize .arrows as an Arrow IPC stream extension - #23960

Open
ianmcook wants to merge 3 commits into
apache:mainfrom
ianmcook:arrows-extension
Open

feat: recognize .arrows as an Arrow IPC stream extension#23960
ianmcook wants to merge 3 commits into
apache:mainfrom
ianmcook:arrows-extension

Conversation

@ianmcook

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

.arrows is the recommended extension for Arrow IPC stream files, but DataFusion only recognizes .arrow during file-format inference.

What changes are included in this PR?

  • Recognize .arrows using the existing Arrow IPC reader.
  • Add a constant for the Arrow IPC stream extension.
  • Add test coverage for querying an .arrows file.

Are these changes tested?

Yes

Are there any user-facing changes?

Yes. Users can directly query Arrow IPC stream files using the recommended .arrows extension.

@github-actions github-actions Bot added core Core DataFusion crate common Related to common crate labels Jul 29, 2026
@codecov-commenter

codecov-commenter commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.84%. Comparing base (541caab) to head (6105948).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23960      +/-   ##
==========================================
- Coverage   80.84%   80.84%   -0.01%     
==========================================
  Files        1096     1096              
  Lines      373936   373951      +15     
  Branches   373936   373951      +15     
==========================================
+ Hits       302313   302319       +6     
- Misses      53584    53589       +5     
- Partials    18039    18043       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment on lines +926 to +937
let ext = ext.to_lowercase();
self.file_formats
.get(&ext)
.or_else(|| {
if ext == DEFAULT_ARROW_STREAM_EXTENSION.trim_start_matches('.') {
self.file_formats
.get(DEFAULT_ARROW_EXTENSION.trim_start_matches('.'))
} else {
None
}
})
.cloned()

@neilconway neilconway Aug 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems a bit regrettable to add special-case logic for Arrow IPC into a generic code path. The current approach also doesn't handle COPY TO for Arrow IPC streams, which presumably might be useful.

What if we added ArrowStreamFormatFactory as a parallel to ArrowFormatFactory? Each format would then have a single extension, and we'd have support for both input and output in a way that fits into the existing API cleanly. The two Arrow format implementations might duplicate code, but maybe we can refactor their implementations to share code where it makes sense. wdyt?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate core Core DataFusion crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support .arrows extension for Arrow IPC stream files

3 participants