feat: recognize .arrows as an Arrow IPC stream extension - #23960
feat: recognize .arrows as an Arrow IPC stream extension#23960ianmcook wants to merge 3 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #23960 +/- ##
==========================================
- Coverage 80.84% 80.84% -0.01%
==========================================
Files 1096 1096
Lines 373936 373951 +15
Branches 373936 373951 +15
==========================================
+ Hits 302313 302319 +6
- Misses 53584 53589 +5
- Partials 18039 18043 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| let ext = ext.to_lowercase(); | ||
| self.file_formats | ||
| .get(&ext) | ||
| .or_else(|| { | ||
| if ext == DEFAULT_ARROW_STREAM_EXTENSION.trim_start_matches('.') { | ||
| self.file_formats | ||
| .get(DEFAULT_ARROW_EXTENSION.trim_start_matches('.')) | ||
| } else { | ||
| None | ||
| } | ||
| }) | ||
| .cloned() |
There was a problem hiding this comment.
It seems a bit regrettable to add special-case logic for Arrow IPC into a generic code path. The current approach also doesn't handle COPY TO for Arrow IPC streams, which presumably might be useful.
What if we added ArrowStreamFormatFactory as a parallel to ArrowFormatFactory? Each format would then have a single extension, and we'd have support for both input and output in a way that fits into the existing API cleanly. The two Arrow format implementations might duplicate code, but maybe we can refactor their implementations to share code where it makes sense. wdyt?
Which issue does this PR close?
Rationale for this change
.arrowsis the recommended extension for Arrow IPC stream files, but DataFusion only recognizes.arrowduring file-format inference.What changes are included in this PR?
.arrowsusing the existing Arrow IPC reader..arrowsfile.Are these changes tested?
Yes
Are there any user-facing changes?
Yes. Users can directly query Arrow IPC stream files using the recommended
.arrowsextension.