Skip to content

New Presplit utility for Fate table - #6471

Open
ArbaazKhan1 wants to merge 5 commits into
apache:mainfrom
ArbaazKhan1:accumulo-4979
Open

New Presplit utility for Fate table#6471
ArbaazKhan1 wants to merge 5 commits into
apache:mainfrom
ArbaazKhan1:accumulo-4979

Conversation

@ArbaazKhan1

Copy link
Copy Markdown
Contributor

closes issue #4979

Added support for pre-splitting Fate tables via accumulo admin command.

  • --numSplits option generates split points using the same partitioning logic as in FateManager.getDesiredPartitions. The logic ensures that pre-split boundires align with how the manger expects transaction IDs to be distributed. The split utility produces raw UUID row strings which should allow it be also be used by scan_ref.

  • --splitsFile flag writes the generated splits to a file.

Comment thread server/base/src/main/java/org/apache/accumulo/server/util/adminCommand/Fate.java Outdated
Comment thread server/base/src/main/java/org/apache/accumulo/server/util/adminCommand/Fate.java Outdated
Comment thread server/base/src/main/java/org/apache/accumulo/server/util/adminCommand/Fate.java Outdated
Comment thread server/base/src/main/java/org/apache/accumulo/server/util/adminCommand/Fate.java Outdated

@ctubbsii ctubbsii left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it useful to have this utility focus exclusively on the Fate table? Would it be better to generalize it to generate splits for any table, or is this necessarily tied to the fate table for some reason?

@ArbaazKhan1

Copy link
Copy Markdown
Contributor Author

Is it useful to have this utility focus exclusively on the Fate table? Would it be better to generalize it to generate splits for any table, or is this necessarily tied to the fate table for some reason?

The split generation logic is tied to the Fate table. The algorithm is modeled after getDesiredPartitions, to produce UUIDs whose string representations align with how Fate maps transaction Ids to tablet rows. I was trying to keep the work done scoped to just Fate since in original issue was asking to generate splits for the FATE table that evenly distribute FATE operations across tablets.

However; I could look into creating a more general purpose split logic if that's more desirable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create utility to presplit the fate table

4 participants