Distributed point evaluation optimization #411

Current distributed point evaluation algorithm implemented in #379 can be described as:
Setup:

• Construct a global search structure on each rank. At this moment, this can be an implicitly represented structured grid.
• On each rank, identify the global search cells intersecting its local mesh partition bounding box using a generalized version of the grid search implementation.
• Use all-to-all communication to build a mapping from each global search cell to its candidate mesh ranks.

Find:

• Given a query point, attempt local point search and field evaluation on the requesting rank.
• If local search fails, locate the query point in the global search structure and retrieve its candidate mesh ranks.
• Send the query point to all candidate ranks for local point search and field evaluation.
• Receive the results from the candidate ranks and determine whether a valid result has been found. The actual owned rank will be cached.

There are a few potential improvements worth discussing.

• The current field already contains ghost layers, but we do not take advantage of them during point evaluation. (Actually, we need to explicitly filter them out in several places) It would be worth exploring ways to use ghost data.
• When multiple candidate ranks successfully locate the same query point, the current tie-breaking strategy selects the rank with the lowest owning ID. Not sure if it is always the best choice. At least we should support a switchable tie breaker API in point search.
• The bounding-box intersection approach can generate a large number of candidate ranks. This increases memory usage, communication overhead, and point evaluation cost. Any strategy that reduces the number of candidates would be great. It could be a different global search structure. For example, an adaptive global search grid with finer resolution near partition boundaries? It could also be some heuristic to rank the candidates.
• There are several potential load-balancing challenges in this approach. If we know that point evaluations mainly happen in certain regions or on certain ranks, can we balance the communication and evaluation workload more effectively?