Fetch several Parquet row groups when appropriate

[severo]

See #213 (comment)

When fetching a row group, we only fetch from the first row to the last one:

hyperparam-cli/src/lib/tableProvider.ts

Lines 46 to 47 in 0d67c6e

	const rowStart = groupEnds[groupIndex - 1] ?? 0
	const rowEnd = groupEnds[groupIndex]

But, when sorting along a column with small data, it can result in many small requests, triggering a rate limit.

This issue aims at fetching several consecutive Parquet row groups at once when appropriate, reducing the number of requests while receiving larger responses.

eg: instead of 2,000 requests with 7KB responses, we prefer 200 requests with 70KB responses (possible only if enough row groups are consecutive).

[platypii]

Yea this has been one of the biggest sources of slowdowns especially for sorting. I've been thinking about how to modify the AsyncBuffer abstraction to better handle many small range requests.

I think we might want something that accepts multiple ranges, like:

interface AsyncBuffer {
  byteLegnth: number
  slice(start: number, end: number): ArrayBuffer
  slices?: (ranges: { start: number, end: number }[]) => ArrayBuffer[]
}

That way the hyparquet library can request many small ranges in one request. IF the request are all to the same-origin, http range request allow for multiple ranges to be specified. IF the requests are cross-origin multi-range requests are not allowed. In the latter case we can either translate it to multiple independent http requests, OR if the total size is small, combine into a single larger range request. This would provide a nice place to insert heuristics like "if total range is less than 100kb, just fetch all ranges as one request" (or whatever).

[severo]

Oh, I discover multi-range requests. But as it's not available cross-origin, it's pretty limited.

Do you want me to work on this, or do you prefer to handle it?

[platypii]

I'll take a look at making the changes in hyparquet and the AsyncBuffer abstraction.

That won't necessarily solve the problem you mention here though. If the CLI tableProvider still requests individual row groups, but each row group is small, then hyparquet can't really help. So in that sense, maybe we do want to make changes here too. If you want to take a look at modifying this to do something like fetch multiple consecutive rowgroups IF the total size would be less than <threshhold> then I'm in favor of you trying that, especially if it solves some of the HF issues.

[severo]

If you want to take a look at modifying this to do something like fetch multiple consecutive rowgroups IF the total size would be less than then I'm in favor of you trying that, especially if it solves some of the HF issues.

If I understand well, hyparquet will fetch group by group anyway, ie one request per group (https://github.com/hyparam/hyparquet/blob/1ca8b4701424b95b475b01a1576978c4b5eab895/src/read.js#L45), so it would have to be implemented in hyparquet, not in hyperparam-cli.

Please tell me @platypii if I misunderstood something.