Open
Conversation
This adds a "pc" fsspec filesystem implementation, which lets us
insert "pc::" in an fsspec URL and automatically sign it when loading
it with an fsspec client.
The primary motivation is integration with fsspec's filesystem where
users would need to call `planetary_computer.sign` in multiple places
1. Once for loading the index JSON files
2. Once for signing the reference filesystem templates
Which lets us replace this:
```python
>>> result = xr.open_dataset(
... fsspec.get_mapper(
... "reference://",
... fo=planetary_computer.sign(requests.get(planetary_computer.sign("https://deltaresreservoirssa.blob.core.windows.net/references/reservoirs/chirps.json")).json()),
... ),
... engine="zarr",
... consolidated=False,
... )
```
With this:
```python
>>> result = xr.open_dataset(
... "pc::reference::pc::https://deltaresreservoirssa.blob.core.windows.net/references/reservoirs/CHIRPS.json",
... engine="zarr",
... consolidated=False,
... )
```
TomAugspurger
commented
Aug 8, 2022
| fo = planetary_computer.sign(fo) | ||
| self.fo = fo | ||
| self.target_fs = fsspec.filesystem(self.target_protocol, **self.target_options) | ||
| if isinstance(self.target_fs, fsspec.implementations.reference.ReferenceFileSystem): |
Author
There was a problem hiding this comment.
I'm not a fan of this block.
The reference filesystem has the idea of "template" URLs, which are the NetCDF files in blob storage. We want to sign those URLs before anyone attempts to access data via this reference filesystem.
It seems that the reference filesystem's __init__ calls a method at https://github.com/fsspec/filesystem_spec/blob/7effb83e8ab31010ec5796c14193b5fcd5774e05/fsspec/implementations/reference.py#L149, which does a lot of work including in-lining the template URLs in the reference (url, start, end) tuples. Unfortunately, we don't have a way to update the template URLs before the tuples are built, so we have to do it again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds a "pc" fsspec filesystem implementation, which lets us
insert "pc::" in an fsspec URL and automatically sign it when loading
it with an fsspec client.
The primary motivation is integration with fsspec's filesystem where
users would need to call
planetary_computer.signin multiple placesWhich lets us replace this:
With this:
Still just a POC. I need to figure out