Skip to content

Add filter_glob and exclude_glob to fs.walk#464

Merged
althonos merged 2 commits intoPyFilesystem:masterfrom
atollk:issue_459_371
Jul 12, 2022
Merged

Add filter_glob and exclude_glob to fs.walk#464
althonos merged 2 commits intoPyFilesystem:masterfrom
atollk:issue_459_371

Conversation

@atollk
Copy link
Member

@atollk atollk commented Mar 22, 2021 •
edited
Loading

Type of changes

  • New feature
  • Documentation / docstrings

Checklist

  • I've run the latest black with default args on new code.
  • I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate.
  • I've added tests for new code.
  • I accept that @PyFilesystem/maintainers may be pedantic in the code review.

Description

Fixes #371. Fixes #459 .
Added parameters filter_glob and exclude_glob to fs.walk.Walker to perform selection on the walked resources based on their global path.

@coveralls
Copy link

coveralls commented Mar 22, 2021 •
edited
Loading

Coverage Status

Coverage decreased (-0.3%) to 94.794% when pulling d85d9db on atollk:issue_459_371 into 50b1c99 on PyFilesystem:master.

@althonos althonos self-assigned this Mar 24, 2021
@atollk atollk mentioned this pull request Apr 21, 2021
Merged
4 tasks
althonos
althonos requested changes Apr 22, 2021
View reviewed changes
Copy link
Member

@althonos althonos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job once again 😉

My largest concern is about whether it would be possible to remove the _translate function completely, and use _PatternMatcher in fs.glob as well to reduce code duplication. Otherwise, it's mostly smaller stuff.

I'll have a closer look at the actual files at some point because GitHub diffs are really not helping here, but the first read didn't raise any red flag 😄

althonos
althonos requested changes Apr 30, 2021
View reviewed changes
@atollk
Copy link
Member Author

atollk commented Apr 30, 2021

Alright, so, after taking a step back and having a look at the larger picture again, it made more sense to me to put all that logic into fs.glob instead of fs.wildcard, so I reverted the change in the latter. I also don't think the two can (or should) be merged anymore, due to the specific behavior of glob. Most notably /**/ matches /, which is not what you would expect from a wildcard.

Either way, several of your previous comments became obsolete through this and I implemented the rest (like partial instead of lambda).

@atollk
Copy link
Member Author

atollk commented May 26, 2021

Hi. Any update on this?

@atollk
Copy link
Member Author

atollk commented Jul 3, 2021

I just rebased this branch onto the newest master, so it's ready to merge now imo.

@althonos
Copy link
Member

althonos commented Jul 6, 2021

@atollk : I think you rebased against your old master without pulling from PyFilesystem/pyfilesystem2 first, the code still have merges from a prior PR.

@atollk
Copy link
Member Author

atollk commented Jul 24, 2021

@althonos Should be fine now.

lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
fs/base.py
Comment on lines +1705 to +1711
patterns (list, optional): A list of patterns, e.g.
``['*.py']``, or `None` to match everything.
Copy link
Contributor

@lurch lurch Jul 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at just the doc-strings, it's hard to see how match_glob() is different from match() ?

Copy link
Member Author

@atollk atollk Sep 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The summaries (first sentences) differ: Check if a name matches any of a list of wildcards. and Check if a path matches any of a list of glob patterns..

lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
fs/glob.py
Comment on lines +153 to +163
if path and path[0] != "/":
path = "/" + path
Copy link
Contributor

@lurch lurch Jul 25, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the same as https://docs.pyfilesystem.org/en/latest/reference/path.html#fs.path.abspath ?

Copy link
Member Author

@atollk atollk Sep 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only with the current implementation of abspath, not with the general idea.

lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
lurch
lurch reviewed Jul 25, 2021
View reviewed changes
@atollk atollk requested review from althonos and lurch September 5, 2021 11:01
@wayconA
Copy link

wayconA commented Jun 21, 2022

Please promote this PR I'm desperately in need of that feature.
@atollk
@lurch
@althonos

@althonos
Copy link
Member

althonos commented Jun 21, 2022

@wayconA

Sorry for not getting this merged sooner, I've been under lots of work for my research projects.

@atollk has definitely done a lot of work on this, and the reason I've been stalling this PR a bit is probably because I was not 100% happy with the API, but honestly I can't come up with something better, so I will merge this ASAP, and we can always refactor in the pyfilesystem3 release (which I swear is not vaporwave).

@lurch
Copy link
Contributor

lurch commented Jun 22, 2022

Looks like there's a bunch of conflicts that need to be resolved first though 😕

@Wallaku
Copy link

Wallaku commented Jun 26, 2022 •
edited
Loading

What needs to be done for this PR to be approved?
I'm determined to assist if needed.
@atollk
@lurch
@althonos

atollk added 2 commits July 9, 2022 13:06
Added filter_glob and exclude_glob to fs.walk.Walker.
a844a9b 
These extend the class by an option to include/exclude resources by their entire path, not just its last component.
To do so, fs.wildcard had to undergo a rework to remove the dependency on the `re` module.
Unit tests were added for all new/changed code.
Changes from code review.
d85d9db
@atollk
Copy link
Member Author

atollk commented Jul 9, 2022

done

@Wallaku
Copy link

Wallaku commented Jul 10, 2022

GREAT!!! Thank you very much @atollk

Shall it be merged now?
@althonos @lurch

@althonos althonos merged commit 691deae into PyFilesystem:master Jul 12, 2022
@althonos
Copy link
Member

althonos commented Jul 12, 2022

Great job @atollk, and thanks for the review @lurch.

@Wallaku
Copy link

Wallaku commented Jul 13, 2022

Great! When the new release will be released? @althonos @lurch

@althonos
Copy link
Member

althonos commented Jul 13, 2022

I'd just like #542 to be merged first and then we'll get a new release 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@althonos althonos althonos approved these changes

@lurch lurch Awaiting requested review from lurch

Assignees

@althonos althonos

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add a global filter to fs.walk.Walker filter_dirs in walk.dirs behaves unintuitive and its description is a bit misleading

6 participants

@atollk @coveralls @althonos @wayconA @lurch @Wallaku