Initialize data version control for managing test images

[weiji14]

Description of proposed changes

Using a data version control package called dvc to manage the PNG test images in our repository!

In a nutshell, we will only store the hash of the PNG on GitHub (in a *.png.dvc file) and the actual PNG will be stored at https://dagshub.com/GenericMappingTools/pygmt (see e.g. https://dagshub.com/GenericMappingTools/pygmt/src/data_version_control/pygmt/tests/baseline/test_logo_on_a_map.png).

List of commands ran so far on this 'data_version_control' branch (will update as I test things out):

Background (only needed to be done once for this repo)

Installing DVC for developing PyGMT

cd ~/path/to/pygmt
conda activate pygmt
make install
pip install dvc

DVC init

dvc init # creates .dvcignore file and .dvc/ folder
# remove .dvc/plots folder as won't be used
# git add only the .dvcignore, .dvc/.gitignore and .dvc/config file
git add .dvcignore .dvc/.gitignore .dvc/config
git commit -m "Initialize data version control"

Setup DVC remote

dvc remote add upstream https://dagshub.com/GenericMappingTools/pygmt.dvc # updates .dvc/config file with remote URL
dvc remote default upstream  # set default dvc remote to 'upstream'

---

When a new test is being written (Everyone will need to run)

DVC remote authentication config (changes .dvc/config.local)

dvc remote modify upstream --local auth basic
dvc remote modify upstream --local user "$DAGSHUB_USER"
dvc remote modify upstream --local password "$DAGSHUB_PASS"  # generate token at https://dagshub.com/user/settings/tokens

Generate new baseline PNG images

pytest --mpl-generate-path=baseline pygmt/tests/test_logo.py  # generate new baseline images
mv baseline/*.png pygmt/tests/baseline/
dvc add pygmt/tests/baseline/test_logo.png pygmt/tests/baseline/test_logo_on_a_map.png
git rm -r --cached 'pygmt/tests/baseline/test_logo.png'
git rm -r --cached 'pygmt/tests/baseline/test_logo_on_a_map.png'
# echo "/test_*.png" > pygmt/tests/baseline/.gitignore

Push images to DVC remote

dvc add pygmt/tests/baseline/test_logo.png pygmt/tests/baseline/test_logo_on_a_map.png
git add pygmt/tests/baseline/.gitignore pygmt/tests/baseline/test_*.png.dvc
git commit -m "Add test_logo_*.png into DVC"

dvc push --remote upstream
git push

---

Pull PNG images from DVC remote (needed for Github Actions CI)

dvc status  # should report any files 'not_in_cache'
dvc pull  # pull down files from DVC remote cache (fetch + checkout)
pytest pygmt/tests/test_logo.py  # run pytest as per usual

Note: Some of the dvc commands may not be necessary if we dvc install some git hooks so that dvc add/checkout/push actions happen simply with git add/checkout/push, but need to try this out a bit more. One forseeable con with this solution is that dvc may (?) run even for commits where no PNG images change, so an extra layer of slowness.

References:

• https://dvc.org/doc/start/data-versioning
• https://hackernoon.com/how-to-get-started-with-data-version-control-dvc-nn2n31bo
• https://dagshub.com/blog/configure-a-dvc-remote-without-a-devops-degree/
• https://dagshub.com/blog/datasets-should-behave-like-git-repositories/
• https://dagshub.com/blog/data-version-control-tools/
• https://dagshub.com/blog/data-science-pull-requests/

Addresses #963

Reminders

• Run make format and make check to make sure the code follows the style guide.
• Add tests for new features or tests that would have caught the bug that you're fixing.
• Add new public functions/methods/classes to doc/api/index.rst.
• Write detailed docstrings for all functions/methods.
• If adding new functionality, add an example to docstrings or tutorials.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash commands are:

• /format: automatically format and lint the code
• /test-gmt-dev: run full tests on the latest GMT development version

[willschlitzer]

Should this update requirements-dev.txt to include dvc?

[weiji14]

Should this update requirements-dev.txt to include dvc?

We don't have a requirements-dev.txt file anymore as of #812 😄. I wanted to put it in environment.yml but ran into some CI issues, might need to finish off #1033 first.

Oh, and will need to document how to use dvc somewhere too. I'm still getting my head around how to make it as easy as possible for everyone.

[weiji14]

Ok, I've added a "Using data version control (dvc)" subsection to CONTRIBUTING.md at 6bd7ba9. Have a read, see if it makes sense, and maybe try and test this Pull Request out! Feel free to ask any questions too.

[seisman]

I see that the dagshub repo is a mirror of the github repo. Do all changes in the github repo will be synchronized to the dagshub repo? How often does the dasghub repo do the synchronization.

I just pushed a few commits to the new-display branch (https://github.com/GenericMappingTools/pygmt/tree/new-display), but I don't see the changes in dagshub repo (https://dagshub.com/GenericMappingTools/pygmt/src/new-display).

[weiji14]

/test-gmt-dev

[weiji14]

Sorry, had to restart the regular GMT6.1.1/Python 3.9 test, dvc push didn't work because I had my credentials on my laptop instead of my uni computer 😅 Edit: Double oops, I forgot to switch back from GMT 6.2.0 to GMT 6.1.1 in my pygmt conda env so the baseline images are wrong, hold on...

I tried triggering the GMT dev version test at #1036 (comment), and looks like test_logo*.py failed (see https://github.com/GenericMappingTools/pygmt/actions/runs/662613609) because dvc pull wasn't run. Actually, the workflow run did not have the dvc pull step at all, and it seems like the changes at f3aa3c5 were not checked out, perhaps a security feature to prevent people changing the Github Actions yaml workflow in a Pull Request from doing unintended things?

[seisman]

I saw you just migrated test_image.png to dvc. We also need to update the test_image.py script:

• Use SI units
• Use long-form aliases, not single-character letters.

[seisman]

My last concern is, how can reviewers review the image changes?

When someone runs dvc push and opens a PR, can we see the images in DAGsHUB, regardless of the 8-hr mirror interval? Do we have to checkout the branch and run dvc pull to download the images?

[weiji14]

My last concern is, how can reviewers review the image changes?

When someone runs dvc push and opens a PR, can we see the images in DAGsHUB, regardless of the 8-hr mirror interval? Do we have to checkout the branch and run dvc pull to download the images?

No need to do a dvc pull locally. You can view the new images directly from the Pull Request branch on DAGsHub. E.g. for this 'data_version_control' branch, go to https://dagshub.com/GenericMappingTools/pygmt/src/data_version_control/pygmt/tests/baseline/test_logo_on_a_map.png and just see the image there:

[seisman]

I just added a new image to my testing branch (commit b8bdb7c in #1071). However, I can't preview the image in DAGsHub: https://dagshub.com/GenericMappingTools/pygmt/src/fix-test-basemap-polar/pygmt/tests/baseline

[seisman]

We may also need to update the maintenance guides about how to review PRs with dvc images. We can do it after we have more experience with the new workflow.

[weiji14]

Will let the full tests run before merging 🚀, thanks for the review and testing things out thoroughly!

We may also need to update the maintenance guides about how to review PRs with dvc images. We can do it after we have more experience with the new workflow.

Yep, will do that in a follow up PR. I'll post a comment on #963 to summarize the new workflow for everyone else on the team.

[core-man]

@weiji14 Thank you so much. This is a really amazing improvement. I already used it in #1065.