Add option to download images

[64bitpandas]

I'm using this tool to mirror some of my Substack posts to my website, and as part of that process I'd really like to host my own images instead of having them link to the Substack CDN!

In case this will help someone else, here's a PR 🙂

Here's a list of some tweaks I made to get that to happen:

• Add an --images flag that will download images for all posts being scraped into a substack_images/ folder
• Add an option to download a single post (by passing in a --url in the format https://example.substack.com/p/postname
• When downloading images, Substack nests them like [![alt](/path)](/path). Change these to just be ![alt](/path) so clicking on the images doesn't link to itself.
• Add some tests, to prove to myself this code works the way I expect it

As a bonus, the progress bars reflect image downloads (since they can take a while)! As an example:

Scraping posts: 100%|██████████| 2/2 [00:30<00:00, 15.00s/post]
  Downloading images for test-post: 100%|██████████| 7/7 [00:14<00:00, 2.00s/image]
  Downloading images for another-post: 100%|██████████| 4/4 [00:08<00:00, 2.00s/image]

[milahu]

commit based on this PR: milahu@5811bb5

[milahu]

filename can be wrong, because substackcdn.com/image/fetch can

• change the image size (crop) (ex: 1536x1024 → 1456x971)
• change the image format (ex: png → webp)

also, when a post uses the same original image in different sizes
then all image versions use the same filename (filename collisions)

example

cd  $(mktemp -d)

$ wget 'https://substackcdn.com/image/fetch/$s_!UkZH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2c2394c-643c-4533-8d74-734252ae02ac_1536x1024.png'

$ mv * d2c2394c-643c-4533-8d74-734252ae02ac_1536x1024.png.webp

$ python
>>> import urllib.parse
>>> urllib.parse.unquote('https://substackcdn.com/image/fetch/$s_!UkZH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2c2394c-643c-4533-8d74-734252ae02ac_1536x1024.png')
'https://substackcdn.com/image/fetch/$s_!UkZH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https://substack-post-media.s3.amazonaws.com/public/images/d2c2394c-643c-4533-8d74-734252ae02ac_1536x1024.png'

$ wget 'https://substack-post-media.s3.amazonaws.com/public/images/d2c2394c-643c-4533-8d74-734252ae02ac_1536x1024.png'

$ identify *
d2c2394c-643c-4533-8d74-734252ae02ac_1536x1024.png PNG 1536x1024 1536x1024+0+0 8-bit sRGB 2.56234MiB 0.000u 0:00.000
d2c2394c-643c-4533-8d74-734252ae02ac_1536x1024.png.webp WEBP 1456x971 1456x971+0+0 8-bit sRGB 207452B 0.000u 0:00.000

simple fix: always download the original image

pro: simple
con: the original images can be huge! in my case 10x larger

def resolve_image_url(url: str) -> str:
    """Get the original image URL."""
    # https://substackcdn.com/image/fetch/xxx/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fxxx
    if url.startswith("https://substackcdn.com/image/fetch/"):
        # substackcdn.com returns a compressed version of the original image
        url = "https://" + unquote(url.split("/https%3A%2F%2F")[1])
    return url

            url = resolve_image_url(url)
            filename = sanitize_image_filename(url)