add support for google adsense user agents

[nitishmittal17]

Added support for google adsense user agents. Fix for issue - #167

[thoop]

Since we remove <script> tags from the page and I know some of those check for the ad tracking script to approve ads, I'd want to be really careful here about adding those.

We haven't heard of any issues caused by not having those user agents being checked in the middleware. Did you experience any issues there?

[jirkadev]

That's a good point, if that's true that you are removing <scripts> just adding User-agent: Mediapartners-Google probably would not solve the crawling issue with AdSense.

As they wrote here: https://support.google.com/adsense/answer/99376?hl=en&ref_topic=1348129

The crawler will attempt to access URLs only where our ad tags are implemented.
Only pages displaying Google ads should be sending requests to our systems and being crawled.

It seems to me, that AdSense will not work properly - Adsense crawler will not be able to crawl a website rendered with client-side JavaScript framework (e.g. Angular, Meteor with Blaze).

Do you guys have some ideas? Maybe create a special way how to handle AdSense crawler in prerender?

[jirkadev]

Since we remove <script> tags from the page and I know some of those check for the ad tracking script to approve ads, I'd want to be really careful here about adding those.

We haven't heard of any issues caused by not having those user agents being checked in the middleware. Did you experience any issues there?

My site couldn't be approved by AdSense, because of the error no valuable content.
Adsense bot saw only a bit of initial HTML (body with one script tag, I am using Meteor).

Only after I add those user agents to prerender-node, and triggered the review process again my site was approved.

It's a good point with having those <script> tags removed.
Hard to say now when my site is approved if they check something.