Learn how to spoof user agent as Googlebot to bypass paywall, so you can read online articles without an account.
Another helpful content from the series, DiGiztal Bypass Paywalls Tips.
This method works is because there are paywalled websites, that allow unrestricted access to Google’s search bot (Googlebot) to maintain SEO ranking.
The trick is you switch your browser’s user-agent to Googlebot’s in the developer tools.
In other words, you impersonate the user-agent as Googlebot to get through the paywall.
This can be done in a number of different ways in different browsers.
Meaning, each time you come to a paywalled article, then you do the following steps as shown below.
It is like this, "How to disable CSS with WebTools to bypass paywall".
Mimic user agent as Googlebot
There are two key ways to switch the browser user agent settings.
Do it manually with the browsers, or install a browser extension or plugin.
Switch user-agent with browser
Google Chrome browser:
This is how to do it:
At the blocked article, press F12 key to bring out the Developer Tools (DevTools).
Click "Network", then click "More network conditions" button to open the Network panel.
It pops up at the bottom of the page
Click the tick symbol next to "Use browser default" to remove it.
Simultaneously a slot appears, click it, and select "Googlebot" from the list.
You can see the remarks below: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Reload the article first, then only close the DevTools.
Now you can read the full article, without having to subscribe.
NOTE: Alternatively, you can also use this: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla Firefox browser:
For Firefox, you need to use its Configuration Editor (about:config page) to do it.
Go to Firefox browser, type about:config in the addres bar and press "Enter" key.
You will get a warning message, but click the "Accept the Risk and Continue" button.
Type general.useragent.override in the space provided.
Paste the following line inside the box.
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Click the tick symbol.
Reload or refresh the paywalled page you’re trying to access, and the paywall should be gone.
Switch user-agent with plugin
This is the simplest way to change the browser user-agent.
First install this User-Agent Switcher Chrome plugin.
After installed, pin it to the toolbar.
When at the blocked page, click User-Agent Swticher icon at the toolbar.
A context menu drops down, click "Googlebot (Google's spider)".
Reload the blocked page, and it should reappear as a full article.
For Firefox users, you can use User-Agent Switcher.
Similarly, install the plugin and pin to toolbar.
At the paywalled article, click the plugin icon at toolbar, then click it and a menu context appears.
Look for "Bot", then click "Google Bot".
Reload the blocked page to bypass its paywall.
Some of the sites that works with this trick include The Atlantic, Wired, New York Magazine, etc.
DISCLAIMER: Using this impersonating Googlebot method could get your IP address blocked or banned by the web application owner.
RELATED TIPS: