Request For Browser Extension: Make Firefox Look Like The Google Spider

[JoeN]

Ever have this happen?  You search for something and Google gives you a nice synopsis of exactly what you are looking for and a link and you follow the link and immediately hit a paywall?  I was looking for "28nm gate capacitance" out of curiosity.  Google gives this synopsis:

[PDF] 24.7 A High-Performance, High-Density 28nm eDRAM ... - IEEE Xplore
ieeexplore.ieee.org/iel5/6123666/6131464/06131608.pdf
Metal-Insulator-Metal (MIM) capacitor and integrated into. 28nm High-K Metal Gate (HKMG) logic technology. This. eDRAM memory features an HKMG CMOS  ...

When you follow the link:

http://ieeexplore.ieee.org/iel5/6123666/6131464/06131608.pdf

An immediate paywall.  And the words above are not even hidden in the HTML.  How did Google read the PDF in order to index it properly?

I don't know, but I bet the web server is giving up the content if it thinks you are Google.  But unless it is going off of known IPs, the only way to know what is at the end of an HTTP connection is by the headers, which can be anything you want them to be.

Hmmm.....     

[Jeroen3]

One can submit the pdf to google to index without the crawler finding it himself.

[NottheDan]

Look for a User Agent switcher addon for your FF and use it to switch your browser's identification to on of the robots when needed.

https://addons.mozilla.org/en-GB/firefox/addon/user-agent-switcher/

[amyk]

Most of these use IP; I believe there was a way using one of Google's services as a proxy, but I don't know if that route has been blocked now. Even Google Translate used to work. Do some searching for more information on that... of course, for articles like that one, check SciHub/Libgen.

[tom66]

Google penalises websites that present different content to their spiders...if you find a paywalled article try copying the title into Google and you'll often get access to it. If you don't, then Google should be seeing the same summary you get.