Faking the Referer Header in PhantomJS

When loading a web page with PhantomJS is it possible to fake the referrer by setting the page.customHeaders property to include “Referer”. However, this will be sent to all page objects (images, CSS etc…), which is not the desired result. The workaround in the PhantomJS API documentation is incorrect. Read on to find out how to fix this.

Faking the Referer

You fake the Referer header by setting the page.customHeaders property like so:

var webPage = require('webpage');
var page = webPage.create();
 
page.customHeaders = {
  "Referer": "https://www.google.co.uk"
};

If you look at the server access logs for the page request, the initial referrer will be Google, as we want, but all page objects get that same referrer. What we want is for them to get the web page as the referrer, which is what a normal web browser would do.

PhantomJS’ API Documentation Is Wrong

At the time of writing, the customHeaders documentation advises that the recommended workaround is to use page.onInitialized to reset the customHeaders property. I tried this and it didn’t work. All of the page objects got Google as the referrer.

page.onLoadStarted

Through a little trial-and-error I found that page.onLoadStarted is the property that you should use to reset the custom headers for page objects.

page.onLoadStarted = function() {
    page.customHeaders = {};
};

The web server access logs correctly showed that the page request had Google as the referrer, but subsequent requests for objects like CSS and images had the page itself as the referrer.

The Complete Script

Putting it all together, you will want to create your page object, set the customerHeaders property, and then set page.onLoadStarted:

var webPage = require('webpage');
var page = webPage.create();
 
page.customHeaders = {
  "Referer": "https://www.google.co.uk"
};
 
page.onLoadStarted = function() {
    page.customHeaders = {};
};
 
page.open(url, function (status) {
...
}