How to view HTTP headers

HTTP headers are the part of the webpage you don't see in your browser (usually); the special data describing the page to your browser software. HTTP headers are wehere you'd put redirects, information about whether a file is a PNG / HTML page / RAR archive, and where you say if the browser should open the file or present it as a download - as well as many other things. The full details are in RFC2616.

I'm going to cover three methods of dealing with headers today - all quick, simple and powerful.

Perl's LWP "HEAD" command

This is a commonly-found *nix command line tool, very simple in its operation, and likely already on your system. To use it, you simply enter "HEAD " at the command prompt, where is the full address (e.g. including http:) that you want to check.

If you don't have it, you can install this as root by whipping up a CPAN console (perl -MCPAN -e shell) and running i LWP::Simple - then just follow the prompts, and opt to install the GET/HEAD aliases.

Quick and simple - but it won't report on redirects, just the final page, and you need root to install it in most circumstances.

Tamper Data

You can use Firefox to examine headers, alter HTTP requests, and find out precisely what every page is doing with this masterpiece of a plugin. If you're using LiveHttpHeaders, I suggest you immediately exchange it for Tamper Data - just as light, and much more powerful. To use this, simply enable Tamper Data in Firefox, click "Start tampering" in the new window, and then visit the page you're interested in; you don't want to go playing with the server immediately, so simply accept the request, and ignore further requests. Tada - more information than you'll ever need - including full request and response headers for everything on the page! This is also great for finding out FLV URLs and other things hidden by Flash apps.

Tamper Data has many additional functions, including page load optimisation, and far too much to cover here. Just check out this tutorial for a taster.

Command-line cURL header script

For a very verbose, quick, minimal and to the point solution, create a file called header somewhere on your *nix server, and fill it thusly (perhaps amending the PHP executable path):

#!/usr/bin/env php

$url = $argv[1];

function url_header($url) {

global $useragent;

global $timeout;

if ($useragent == "") {$useragent = "Mozilla 8.0 +http://seorant.blogspot.com";}

if ($timeout == "") {$timeout = 20;}

$ch = curl_init();

curl_setopt ($ch, CURLOPT_URL, $url);

curl_setopt ($ch, CURLOPT_USERAGENT, $useragent);

curl_setopt ($ch, CURLOPT_HEADER, 1);

curl_setopt ($ch, CURLOPT_NOBODY, 1);

curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);

curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);

curl_setopt ($ch, CURLOPT_TIMEOUT, $timeout);

curl_setopt ($ch, CURLOPT_MUTE, 1);

$result = curl_exec ($ch);

curl_close($ch);

return $result;

}

$useragent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040614 Firefox/0.8';

echo $url."\n\n"; flush();

echo url_header($url);

?>

Make the file executable (chmod u+x header) and run with ./header

This will include the full details of redirects as and when they're performed, and should get a reponse more akin to that a browser receives when compared to the LWP method, which uses a different useragent string.

Breaking through Google's 1000 result limit

Sometimes we want to get a huge list of URLs from a search engine. For example, you might want to find all the pages linking back to you. However, Google won't return anything past 1000 results; see this search for "south"; we're on page 99, with 10 results per page - and if you scroll to the bottom, you'll see that page numbers run out. So - how can we get more results?

One technique for measuring the propreties of words and phrases in text is n-gram analysis. This counts the numbers of single-word (unigram), two-word (bigram), three-word etc (up to n-word) phrases in a text.

E.g.: given the phrase "The cat sat on the mat", we have the following unigrams:

• The - 2
• Cat - 1
• Sat - 1
• On - 1
• Mat - 1

And the following bigrams:

• The cat - 1
• cat sat - 1
• sat on - 1
• on the - 1
• the mat - 1

So how does this help us? Well, n-gram counts of large amounts of text tell us what the most common words we'll find are. Once we ignore stopwords (the search engines will), we get terms that we can use as part of a search query to split up the results. If we know that "fish" and "knee" are common words, we could run two queries:

• link:mysite.com knee
• link:mysite.com fish

This would return 2000 links to mysite.com. Of course, some of these pages will have the words both "fish" and "knee" on, so there'll be some kind of overlap, but we'll still get say 1700-1900 useful unique sites. Once we have a good list to exploit, we can take the top 1000 results for our query divided up with 40 different ngrams to get a good 25000-35000 results - way past the 1000 limit usually imposed.

Implementing something like this would probably look like:

URL table - with unique URL field

query = "link:competitor.com"   
for ngram in ngrams   
   for page = 1, page < 10, page ++   
       offset = (page - 1) * 10   
       results = getGoogleResults(query + " " + ngram, offset)   
       for result in results   
           sql("insert ignore into URL values(?)", result)   

Of course, this is massively open to optimisation; post in the comments if you have any questions.

To help you out, I've included a list of over 700 of the most common English unigrams, derived from a good web-based source. If you're interested in versions in other languages, or a longer list, let me know why and I'll see what I can do. Here's the link:

http://seorant.ath.cx/unigrams.txt

Glasgow stopword list

MSN Search API in PHP

Here's some code for accessing the MSN Live Search API from PHP. You can get a developer key here by going to Configure Applications, Create and Manage Application IDs. You'll need a Microsoft Passport to get your key.

You'll need the PHP5 SOAP library enabled (make sure

extension=php_soap.dll

is uncommented in your php.ini).

To run a search on MSN and scrape results, set up a variable called

$msnsoapkey

with your key in it as a string, then call this function with three parameters:

1. A string of your query - "site:fish.com" or "seo ranter", for example


2. How many results you'd like, up to a maximum of 50


3. The offset for the start of results; 0 means give results from number 1 to $querysize; 100 means from 101 to 100+$querysize.

/////
// fetches from search.msn.com results for the query $query, using the API

function fetchMSNResults($query, $querysize, $offset) {
    global $msnsoapkey;
    static $msnsoap;

    // only generate this WSDL proxy once
    if (!isset($msnsoap)) {
        $msnsoap = new soapclient("http://soap.search.msn.com/webservices.asmx?wsdl"); 
    }

    $request = array(
        'Request' => array(
            'AppID' => $msnsoapkey,
            'Query' => $query,
            'CultureInfo' => 'en-US',
            'SafeSearch' => 'Off',
            'Flags' => '',
            'Requests' =>  array(
                'SourceRequest' => array(
                    'Source' => 'Web',
                    'Offset' => $offset,
                    'Count' => $querysize,
                    'ResultFields' => 'Url'
                )
            )
        )
    );

    $response = $msnsoap->Search($request);

    foreach($response->Response->Responses->SourceResponse->Results->Result as $hit) {
        $results[] = $hit->Url;
    }

    return $results;
}

It will return an Array() of Strings, each one containing a result URL. Easy! This is contained by a wrapper function in my code library, which manipulates the

$offset 

and

$querysize

to allow for any number of results to be returned at the courtesy of the MSN API; you can figure one out pretty easily if you need.

1400+ PHP Link Directories, catalogued for you

So, in advance of a little scripting, here's a list of over 1400 installations of the PHP Link Directory. I'd like to give more - there are easily 20k out there - but sadly Yahoo! and Google's search APIs don't like queries past result number 1000. In fact, they positively hurl their dummies out of their respective cradles.

The spreadsheet containing the results has the following information:

• Submit URL


• Cost of submission


• Reciprocal link code


• Whether or not the site uses a captcha

phpld-20070630.xls

It's spartan, but functional, and certainly open to further use. I was surprised by how many of these directories are wide open; further code is coming.

There's no source with this post as the abomination that created the data was truly awful, and probably still will be next time round. I even spent time in Excel updating individual entries; big to-do list entries include: add homepage pr, make backlink scraping code more accurate, de-dupe by domain and not hostname, add express submission price column, add flag to detect if unique sessions are required for submission, and detect PHPld version.

This list's probably very abusable. For example, those lovely chaps at the PHP Link Directory could abuse it to check that everyone's bought a license. In fact, this list should only contain the cheapskates that haven't paid to remove the link to the software creators, but hey, who am I to judge.

Teoma / Ask scraping code

Alas, Teoma's search API is down. If it ever returns, you can find great Teoma Search API documentation. For the meantime, here's code to do scraping for you:

/////
// fetches URLs from Teoma results for the query $query
// string $query is the search query
// int $querysize is the number of results needed
// int $offset says where the results should begin from (put 11 to get results 11-20)

function fetchTeomaResults($query, $querysize, $offset) {

    $page = 1 + intval($offset / 10);
    $requestUrl = 'http://www.ask.com/web?q='.urlencode($query).'&page='.$page;

    $oldua = ini_set('user_agent', 'Please bring back http://xml.teoma.com/.');
    $response = file_get_contents($requestUrl);
    ini_set('user_agent', $oldua);

    preg_match_all('|<a id="r[0-9]+_t" href="(.+?)"|', $response, $matches);

    $results = array_slice($matches[1], 0, $querysize);

    return $results;
}

It's dirty, nasty, and many other mean things. For example,

• anyone sane wouldn't enable fopen wrappers;


• the user agent is a little non-standard;


• there's no HTTP From: header;


• it's quite possibly against Ask TOS;


• $offset should be a factor of ten, because I can't be bothered writing preference setting code and controlling the number of results per page isn't controllable via URL (as far as I can see)


• scraping is never a permanent solution

- and other things. Bring back xml.teoma.com!


Use of fetchTeomaResults is usually wrapped up by another function, for accessing SERPs in general and aggregating results. The function signature conforms to this - else we could just specify a page instead of an offset.

Enjoy.