juust ~ php oddities » simplexml http://www.juust.org Link theory and search engine optimization Thu, 19 Jan 2012 09:39:32 +0000 en hourly 1 http://wordpress.org/?v=3.2.1 google suggest scraper (php & simplexml) http://www.juust.org/index.php/google-suggest-scraper-php-simplexml/2011/12/ http://www.juust.org/index.php/google-suggest-scraper-php-simplexml/2011/12/#comments Mon, 19 Dec 2011 00:08:12 +0000 admin http://www.juust.org/?p=1834 Continue reading →]]> Today’s goal is a basic php Google Suggest scraper because I wanted traffic data and keywords for free.

Before we start :

google scraping is bad !

Good People use the Google Adwords API : 25 cents for 1000 units, 15++ units for keyword suggestion so they pay 4 or 5 dollar for 1000 keyword suggestions (if they can find a good programmer which also costs a few dollars). Or they opt for SemRush (also my preference), KeywordSpy, Spyfu, and other services like 7Search PPC programs to get keyword and traffic data and data on their competitors but these also charge about 80 dollars per month for a limited account up to a few hundred per month for seo companies. Good people pay plenty.

We tiny grey webmice of marketing however just want a few estimates, at low or better no cost : like this :

data num queries
google suggest 57800000
google suggestion box 5390000
google suggest api 5030000
google suggestion tool 3670000
google suggest a site 72700000
google suggested users 57000000
google suggestions funny 37400000
google suggest scraper 62800
google suggestions not working 87100000
google suggested user list 254000000

Suggestion autocomplete is AJAX, it outputs XML :

< ?xml version="1.0"? >
   <toplevel>
     <CompleteSuggestion>
       <suggestion data="senior quotes"/>
       <num_queries int="30000000"/>
     </CompleteSuggestion>
     <CompleteSuggestion>
       <suggestion data="senior skip day lyrics"/>
       <num_queries int="441000"/>
     </CompleteSuggestion>
   </toplevel>

Using SimpleXML, the PHP routine is as simple as querying g00gle.c0m/complete/search?, grabbing the autocomplete xml, and extracting the attribute data :

  1.         if ($_SERVER['QUERY_STRING']=='') die('enter a query like http://host/filename.php?query');
  2.  $contentstring = @file_get_contents("http://g00gle.c0m/complete/search?output=toolbar&amp;q=".urlencode($kw));  
  3.    $content = simplexml_load_string($contentstring );
  4.  
  5.         foreach($content-&gt;CompleteSuggestion as $c) {
  6.             $term = (string) $c-&gt;suggestion-&gt;attributes()-&gt;data;
  7.             //note : traffic data is sometimes missing  
  8.             $traffic = (string) $c-&gt;num_queries-&gt;attributes()-&gt;int;
  9.             echo $term. " ".$traffic . "
  10. " ;
  11.  }

I made a quick php script that outputs the terms as a list of new queries so you can walk through the suggestions :

The source is as text file up for download overhere (rename it to suggestit.php and it should run on any server with php5.* and simplexml).

]]> http://www.juust.org/index.php/google-suggest-scraper-php-simplexml/2011/12/feed/ 4 ga api sample : get pageviews http://www.juust.org/index.php/google-analytics-api-sample-get-pageviews/2009/05/ http://www.juust.org/index.php/google-analytics-api-sample-get-pageviews/2009/05/#comments Wed, 13 May 2009 14:58:01 +0000 juust http://www.juust.org/?p=504 Continue reading →]]> I was going to put that online : how to get the pageviews out of the google analytics api, using simplexml and php. Google use three namespaces in the output file which make it less easy accessible, so here’s a quick sample of how to get your sites pageviews out of it :

  1. //ids           = site identifier (from the site data feed)
  2. //metrics     = what i want to see
  3. //start-date
  4. //end-date
  5.  
  6. $feedUri = "https://www.google.com/analytics/feeds/data?ids=ga:10516419&metrics=ga:pageviews&start-date=2009-04-01&end-date=2009-05-01";    
  7.  
  8.  $curl = curl_init();
  9.  curl_setopt($curl, CURLOPT_URL, $feedUri);
  10.  curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 3);
  11.  curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
  12.  
  13.        $headers[] = "Authorization: GoogleLogin auth=".$Authtoken;
  14.  
  15. //for authtoken : see previous post
  16.  curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
  17.  curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);
  18.  curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
  19.  curl_setopt($curl, CURLOPT_VERBOSE, 1);
  20.  
  21. //get the string containing the xml file
  22.  $gA = curl_exec($curl);

the feed has three namespaces (atom, opensearch and dxp/analytics), a simple way is accessing the ENTRY tags (from the Atom namespace), in that tag is one DXP: line and that has the answer to the question.

<dxp:metric confidenceInterval=’0.0′ name=’ga:pageviews’ type=’integer’ value=’755′/>

  1. //load the string into a simple xml object
  2.  $feed = simplexml_load_string($gA);
  3.  
  4. //take the atom namespace
  5.  $children =  $feed->children('http://www.w3.org/2005/Atom');
  6.  
  7. //take the entry tags
  8.  $parts = $children->entry;
  9.  foreach ($parts as $entry) {
  10.  
  11.         //from the entry tag,
  12.         //access the dxp namespace
  13.   $dxp = (object) $entry->children('http://schemas.google.com/analytics/2009');
  14.  
  15.         //METRIC contains the answer to the question
  16.         //grab from the tag METRIC the attribute VALUE
  17.                 echo   (string) $dxp->metric->attributes()->value;
  18.  
  19.         }

Important is using the (string) typecast, normally simplexml returns a simplexml object, when you force a string type, it gives the actual metric ga:pageview value attribute as number.

]]> http://www.juust.org/index.php/google-analytics-api-sample-get-pageviews/2009/05/feed/ 0