Before we start :

google scraping is bad !

Good People use the Google Adwords API : 25 cents for 1000 units, 15++ units for keyword suggestion so they pay 4 or 5 dollar for 1000 keyword suggestions (if they can find a good programmer which also costs a few dollars). Or they opt for SemRush (also my preference), KeywordSpy, Spyfu, and other services like 7Search PPC programs to get keyword and traffic data and data on their competitors but these also charge about 80 dollars per month for a limited account up to a few hundred per month for seo companies. Good people pay plenty.

We tiny grey webmice of marketing however just want a few estimates, at low or better no cost : like this :

Suggestion autocomplete is AJAX, it outputs XML :

< ?xml version="1.0"? >
   <toplevel>
     <CompleteSuggestion>
       <suggestion data="senior quotes"/>
       <num_queries int="30000000"/>
     </CompleteSuggestion>
     <CompleteSuggestion>
       <suggestion data="senior skip day lyrics"/>
       <num_queries int="441000"/>
     </CompleteSuggestion>
   </toplevel>

Using SimpleXML, the PHP routine is as simple as querying g00gle.c0m/complete/search?, grabbing the autocomplete xml, and extracting the attribute data :

I made a quick php script that outputs the terms as a list of new queries so you can walk through the suggestions :

The source is as text file up for download overhere (rename it to suggestit.php and it should run on any server with php5.* and simplexml).

Just give us your endpoint and we’ll take it from there, sparky!

I was going to make one of these tools to scrape google and conjur a full blog out of nowhere, as Christmas special, RedHat Seo. The rough sketch has arrived , far from perfect, but it does produce a blog and don’t even look too shabby. I scraped a small batch of posts off of blogs, keeping the links intact and adding a tribute links. I hope they will pardon me for it.

structure

I use three main classes,

…and two support classes

Target blogs have three texts,

routine

The BlogMaker class grabs a result list (up to 1000 urls per phrase) from Google, extracts the urls and stores them in SerpResult, scrapes the urls and extracts the entry divs, stores div-entries in the WPContent class (that has some basic functions to sanitize the text), and uses the BlogTarget-definitions to post it up blogs with xml-rpc.

usage

notes

• xml-rpc needs to be activated explicitly on the wordpress dashboard under settings/writing.
• categories must be present in the blog
• url file must be writeable by the server (777)

It seems wordpress builds the sitemap as background process, the standard google xml sitemap plugin wil attempt to build in the cache (takes anywhere between 2 and 10 seconds), and apart from building a sitemap the posts also get pinged around. Giving the install 10 to 20 seconds between posts allows for all the hooked in functions to be completed.

period

That’s about all,
consider it gpl, I added some comments in the source but I will not develop this any further. A mysql backed blogfarm tool (euphemistically called ‘publishing tool’) is more interesting, besides, I am off to the wharves to do some painting.

if you use it, send some feedback,
merry christmas dogheads

For a quick test I dive into the money segment, got the top 70 search phrases of October off of 7Search and for each get the top 100 results, resulting in a mysql database with 7.000 url’s on 3500 domains.

Then I try three cross-sections to extract the top 25 domains in the segment :

• (100 – position) / 100 per result
• (100 – position) / 100 per result, +1 for first 10, +1 for first 6, +1 for first three
• AOL experience based percentage of search volume

rating by rank

If I rate the sites’ urls based on (100 – position) / 100 per result.

place 1 = 0.99, place 100 = 0, that yields this table :

That makes Amazon top dog for all phrases containing ‘money’. I know the first pages get most traffic, and that first method doesn’t express that, so I change the routine and do my old trick.

rating by rank, bonus for front page

(100 – pos) / 100, and
for place 1-3 3 points
for place 4-6 2 points
for place 7-10 1 points

That gives this table :

That still does not show what domains actually get the traffic I need for conversion.

It is easy to score in a low traffic niche and anyone can be a winner on long tails, “alabama ski resort”, but long tails only get you a few hits (and wonder if they are serious?). And these results doesn’t give me a clue what I can spend (or what my competition would be willing to spend) on the actual traffic spots.

So I am going to estimate what traffic every site gets. I need search volumes and percentages for the serp ranks. I grabbed the search volumes per phrase from 7Search. For percentages, Aaron Wall quotes an old AOL source on the average click through rate of search engine result pages per spot on the first page :

That was what i was looking for. Given these percentages I can estimate the traffic any spot on the search engine result front page in Google would generate, and that yields a more realistic table of the money segment :

That shows which sites actively target and get the traffic in the segment, a site like amazon doesn’t show in the top of the last table.

One way of testing the validity of the estimates is comparing with alexa rank, doing a quick SeoQuake toolbar check :

That’s roughly correct based on old AOL percentages (well done, Aaron) and Oktober’s search volumes. It does not work for youtube, msn and others that have an alexa rank based on the entire domain. Being a search engine traffic estimate it doesn’t cover your ‘audience’ (returning visitors on bookmarks and referrals off of other sites (which are included in the alexa ranking) or other segments.

Problogger for example, according to that calculation would get about 6K hits through Google but the Alexa rank indicates it has ~~160K hits a month. That indicates high return visits and direct traffic, a high usability and I think they get traffic from other segments (‘blog’ ?).

The use of it

Why did I start doing this ? To estimate a budget for niche penetration.

If I were interested in a niche and have a 1% conversion at $100,- per year per conversion, what would I be able to spend to get Bidvertisers’ traffic in the search engines ? With their traffic and my conversion, 730.000/100 = 7300 conversions per month =y 88.000 a year @100/conversion = 8.8 million dollar gross revenue on search engine traffic conversions.

If my marketing budget is 20% of gross sales, for a year project that amounts to 1.7 million dollar. If I already had 300K hits my budget would be a million.

That (roughly) answers my question.

I have my doubts about the use of a tool like this, it could function as quick way to scan a segment, make a ‘heatmap’ and pinpoint the soft spots that are easy to penetrate and get a foothold, but that would require some stiff programming and a lot of switches, as well as processing a lot more segments. Nice project for the winter.

I wanted an automated one with a history that sends me an email every day, so I started putting on together but I got distracted because the gethost server was shut down and my moms Mary Chapel event blog was on it. My old site was also on it, with my crappy blog, but losing that ain’t half as bad as losing yer moms Mary Chapel blog, that is bad karma.

So I got her a domain and put it on one of my accounts, installed a new blog and made sure it ranks number one in google again. Apart from that I was putting together a scraper and still needed some cron jobs for the scraper scripts.

I found a nice free cron job site, standard 5 jobs a day. It’s free and it works flawless, no hassles. Very nice.

Once I had that I remembered I still had to finish that serp thing as well, so I finished the basic routines today. Still needs some testing, and it could use the yahoo and msn serps. I’ll grab them from a wordpress widget and add some language options, after that it should be a fine tool.

I’ll put the scripts up for download in two or three weeks or something.

update: Google updated their API in oktober (almost at the time I wrote these posts) and this code fails as it still based on the V1 APi. You can access the whole WT: toolset namespace (including sitemaps, verification) through the V2 API now, but you need to send a version id along with your request, that is handled in the new Zend 1.7 download.

The Problem

I can add 32.000 blogs on a standard WordPressMu install. How do I add 32.000 subdomains, verify them and add their sitemaps to Google Webmaster, without having to go to the webmaster page about 96.000 times ?

The solution

Integrating Google Webmaster Tools API into my WordPress Mu install.

What is it worth ?

If registering and verifying a site and adding a sitemap takes 5 minutes per domain, at E12,- per hour, that makes it 96.000 euros and 4 labor years for 32.000 sites. Writing a script is worth E96.000,- and saves me four years of mindless drone work, so that is well worth having a look at.

Software : Zend

Zend gData is a php framework that is programmed to handle Google Data. Their ClientLogin routine isn’t very flexible and they haven’t covered GWT Api yet, so I’ll have to hack some routines together.

After getting stonewalled by the zend program a few times, I went searching and ended up on ngoprekweb who have a nice post on ClientLogin authorization for the blogger api. Eris Ristemena uses a modified Zend ClientLogin, very nice work. I installed the adapted classes and tried that one to get through the ClientLogin, and it paid off.

The good stuff : Gwt api access

I am not interested in the blogger stuff though, I want access to GWT Google Webmaster Tools, so I worked Eris Ristemena’s blogger routine around a little.

1. ';
2.         }
3.    }
4.    elseif ( $resp['response']=='captcha' )
5.    {
6.      echo 'Google requires you to solve this CAPTCHA image';
7.      echo '

';

1.       echo '

‘; echo ‘Answer :
‘; echo ‘
‘; echo ‘
‘; echo ‘

1. ';
2.      exit;
3.    }
4.    else
5.    {
6.      // there is no way you can go here, some exceptions must have been thrown
7.    }
8.  
9.  } catch ( Exception $e )  {
10.    echo $e->getMessage();
11.  }

(I added https://www.google.com/accounts/ to the captcha image source, otherwise it keeps drawing blanks.)

Zend uses a “HttpClient” for the connection to Google, and a gData class (usually the main ‘feed’, blogs, sites) that you use to do basic data manipulation. All feed entries are an atom format with a custom namespace.

Now I am going to add a domain. In my add_site function I put an XML Atom together to post (using the post() function of the gData class) to the sites feed url, and the Google API does the rest :

function add_site($domain, $client) {

1.   $xml='';
2.   $xml.='';
3.   $xml.='';
4.   $fdata = new Zend_Gdata($client);
5.   $result=$fdata->post($xml,"https://www.google.com/webmasters/tools/feeds/sites/");
6.   return $result;
7. }

In the main routine I pass the domain and the running httpclient to the add_site() function :

if ( $resp['response']=='authorized' )

1.     {
2.       $client = Zend_Gdata_ClientLogin::getHttpClient($resp['auth']);
3.       echo add_site('test.blacknorati.com', $client);
4.     }

Cool. That saves me up to 32.000 site registrations. The rest of it is still greek to me, but this part functions. Next week : more nonsense (verify the site, add a sitemap, and integrate it in the blog creation function of wordpress mu).

1) about the blogger function : I tried to list the blogger posts with the ngoprekweb php code, but it seems blogger use a different string these days to identify the blog in gData, the id is returned as “tag:blogger.com-blabla-(blogid)” and you want the last part to access the blogs post atom feed :

$idText = split('-', $item->id());

1.         $blogid = $idText[2];

(modified from the Zend 1.6.1 codebase)

foreach ($feed as $item) {

1.         echo '<a href="'.$item->link(">' . $item-&gt;title() . '</a>';
2.  
3. $idText = split('-', $item-&gt;id());
4.        $blogid = $idText[2];
5.  
6.        $feed1 = $gdata-&gt;getFeed("http://www.blogger.com/feeds/$blogid/posts/summary");
7. //…
8. }

]]> http://www.juust.org/index.php/zend-php-and-google-webmaster-tools-api/2008/10/feed/ 4 http://www.juust.org/index.php/serp-tool-2008/2008/09/ http://www.juust.org/index.php/serp-tool-2008/2008/09/#comments Sun, 21 Sep 2008 23:55:09 +0000 juust http://www.juust.org/?p=152 Continue reading →]]> I never got around to building a serp tool with a mysql backend, one on a cronjob with an email option.
This weekend I got an OSWD free template, then it at least looks like a website. I develop a lot easier when it has a template. Filling in the blanks.

Often if you program a rough sketch, in time it sort of develops itself. If you don’t start nothing gets done.

I am developing it on juust.org/serp/, once its finished i’ll put one on trismegistos.net and put the source up for download.

It’s flimsy but the idea of a serp minisite with emailer option is commercially attractive.

It’s also important for the blog to have a ‘serp’ page in the menu, and a serp page should have serp tools, in abundance, all kinds of them.

]]> http://www.juust.org/index.php/serp-tool-2008/2008/09/feed/ 0 http://www.juust.org/index.php/blogger-auto-poster/2008/08/ http://www.juust.org/index.php/blogger-auto-poster/2008/08/#comments Thu, 28 Aug 2008 23:23:15 +0000 juust http://www.juust.org/?p=130 Continue reading →]]> I needed to get my new linkdirectory’s pages indexed and crawled and google needs some stimulation.

So I take a blogger subdomain,
and a 700 category php link directory
and make a table PLD_TAGCLOUD(CAT_ID, POSTED, TAG, FULLPATH)

CREATE TABLE `PLD_TAGCLOUD` (
`ID` BIGINT( 11 ) NOT NULL ,
`CAT_ID` DOUBLE NOT NULL ,
`POSTED` DOUBLE NOT NULL ,
`LEVEL` DOUBLE NOT NULL ,
`TAG` VARCHAR( 250 ) NOT NULL ,
`FULLPATH` VARCHAR( 250 ) NOT NULL ,
PRIMARY KEY ( `ID` )
) ENGINE = MYISAM

I fill the table with a recursive tree traversal on the category table where “tag” is the ‘title’ field,
and I get the FULLPATH url by using the domain root and the path generated by traversing the tree :

1.  
2. function connect() {
3.  $DB_USER =  "";
4.  $DB_PASSWORD = "";
5.  $DB_HOST = "";
6.  $DB_DATA = "";
7.  $link =  mysql_connect($DB_HOST, $DB_USER, $DB_PASSWORD) or $error = mysql_error();
8.  if (!$link) {
9.      echo $error;
10.   exit;
11.  } else {
12.     mysql_select_db($DB_DATA, $link) or $error = mysql_error();
13.  return $link;
14.  }
15. }
16.  
17. $link = connect();
18. $del="DELETE FROM `PLD_TAGCLOUD`";
19. $mydel = mysql_query($del, $link) or die(mysql_error());
20. @mysql_close($link);
21.  
22. $root='http://links.trismegistos.net';
23. $content .= read(0, $root, 1);
24.  
25. function read($rootid, $pathid, $thislevel) {
26.  $link = connect();
27.  $myqry = "SELECT * FROM `PLD_CATEGORY` WHERE `PARENT_ID`='".$rootid."'";
28.  $myres = mysql_query($myqry, $link) or die(mysql_error());
29.  if(mysql_num_rows($myres)&lt;1) return;
30.  while($row=mysql_fetch_assoc($myres)) {  
31.   $thispath= $pathid ."/".$row['TITLE_URL'];
32.   $link2 = connect();
33.   $add="INSERT INTO `PLD_TAGCLOUD` (`CAT_ID`, `LEVEL`, `TAG`, `FULLPATH`) VALUES ('".$row['ID']."', '".$thislevel."', '".htmlentities($row['TITLE'], ENT_QUOTES)."' ,'".$thispath."/')";
34.   $addit = mysql_query($add, $link2) or die(mysql_error());
35.   @mysql_close($link2);
36.   read($row['ID'], $thispath, $thislevel+1);
37.  }
38.  @mysql_close($link);
39. }

note : I also use a field level to store the depth of a category-page (o for root, 1 for main categories and mine goes down to 4)

Then we make a simple routine to grab the first record for posted=0,
grab the url
grab the title
grab 3 posts off of google-blogsearch on the TITLE, add em to an email,
add a link to the category page url,
mail(email, subject, message-body, headers)

and ofcourse the coupe-de-grace, the cronjob, 700 posts, 4 per hour, so in about 170 hours my entire site is listed on a nice juicy blog. Just for the hell of it i put the links of the blogsearch on ‘follow’ so my poor victims get a link as well.

1.  
2. function connect() {
3.  $DB_USER =  "";
4.  $DB_PASSWORD = "";
5.  $DB_HOST = "";
6.  $DB_DATA = "";
7.  $link =  mysql_connect($DB_HOST, $DB_USER, $DB_PASSWORD) or $error = mysql_error();
8.  if (!$link) {
9.      echo $error;
10.   exit;
11.  } else {
12.     mysql_select_db($DB_DATA, $link) or $error = mysql_error();
13.  return $link;
14.  }
15. }
16.  
17.  $link = connect();
18.  $myqry = "SELECT * FROM `PLD_TAGCLOUD` WHERE `POSTED`='0' ORDER BY ID DESC";
19.  $myres = mysql_query($myqry, $link) or die(mysql_error());
20.  if(mysql_num_rows($myres)&lt;1) return;
21.  while($row=mysql_fetch_assoc($myres)) {  
22.   $myurl = $row['FULLPATH'];
23.   $mykey = urlencode($row['TAG']);
24.   $link2 = connect();
25.   $add="UPDATE `PLD_TAGCLOUD` SET `POSTED`='1' WHERE `ID`='".$row['ID']."'";
26.   $addit = mysql_query($add, $link2) or die(mysql_error());
27.   @mysql_close($link2);
28.   break;
29.  }
30.  @mysql_close($link);
31.  
32.  
33. $xmlSource="http://blogsearch.google.com/blogsearch_feeds?hl=en&c2coff=1&lr=&safe=active&as_drrb=q&as_qdr=d&q=".$mykey."&ie=utf-8&num=3&output=rss";
34. $title="";
35. $link="";
36. $description="";
37. $author="";
38. $pubDate="";
39. $currentElement="";
40. $nieuwsitems = array();
41.  
42. function startElement($parser,$name,$attr){
43.  if(strcmp($name,"item")==0){
44.  $GLOBALS['title']="";
45.  $GLOBALS['link']="";
46.  $GLOBALS['description']="";
47.  $GLOBALS['author']="";
48.  $GLOBALS['pubDate']="";
49.  }
50.  $GLOBALS['currentElement']=$name;
51.  if(strcmp($name,"link")==0){ $GLOBALS['href']=$attr["href"]; }
52.  
53. }
54.  
55. function endElement($parser,$name){
56.  $elements=array('title','link','description','author','pubDate');    
57.  if(strcmp($name,"item")==0){
58.   foreach($elements as $element){
59.    $temp[$element] = $GLOBALS[$element];      
60.   }
61.  $GLOBALS['nieuwsitems'][]=$temp;
62.  $GLOBALS['title']="";
63.  $GLOBALS['link']="";
64.  $GLOBALS['description']="";
65.  $GLOBALS['author']="";
66.  $GLOBALS['pubDate']="";
67.  }
68.  if(strcmp($name,"item")==0){
69.   $GLOBALS['title']="";
70.   $GLOBALS['link']="";
71.   $GLOBALS['description']="";
72.   $GLOBALS['author']="";
73.   $GLOBALS['pubDate']="";
74.  }
75. }
76.  
77. function characterData($parser, $data) {
78.  $elements = array ('title', 'link', 'description','author','pubDate');
79.  foreach ($elements as $element) {
80.   if ($GLOBALS["currentElement"] == $element) {
81.    $GLOBALS[$element] .= $data;
82.   }
83.  }
84. }
85.  
86. function parseFile(){
87.  global $xmlSource,$nieuwsitems;
88.  $xml_parser=xml_parser_create();
89.  xml_set_element_handler($xml_parser,"startElement","endElement");
90.  xml_set_character_data_handler($xml_parser,"characterData");
91.  xml_parser_set_option($xml_parser,XML_OPTION_CASE_FOLDING,false);
92.  if(!($fp=fopen($xmlSource,"r"))){
93.   die("Cannot open  $xmlSource  ");
94.  }
95.  while(($data=fread($fp,4096))){
96.   if(!xml_parse($xml_parser,$data,feof($fp))){
97.    die(sprintf("XML error at line %d column %d ",
98.    xml_get_current_line_number($xml_parser),
99.    xml_get_current_column_number($xml_parser)));
100.   }
101.  }
102.  xml_parser_free($xml_parser);
103.  return $nieuwsitems;
104. }
105.  
106. $result = parseFile();
107.  
108. foreach($result as $arr){
109.  $strResult .= '< hr />';
110.  $strResult .= '< h4>'.$arr["title"].'< /h4>'.$arr["description"].'< br /><a href="'.$arr["link"].'" title="'.$arr["title"].' ('.parse_url($arr["link"], PHP_URL_HOST).')">"'.$arr["title"].' ('.parse_url($arr["link"], PHP_URL_HOST).')</a>< br />< br />';
111. }
112.  
113. $strResult .= '< br /> < a href="'.$myurl.'" title="'.$mykey.'">trismegistos links : '.$mykey.'< /a>< br />';
114.  
115. $email='juustout.linkdirectory@blogger.com';
116. $subject = $mykey;
117. mail($email,$subject,$strResult, "MIME-Version: 1.0\n"."Content-type: text/html; charset=iso-8859-1");
118.  
119. echo $strResult;

(note the html markup in the last lines is < br />, if you cut and paste it, remove the space or you get a mess, also note the extra header in the php mail function, makes it possible to post html-marked up text (otherwise you get flat text posted and your site looks like ****).

]]> http://www.juust.org/index.php/blogger-auto-poster/2008/08/feed/ 2