juust ~ php oddities » seo tips and tricks http://www.juust.org Unordered list of one element Thu, 02 Sep 2010 16:58:24 +0000 http://wordpress.org/?v=2.8 en hourly 1 curl trackbacks http://www.juust.org/index.php/curl-trackbacks/2009/03/ http://www.juust.org/index.php/curl-trackbacks/2009/03/#comments Wed, 25 Mar 2009 09:53:13 +0000 juust http://www.juust.org/?p=375 I figure i’d blog a post on trackback linkbuilding. A trackback is … (post a few and you’ll get it). The trackback protocol isn’t that interesting, but the implementation of it by blog-platforms and cms’es makes it an excellent means for network development, because it uses a simple http-post. cUrl makes that easy).

To post a succesful link proposal I need some basic data :

about my page

  • url (must exist)
  • blog owner (free)
  • blog name (free)

about the other page

  • url (must exist)
  • excerpt (should be proper normal text)

my page : this is preferably a php routine that hacks some text, pictures and video’s, PLR or articles together, with a url rewrite. I prefer using xml textfiles in stead of a database, works faster when you set stuff up.

other page : don’t use “I liked your article so much…”, use text that maches text on target pages, preferably get some proper excerpts from xml-feeds like blogsearch, msn and yahoo (excerpts contain the keywords I searched for, as anchor text it works better for search engine visibility and link value).

Let’s get some stuff from the MSN rss feed :

  1. //a generic query = 5% success
  2. //add "(powered by) wordpress"
  3.       $query=urlencode('keywords+wordpress+trackback');
  4.       $xml = @simplexml_load_file("http://search.live.com/results.aspx?q=$query&count=50&first=1&format=rss");
  5.       $count=0;
  6.       foreach($xml->channel->item as $i) {
  7.  
  8.            $count++;
  9.  
  10. //the data from msn
  11.            $target['link'] = (string) $i->link;
  12.            $target['title'] = (string) $i->title;
  13.            $target['excerpt'] = (string) $i->description;
  14.  
  15. //some variables I'll need later on
  16.            $target[id'] = $count;
  17.           $target['trackback'] = '';
  18.           $target['trackback_success'] = 0;
  19.  
  20.           $trackbacks[]=$target;
  21.       }

25% of the cms sites in the top of the search engines are Wordpress scripts and Wordpress always uses /trackback/ in the rdf-url. I get the source of the urls in the search-feed and grab all link-url’s in it, if any contains /trackback/, I post a trackback to that url and see if it sticks.

(I can also spider all links and check if there is an rdf-segment in the target’s source (*1), but that takes a lot of time, I could also program a curl array and use multicurl, for my purposes this works fast enough).

  1. for($t=0;$t<count ($trackbacks);$t++) {
  2. //I could use curl
  3. //but 95% of the urls offered are kosher and respond fast
  4.      $content = @file_get_contents($trackbacks[$t]['link']);
  5.      preg_match_all ("/a[\s]+[^>]*?href[\s]?=[\s\"\']+".
  6.            "(.*?)[\"\']+.*?>"."([^< ]+|.*?)?<\/a>/",
  7.         $content, &$matches);
  8.  $uri_array = $matches[1];
  9.  foreach($uri_array as $key => $link) {
  10.              if(strpos($link, 'rackbac')>0) {
  11.                 $trackbacks[$t]['trackback'] = $link;
  12.                 break;
  13.              }
  14.         }
  15. }
  16. </count>

When I fire a trackback, the other script will try and assert if my page has a link and matching text. I have to make sure my page shows the excerpts and links, so I stuff all candidates in a cached xml file.

  1. function cache_xml_store($trackbacks, $pagetitle)
  2. {
  3.  $xml = '< ?xml version="1.0" encoding="UTF-8"?>
  4. <trackbacks>';
  5.  for($a=0;$a<count ($trackbacks);$a++) {
  6.   $arr = $trackbacks[$a];
  7.   $xml .= '<entry>';
  8.   $xml .= '<id>'.$arr['id'].'</id>';
  9.   $xml .= '<excerpt>'.$arr['excerpt'].'</excerpt>';
  10.   $xml .= '<link>'.$arr['link'].'</link>';
  11.   $xml .= '<title>'.$arr['title'].'</title>';
  12.   $xml .= '';
  13.  }
  14.  $xml .= '</count></trackbacks>';
  15.  
  16.  $fname = 'cache/trackback'.urlencode($pagetitle).'.xml';
  17.  if(file_exists($fname)) unlink('cache/'.$fname);
  18.  $fhandle = fopen($fname, 'w');
  19.  fwrite($fhandle, $xml);
  20.  fclose($fhandle);
  21.  return;
  22. }

I use simplexml to read that cached file and show the excertps and links once the page is requested.

  1. // retrieve the cached xml and return it as array.
  2. function cache_xml_retrieve($pagetitle)
  3. {
  4.  $fname = 'cache/trackback'.urlencode($pagetitle).'.xml';
  5.  if(file_exists($fname)) {
  6.   $xml=@simplexml_load_file($fname);
  7.   if(!$xml) return false;
  8.   foreach($xml->entry as $e) {
  9.    $trackback['id'] =(string) $e->id;
  10.    $trackback['link'] =  rid((string) $e->link);
  11.    $trackback['title'] =  (string) $e->title;
  12.    $trackback['description'] =  (string) $e->description;
  13.  
  14.    $trackbacks[] = $arr;
  15.   }
  16.   return $trackbacks;
  17.  }
  18.  return false;
  19. }

(this setup requires a subdirectory cache set to read/write with chmod 777)

I use http://www.domain.com/financial+trends.html and extract the pagetitle as “financial trends’, which has an xml-file http://www.domain.com/cache/financial+trends.xml. (In my own script I use sef urls with mod_rewrite, you can also use the $_SERVER array).

  1. $pagetitle=preg_replace('/\+/', ' ', htmlentities($_REQUEST['title'], ENT_QUOTES, "UTF-8"));
  2.  
  3. $cached_excerpts = cache_xml_retrieve($pagetitle);
  4.  
  5. //do some stuff with, make it look nice  :
  6. for($s=0;$s<count ($cached_excerpts);$s++) {
  7. //this lists the trackback (candidates)
  8.     echo $cached_excerpts[$s]['excerpt'];
  9.     echo '<a href="'.$cached_excerpts[$s]['link'].'">'.$cached_excerpts['title'].'';
  10. }
  11. </count>

Now I prepare the data for the trackback post :

  1. for($t=0;$t<count ($trackbacks);$t++) {
  2.  
  3.     $trackback_url = $trackbacks[$t]['trackback'];
  4. //does it have a trackback target url ? then prepare data :
  5.     if($trackback_url !='') {
  6.         $trackback_data = array(
  7.  "url" => "url of my page with the link to the target",
  8.   "title" => "title of my page",
  9.  "blog_name" => "name of my blog",
  10.  "excerpt" => '[...]'.trim(substr($trackbacks[$t]['description'], 0, 150).'[...]'
  11.         );
  12.         //…and try the trackback
  13.         $trackbacks[$t]['trackback_success'] = trackback_ping($trackback_url, $mytrackbackdata);
  14.     }
  15. }
  16. </count>

This the actual trackback post using cUrl. cUrl has a convenient timeout setting, I use three seconds. If a host does not respond in half a second it’s probably dead. Three seconds is generous.

  1. function trackback_ping($trackback_url, $trackback)
  2.  {
  3.  
  4. //make a string of the data array to post
  5.  foreach($trackback as $key=>$value) $strout[]=$key."=".rawurlencode($value);
  6.         $postfields= implode('&', $strout);
  7.  
  8. //create a curl instance
  9.  $ch = curl_init();
  10.  curl_setopt($ch, CURLOPT_URL, $trackback_url);
  11.  curl_setopt($ch, CURLOPT_TIMEOUT, 3);
  12.  curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
  13.  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  14.  
  15. //set a custom form header
  16.  curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-type: application/x-www-form-urlencoded'));
  17.  
  18.  curl_setopt($ch, CURLOPT_NOBODY, true);
  19.  
  20.         curl_setopt($ch, CURLOPT_POST, true);
  21.  curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);
  22.  
  23.  $content = curl_exec($ch);
  24.  
  25. //if the return has a tag 'error' with as value 0 it went flawless
  26.  $success = 0;
  27.  if(strpos($content, '>0')>0) $success = 1;
  28.  curl_close ($ch);
  29.  unset($ch);
  30.  return $success;
  31.  }

Now the last routine : rewrite the cached xml file with only the successful trackbacks (seo stuff) :

  1. for($t=0;$t<count ($trackbacks);$t++) {
  2.     if($trackbacks[$t]['trackback_success']>0) {
  3.         $store_trackbacks[]=$trackbacks[$t];
  4.     }
  5. }
  6. cache_xml_store($store_trackbacks, $pagetitle);
  7. </count>

voila : a page with only successful trackbacks.

Google (the backrub engine) don’t like sites that use automated link-building methods, other engines (Baidu, MSN, Yahoo) use a more normal link popularity keyword matching algorithm. Trackback linking helps getting you a clear engine profile at relative low cost.

0) for brevity and clarity, the code above is rewritten (taken from a trackback script I am developing on another site), it can contain some typo’s.

*1) If you want to spider links for rdf-segments : TYPO3v4 have some code for easy retrieval of trackback-uri’s :

  1. /**
  2.   * Fetches ping url from the given url
  3.   *
  4.   * @param string $url URL to probe for RDF
  5.   * @return string Ping URL
  6.   */
  7.  protected function getPingURL($url) {
  8.   $pingUrl = '';
  9.   // Get URL content
  10.   $urlContent = t3lib_div::getURL($url);
  11.   if ($urlContent && ($rdfPos = strpos($urlContent, '<rdf :RDF')) !== false) {
  12.    // RDF exists in this content. Get it and parse
  13.    $urlContent = substr($urlContent, $rdfPos);
  14.    if (($endPos = strpos($urlContent, '</rdf:RDF>', $rdfPos)) !== false) {
  15.     // We will use quick regular expression to find ping URL
  16.     $rdfContent = substr($urlContent, $rdfPos, $endPos);
  17.     $pingUrl = preg_replace('/trackback:ping="([^"]+)"/', '\1', $rdfContent);
  18.    }
  19.   }
  20.   return $pingUrl;
  21.  }
  22. </rdf>
]]> http://www.juust.org/index.php/curl-trackbacks/2009/03/feed/ 6 proxies ! http://www.juust.org/index.php/icanhazproxies/2009/02/ http://www.juust.org/index.php/icanhazproxies/2009/02/#comments Sat, 21 Feb 2009 03:41:16 +0000 juust http://www.juust.org/?p=336 I got a site banned at Google so I got pissed and took a script from the blackbox @ digerati marketing to scrape proxy addresses, wired a database and curl into it, so now it scrapes proxies, random picks a proxy, prunes dead proxies and returns data.

Basic, it uses anonymous (level 2) proxies, but it works. You can check the source here

  1.  
  2. /* (mysql table)
  3. CREATE TABLE IF NOT EXISTS `serp_proxies` (
  4.   `id` int(11) NOT NULL auto_increment,
  5.   `ip` text NOT NULL,
  6.   `port` text NOT NULL,
  7.   PRIMARY KEY  (`id`)
  8. ) ENGINE=MyISAM  DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;
  9. */
  10.  
  11. //initialize database class, replace with own code
  12. include('init.php');
  13.  
  14. //main class
  15. $p=new MyProxies;
  16.  
  17. //do I have proxies in the database ?
  18. //if not, get some and store them
  19. if($p->GetCount() < 1) {
  20.  $p->GetSomeAir(1);
  21.  $p->store2database();
  22. }
  23.  
  24. //pick one
  25. $p->RandomProxy();
  26.  
  27. //get the page
  28. $p->ThisProxy->DoRequest('http://www.domain.com/robots.txt');
  29.  
  30. //error handling
  31. if($p->ThisProxy->ProxyError > 0) {
  32. //7   no connect
  33. //28   timed out
  34. //52   empty reply
  35. //if it is dead, doesn't allow connections : prune it
  36.  if($p->ThisProxy->ProxyError==7) $p->DeleteProxy($p->ThisProxy->proxy_ip);
  37.  if($p->ThisProxy->ProxyError==52) $p->DeleteProxy($p->ThisProxy->proxy_ip);
  38. }
  39. //you could loop back until you get a 0-error proxy, but that ain't the point
  40.  
  41. //give me the content
  42. echo $p->ThisProxy->Content;
  43.  
  44.  
  45. Class MyProxies {
  46.  
  47.  var $Proxies = array();
  48.  var $ThisProxy;
  49.  var $MyCount;
  50.  
  51.  
  52. //picks a random proxy from the database
  53.  function RandomProxy() {
  54.  
  55.   global $serpdb;
  56.   $offset_result =  $serpdb->query("SELECT FLOOR(RAND() * COUNT(*)) AS `offset` FROM `serp_proxies`");
  57.   $offset_row = mysql_fetch_object($offset_result);
  58.   $offset = $offset_row->offset;
  59.   $result = $serpdb->query("SELECT * FROM `serp_proxies` LIMIT $offset, 1" );
  60.   while($row=mysql_fetch_assoc($result)) {
  61. //make instance of Proxy, with proxy_host ip and port
  62.    $this->ThisProxy = new Proxy($row['ip'].':'.$row['port']);
  63.    $this->ThisProxy->proxy_ip = $row['ip'];
  64.    $this->ThisProxy->proxy_port = $row['port'];
  65.    break;
  66.   }
  67.  }
  68.  
  69. //visit the famous russian site
  70.  function GetSomeAir($pages) {
  71.    for($index=0; $index< $pages; $index++)
  72.    {
  73.     $pageno = sprintf("%02d",$index+1);
  74.     $page_url = "http://www.samair.ru/proxy/proxy-" . $pageno . ".htm";
  75.     $page_html = @file_get_contents($page_url);
  76.  
  77. //get rid of the crap and extract the proxies
  78.     preg_match("/<tr><td>(.*)< \/td>< \/tr>/", $page_html, $matches);
  79.     $txt = $matches[1];
  80.     $main = split('</td><tr><td>', $txt);
  81.     for($x=0;$x<count ($main);$x++) {
  82.      $arr = split('</td><td>', $main[$x]);
  83.      $this->Proxies[] = split(':', $arr[0]);
  84.     }
  85.    }
  86.  }
  87.  
  88. //store the retrieved proxies (stored in this->Proxies) in the database
  89.  function store2database() {
  90.   global $serpdb;
  91.   foreach($this->Proxies as $p) {
  92.    $result = $serpdb->query("SELECT * FROM serp_proxies WHERE ip='".$p[0]."'");
  93.    if(mysql_num_rows($result)&lt;1) $serpdb->query("INSERT INTO serp_proxies (`ip`, `port`) VALUES ('".$p[0]."', '".$p[1]."')");
  94.   }
  95.   $serpdb->query("DELETE FROM serp_proxies WHERE `ip`=''");
  96.  }
  97.  
  98.  
  99.  function DeleteProxy($ip) {
  100.   global $serpdb;
  101.   $serpdb->query("DELETE FROM serp_proxies WHERE `ip`='".$ip."'");  
  102.  }
  103.  
  104.  
  105.  function GetCount()
  106.  {
  107. //use this to check how many proxies there are in the database
  108.   global $serpdb;
  109.   $this->MyCount = mysql_num_rows($serpdb->query("SELECT * FROM `serp_proxies`"));
  110.   return $this->MyCount;
  111.  }
  112.  
  113.  
  114. }
  115.  
  116. Class Proxy {
  117.  
  118.  var $proxy_ip;
  119.  var $proxy_port;
  120.  
  121.  var $proxy_host;
  122.  var $proxy_auth;
  123.  var $ch;
  124.  var $Content;
  125.  var $USERAGENT = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
  126.  var $ProxyError = 0;
  127.  var $ProxyErrorMsg = '';
  128.  var $TimeOut=3;
  129.  var $IncludeHeaders = 0;
  130.  
  131.  function Proxy($host, $username='', $pwd='') {
  132. //initialize class, set host
  133.          $this->proxy_host = $host;
  134.          if (strlen($username) > 0 || strlen($pwd) > 0) {
  135.             $this->proxy_auth = $username.":".$pwd;
  136.          }
  137.       }
  138.  
  139.  function CURL_PROXY($cc) {
  140.    if (strlen($this->proxy_host) > 0) {
  141.     curl_setopt($cc, CURLOPT_PROXY, $this->proxy_host);
  142.     if (strlen($this->proxy_auth) > 0)
  143.      curl_setopt($cc, CURLOPT_PROXYUSERPWD, $this->proxy_auth);
  144.    }
  145.  }
  146.  
  147.  function DoRequest($url) {
  148.   $this->ch = curl_init();
  149.   curl_setopt($this->ch, CURLOPT_URL,$url);
  150.   $this->CURL_PROXY($this->ch);
  151.   curl_setopt($this->ch, CURLOPT_HEADER, $this->IncludeHeaders); // baca header
  152.  
  153.   curl_setopt($this->ch, CURLOPT_USERAGENT, $this->USERAGENT);
  154.   curl_setopt($this->ch, CURLOPT_RETURNTRANSFER, 1);
  155.   curl_setopt($this->ch, CURLOPT_TIMEOUT, $this->TimeOut);
  156.      $this->Content = curl_exec($this->ch);
  157.  
  158. //if an error occurs, store the number and message
  159.   if (curl_errno($this->ch))
  160.    {
  161.     $this->ProxyError =  curl_errno($this->ch);
  162.     $this->ProxyErrorMsg =  curl_error($this->ch);
  163.    }
  164.  }
  165.  
  166. }
  167. </td></count></td></tr>

There is not much to say about it, just a rough outline. I would prefer elite level 1 proxies but for now it will have to do.

]]> http://www.juust.org/index.php/icanhazproxies/2009/02/feed/ 0 synonymizer with api http://www.juust.org/index.php/synonymizer-with-api/2008/12/ http://www.juust.org/index.php/synonymizer-with-api/2008/12/#comments Sun, 28 Dec 2008 12:09:46 +0000 juust http://www.juust.org/?p=172 If you want to put some old content on the net and have it indexed as fresh unique content, this works wonders for seo-friendly backlinks : the automated synonymizer. I want one that makes my content unique without having to type one character.

Lucky for me, mister John Watson’s synonym database comes with a free 10.000 request a day API and boy is it sweet!

API Requests are straightforward :
http://words.bighugelabs.com/api/2/[apikey]/[keyword]/xml

A number of return formats are supported but xml is easiest, either for parsing with simplexml or regular pattern matching.

It returns on request :
black (slightly shortened)
an xml file like :
<words>
<w p=”adjective” r=”syn”>bleak</w>
<w p=”adjective” r=”syn”>sinister</w>
<w p=”adjective” r=”sim”>dark</w>
<w p=”adjective” r=”sim”>angry</w>
<w p=”noun” r=”syn”>blackness</w>
<w p=”noun” r=”syn”>inkiness</w>
<w p=”verb” r=”syn”>blacken</w>
<w p=”verb” r=”syn”>melanize</w>
</words>

…which is easiest handled with preg_match_all :

  1. function getsynonyms($keyword) {
  2.         $pick = array();
  3.  $apikey = 'get your own key';
  4.  $xml=file_get_contents('http://words.bighugelabs.com/api/2/'.$apikey.'/'.$keyword.'/xml');
  5.  
  6.  if(!$xml) return $pick; //return empty array
  7.  
  8.  preg_match_all('/<w p="adjective" r="syn">(.*?)< \/w>/', $xml, $adj_syns);
  9.  //preg_match_all('/</w><w p="adjective" r="sim">(.*?)< \/w>/', $xml, $adj_sims);
  10.  //preg_match_all('/</w><w p="noun" r="syn">(.*?)< \/w>/', $xml, $noun_syns);
  11.  //preg_match_all('/</w><w p="verb" r="syn">(.*?)< \/w>/', $xml, $verb_syns);
  12.  
  13.  foreach($adj_syns[0] as $adj_syn) $pick[]=$adj_syn;
  14.         //same for verb/noun synonyms, I just want adjectives
  15.  
  16.  return $pick;
  17. }
  18. </w>

practically applying it,
I take a slab of stale old content and…

  • strip tags
  • do a regular match on all alphanumeric sequences dropping other stuff
  • trim the resulting array elements
  • (merge all blog tags, categories, and a list of common words)
  • excluding common terms from the array with text elements
  • excluding words smaller than N characters
  • set a percentage words to be synonimized
  • attempt to retrieve synonyms for remaining terms
  • replace these words in the original text, keep count
  • when I reach the target replacement percentage, abort
  • return (hopefully) a revived text
  1. function synonymize($origtext) {
  2.  
  3. //make a copy of the original text to dissect
  4.  $content=$origtext;
  5.  //content = $this->body;
  6.  
  7.  $perc=3;   //target percentage changed terms
  8.  $minlength=4;  //minimum length candidates
  9.  $maxrequests=80; //max use of api-requests
  10.  
  11.  
  12.  //dump tags
  13.  $content =  strip_tags($content);
  14.  
  15.  //dump non-alphanumeric string characters
  16.  $content = preg_replace('/[^A-Za-z0-9\-]/', ' ', $content);
  17.  
  18.  //explode on blank space
  19.  $wrds = explode(' ', strtolower($content));
  20.  
  21.  //trim off blank spaces just in case
  22.  for($w=0;$w<count ($wrds);$w++) $words[] = trim($wrds[$w]);
  23.  
  24.  //this should be all words
  25.  $wordcount = count($words);
  26.  
  27.  //how many words do I want changed ?
  28.  $toswitch = round($wordcount*$perc/100);
  29.  
  30.  //only use uniques
  31.  $words_unique=array_unique($words);
  32.  
  33.  //sort, start with words at the end of the text
  34.  sort($words_unique);
  35.  
  36.  //merge common with tags, categories, linked_tags
  37.  $common = array("never", "about", "price");
  38. //note : setting the minlength to 4 excludes lots of common terms
  39.    
  40.  for($i=0;$i<count($words_unique);$i++) {
  41.  //if in common array, not selectable for synonymizing
  42.   if(in_array($words_unique[$i], $common)) {} else {
  43.    //only terms bigger than minlength
  44.    if(strlen($words_unique[$i])>$minlength) {
  45.    //words_select contains candidates for synonyms
  46.     $words_select[] = trim($words_unique[$i]);
  47.    }
  48.   }
  49.  }
  50.  
  51.  //terms that can be changed
  52.  $max = count($words_select);
  53.  
  54.  //no more requests than max
  55.  if($max>$maxrequests) $max=$maxrequests;
  56.  
  57.  for($i=0;$i< $max;$i++) {
  58.  //get synonyms, give server some time
  59.   usleep(100000);
  60.   //retrieve synonyms etc.
  61.   $these_words = getsynonyms($words_select[$i]);
  62.   $jmax=count($these_words);
  63.   if($jmax&lt;1) {
  64.   //no results
  65.   } else {
  66. $count=0;
  67.    $j=0;
  68. //the replacements are done in the original text
  69.    $origtext= preg_replace('/'.$words_select[$i].'/i', $these_words[$j], $origtext, -1, $count);
  70.    $total_switched+=$count;
  71.  
  72.   } //have we reached the percentage ?
  73.   if($total_switched>=$toswitch) break;
  74.  }
  75.  //okay!
  76.  return $origtext;
  77. }
  78.  
  79. function getsynonyms($keyword) {
  80.  $pick=array ();
  81.  $apikey = 'get your own key at bighugelabs.com';
  82.  $xml=@file_get_contents('http://words.bighugelabs.com/api/2/'.$apikey.'/'.urlencode($keyword).'/xml');
  83.  if(!$xml) return $pick;
  84.  preg_match_all('/<w p="adjective" r="syn">(.*?)< \/w>/', $xml, $adj_syns);
  85.  foreach($adj_syns[0] as $adj_syn) $pick[]=$adj_syn;
  86.  return $pick;
  87. }
  88. </w></count>

Nothing fancy, a straightforward search-replace routine. A 1200 word text has about 150 candidates and for 3% synonyms I need to replace 36 words, it can do that. If I were to use it for real I would build a table with non-returning terms, and store often used terms, that would speed up the synonimizing, allow the use of preferences and take a load of the api use.

]]> http://www.juust.org/index.php/synonymizer-with-api/2008/12/feed/ 1 RedHat Seo : scraper auto-blogging http://www.juust.org/index.php/redhat-seo-christmas-edition/2008/12/ http://www.juust.org/index.php/redhat-seo-christmas-edition/2008/12/#comments Fri, 26 Dec 2008 18:07:01 +0000 juust http://www.juust.org/?p=270

Just give us your endpoint and we’ll take it from there, sparky!

I was going to make one of these tools to scrape google and conjur a full blog out of nowhere, as Christmas special, RedHat Seo. The rough sketch has arrived , far from perfect, but it does produce a blog and don’t even look too shabby. I scraped a small batch of posts off of blogs, keeping the links intact and adding a tribute links. I hope they will pardon me for it.

structure

I use three main classes,

BlogMaker the application
Target the blogs you aim for
WPContent the scraped goodies

…and two support classes

SerpResult scraped urls
Custom_RPC a simple rpc-poster

Target blogs have three texts,

file contents maintenance
blog categories category you post under manual
blog tags tags you list on the blog manual
blog urls urls already used for the blog system

routine

The BlogMaker class grabs a result list (up to 1000 urls per phrase) from Google, extracts the urls and stores them in SerpResult, scrapes the urls and extracts the entry divs, stores div-entries in the WPContent class (that has some basic functions to sanitize the text), and uses the BlogTarget-definitions to post it up blogs with xml-rpc.

usage

My highlighter tends to mess up text with div markers in it, copying off the blog may not work,
the full text source (about 500 lines) is overhere. Underneath I’ll list the main program loop :

  1.  
  2. //make main instance
  3. $Blog = new BlogMaker("keyword");
  4.  
  5. //define a target blog, you can define multiple blogs and refer with code
  6. //then add rpc-url, password and user
  7. //and for every target blog three text-files
  8.  
  9. $T=$Blog->AddTarget(
  10.  'blogcode',
  11.  'http://my.blog.com/xmlrpc.php',
  12.  'password',
  13.  'user',
  14.  'keyword.categories.txt',
  15.  'keyword.tags.txt',
  16.  'keyword.urls.txt'
  17.  );
  18.  
  19. //read the tags, cats and url text files stored on the server
  20. //all retrieved urls are tested, if the target blog already has that
  21. //scraped url, it is discarded.
  22. $T->CSV_GetTags();
  23. $T->List_GetCats();
  24. $T->ReadURL();
  25.  
  26. //grab the google result list
  27. //use params (pages, keywords) to specify search
  28. $Blog->GoogleResults();
  29.  
  30. $a=0;
  31. foreach($Blog->Results as $BlogUrl) {
  32.   $a++;
  33.   echo $BlogUrl->url;
  34. //see if the url isnt used yet
  35.  
  36.  if($T->checkURL(trim($BlogUrl->url))!=true) {
  37.    echo '…checking ';
  38.    flush();
  39. //if not used, get the source
  40.    $BlogUrl->scrape();
  41. //check for divs marked "entry", if they arent there, check "post"
  42. //some blogs use other indications for the content
  43. //but entry and post cover 40%
  44.  
  45.    $entries = $BlogUrl->get_entries();
  46.    if(count($entries)&lt;1) {
  47.     echo 'no entries…';
  48.     flush();
  49.     $entries = $BlogUrl->get_posts();
  50.      if(count($entries)&lt;1) {
  51.       echo 'no posts either…';
  52. //if no entry-post div, mark url as done
  53.  
  54.       $T->RegisterURL($BlogUrl->url);
  55.      }
  56.    }
  57.  
  58.    $ct=0;
  59.    foreach($BlogUrl->WpContentPieces as $WpContent) {
  60. //in the get_entries/get_post function the fragments are stored
  61. //as wpcontent
  62.     $ct++;
  63.  
  64.     if($WpContent->judge(2000, 200, 5)) {
  65.      $WpContent->tribute();  //add tribute link
  66.      $T->settags($WpContent->divcontent); //add tags
  67.      $T->postCustomRPC($WpContent->title, $WpContent->divcontent, 1); //1=publish, 0=draft
  68.      $T->RegisterURL($WpContent->url);  //register use of url
  69. usleep(20000000);  //20 seconds break, for sitemapping
  70.     }
  71.    }
  72.   }
  73.  }

notes

  • xml-rpc needs to be activated explicitly on the wordpress dashboard under settings/writing.
  • categories must be present in the blog
  • url file must be writeable by the server (777)

It seems wordpress builds the sitemap as background process, the standard google xml sitemap plugin wil attempt to build in the cache (takes anywhere between 2 and 10 seconds), and apart from building a sitemap the posts also get pinged around. Giving the install 10 to 20 seconds between posts allows for all the hooked in functions to be completed.

period

That’s about all,
consider it gpl, I added some comments in the source but I will not develop this any further. A mysql backed blogfarm tool (euphemistically called ‘publishing tool’) is more interesting, besides, I am off to the wharves to do some painting.

if you use it, send some feedback,
merry christmas dogheads

]]> http://www.juust.org/index.php/redhat-seo-christmas-edition/2008/12/feed/ 1 How to grab keywords from 7search http://www.juust.org/index.php/how-to-grab-keywords-from-7search/2008/10/ http://www.juust.org/index.php/how-to-grab-keywords-from-7search/2008/10/#comments Fri, 24 Oct 2008 09:00:57 +0000 juust http://www.juust.org/?p=198 “Seo tips and tricks” was not due til November, but this one just popped up today. I was looking for a tool to build a rapid keyword set for a blog, without doing extensive keyword research. The blackhat ’scraper’ scripts I found come up with ‘michigan seo’ far too many times :) so I built me a quick alternative.

How to grab keyword sets from 7Search

I want a set of keywords as blog categories to write a blog that contains material with the most popular keywords covering the whole active search pattern set. A nice tool for that is 7Search’s keyword tool.

It has a captcha protection, you have to answer it once and then you can query as much as you like, it shows last months top 100 search patterns with that keyword and the search volumes :

seo
 1,991,112 $0.34 $0.33 $0.21 $0.09 $0.08
seo web design 8,085 $0.07 $0.02 $0.01
seo tool 2,647 $0.05 $0.02 $0.02 $0.01

As I am extremely lazy and hate typing data, I’ll make a quick script to cut and paste that list and have it magically transformed in a wordpress blog category list.

It turned out to be a simple one page program : source text file. Cut and paste stuff, I do a query on a keyword, select the result table area of the 7Search page (with the mouse : ) and paste it as text into my own form’s textarea, add the main key, and post it.

From the $_POST array, I take the textarea input and explode it on linebreaks. To get the keywords, I check for the first occurrence of 0-9, take the part that comes before it, and have the keywords.

In this function I test for the first 0-9. Had I stopped at the first number and started at 0, I would get thrown out of the loop if there is any 0 in the line (or 1, 2, 3…), regardless of there being any number before the first 0 :

  1. $pos = strpos($linesarr[$x], $i);
  2. if($pos >0) {
  3.     if($pos< $minpos) {
  4.         $mykeys = substr($linesarr[$x],0,$minpos);
  5.         echo $mykeys;
  6.         break;
  7.  }}}

So I test for the first 0 and store the position in minpos, then test for the first 1, 2, 3…, if it comes before the first 0, minpos is set to the lowest position.

  1. $lines=$_POST['textarea'];
  2. $linesarr = explode("\r\n", $lines);
  3.  
  4. for ($x=0;$x<count ($linesarr);$x++) {
  5. //set minpos to the length of the line
  6.     $minpos=strlen($linesarr[$x]);
  7.  
  8. //check numbers 0-9
  9.     for($i=0;$i&lt;10;$i++) {
  10. //get position of first number $i in the line
  11.         $pos = strpos($linesarr[$x], $i);
  12.         if($pos >0) {
  13.             if($pos< $minpos) $minpos=$pos;
  14.         }
  15.     }  
  16.  
  17. //is minpos smaller than the length of the line ? then its valid data
  18.         if($minpos<strlen($linesarr[$x])) {
  19.             $mykeys = substr($linesarr[$x],0,$minpos);
  20.             echo $mykeys;
  21.         }
  22. }

That way I always get the first number in the line and the part before it is the whole keyword text.

I also want the search volumes, which is the first full string after the keywords up till the first $-dollar sign. The minpos counter is already at the start digit of the volume. I can get the position of the first dollar sign, and trim off the blanks.

  1. //volume is the is at the start of the string after minpos
  2.  $volstr = trim(substr($linesarr[$x], $minpos));
  3. //and before the first dollar sign
  4.  $volcut = strpos($volstr, "$");
  5. //it contains "," : 9,111,222 so filter out the nonsense for mysql :
  6.  $vol = preg_replace('/,/', '', trim(substr($volstr, 0, $volcut)));
  7.  
  8.  if($minpos<strlen ($linesarr[$x])) {
  9.     $mykeys = substr($linesarr[$x],0,$minpos);
  10.     echo $mykeys."_".$vol;
  11.         }

[After this I stuff the data in a mysql table `sevencats`].

How to add a keyword list to wordpress as categories

Let's add the keywords to a wordpress blog as categories. Wordpress has a very simple function for it wp_insert_term in the taxonomy.php file.

In wpmu you do first have to pick the target blog, as you work on a blogs tableset, wp1_, wp2_ etcetera and if you start it up you get the admin users main blog as active tableset. If you want to add data like categories in another blogs taxonomy table you have to switch to that table set first.

  1. function connect_data() {
  2.   $DB_USER =  "";
  3.   $DB_PASSWORD = "";
  4.   $DB_HOST = "";
  5.   $DB_DATA = "";
  6.   $link =  mysql_connect($DB_HOST, $DB_USER, $DB_PASSWORD) or $error = mysql_error();
  7.   if (!$link) {
  8.       return $error;
  9.   }  
  10.         mysql_select_db($DB_DATA, $link) or $error = mysql_error();
  11.   return $link;
  12.  }
  13.  
  14. //link
  15. $cats=connect_data();
  16.  
  17. //get array with categories
  18. $categories=array();
  19.  
  20. $qry="SELECT cat FROM `sevencats`";
  21. $lst=mysql_query($qry, $cats) or die('list error '.mysql_error());
  22. while($row=mysql_fetch_assoc($lst)) {
  23.  $categories[]=$row['cat'];
  24. }
  25. //close db connection
  26. mysql_close($cats);
  27.  
  28. //open wordpress connection
  29. include_once('wp-config.php');
  30. include_once('wp-includes/wp-db.php');
  31. include_once('wp-includes/taxonomy.php');
  32.  
  33. //select target blog by id
  34. switch_to_blog(3);
  35.  
  36. //insert categories
  37. for ($i=0;$i<count ($categories);$i++) {
  38.       wp_insert_term($categories[$i], 'category');
  39. }
  40.  
  41. //switch back to users main blog
  42. restore_current_blog();

For a normal wordpress install you'd not have to switch blogs :

  1. //open wordpress connection
  2. include_once('wp-config.php');
  3. include_once('wp-includes/wp-db.php');
  4. include_once('wp-includes/taxonomy.php');
  5.  
  6. //insert categories
  7. for ($i=0;$i<count ($categories);$i++) {
  8.       wp_insert_term($categories[$i], 'category');
  9. }

That gets me the top 100 searches of last month as categories for my new blog all. You can fiddle with it a bit and only pick searches with a volume above 2000 monthly searches (just in case you want to go scraping and only want material that gets you in the serp pages for the high volume search terms).

Next edition : Red Hat Seo (with jingle bells) the Christmas Special :)

]]> http://www.juust.org/index.php/how-to-grab-keywords-from-7search/2008/10/feed/ 1 seo tricks : the magpie incident http://www.juust.org/index.php/seo-tricks-the-magpie-incident/2008/10/ http://www.juust.org/index.php/seo-tricks-the-magpie-incident/2008/10/#comments Wed, 01 Oct 2008 22:05:07 +0000 juust http://www.juust.org/?p=166 Some universities like Southern California, Harvard and Michigan State have their web-guru’s explain to us how rss feeds work with the elegant Magpie parser demo :

Some example on how to use Magpie:

* magpie_simple.php *
Simple example of fetching and parsing an RSS file. Expects to be
called with a query param ‘rss_url=http://(some rss file)’
….

* magpie_debug.php *
Displays all the information available from a parsed feed.

Note : magpie_debug.php is the one to watch for, you can do a google search on :

site:.edu magpie_debug.php

and you get a number of educational facilities that kindly demonstrate the use of the magpie rss parser.

These demo pages have a textbox where you can enter an rss feed url, the magpie demo parses your feed and outputs it as an html-page.

You have to be careful with these programs, though : I actually found one domain (www.scripps.edu) with this remark under the ‘parse rss’ button :

Security Note:
This is a simple example script. If this was a real script we probably wouldn’t allow strangers to submit random URLs, and we certainly wouldn’t simply echo anything passed in the URL. Additionally its a bad idea to leave this example script lying around.

Thank you, you are surely wise like the buddha, I shall try to remember your insight !

….
note: after a while I decided I had had enough fun with magpies and took the blog off-line.

]]> http://www.juust.org/index.php/seo-tricks-the-magpie-incident/2008/10/feed/ 1 seo tricks : old wine in new bags… http://www.juust.org/index.php/seo-tricks-old-wine-in-new-bags/2008/09/ http://www.juust.org/index.php/seo-tricks-old-wine-in-new-bags/2008/09/#comments Fri, 26 Sep 2008 05:19:05 +0000 juust http://www.juust.org/?p=160 Get some pagerank : this trick would require tedious boring link checking, but since SeoLinx (an extension of SeoQuake) that has become a lot easier. SeoLinx shows the stats of a links target url so you don’t have to go to every page to retrieve the stats. Cool plugin. Let’s put it to some practical use.

the trick : comment on old forum threads

Once you have SeoLinx installed find an ‘old’ forum, register if you haven’t already and make sure you get a signature link. Sometimes you first have to be a member for a week or write ten posts, but once you have a sig-link you get backlinks off the forum.

Then go comment on really old forum threads.

With SeoLinx you can easily spot the juicy old threads. Old threads on for instance DigitalPoint or Webmasterworld are sometimes pagerank 3. In case of the DP post, PR2 with 8 posts at the time of writing.

Pick a forum, and browse to the last page of the threads. Hover over the thread anchor and SeoLinx shows you the pagerank of the thread page. As long as the number of posts is below (10, 16 depending on the forum settings) you can put your comments in and they will appear on the first page of that thread, that has that nice pagerank and juice.

Old wine in new bags can be a sweet thing.

the benefit

A pagerank 3 ‘targetted’ anchor is worth about $9,- a month, $100,- per year. It can take an hour to find a juicy one, but hey, $100,- value for an hours work is well worth the trouble.

I might make this a blog feature, seo tips and tricks of the month.

]]> http://www.juust.org/index.php/seo-tricks-old-wine-in-new-bags/2008/09/feed/ 2