spot the bot

juust | 10/09/2009

I have some overhead scripts fetching data that can cost a few seconds extra loading time. Having traffic trigger tasks saves me the trouble of using cron-jobs, but I don’t want to run overhead scripts with visitors or googlebot on the site. Apart from that, some routines can use a lot of resources which are wasted on some crawlers.

I actually want the crawlers to come around, so I will make an array with bots and allowed_bots. Whatever is not on the white-list gets a meager page with overhead jobs attached to it, the rest (iow visitors and the big search engines) get the standard page.

There are truckloads of bots (see crawltrack), for my purposes a few regulars will do.

  1.  
  2. //hook it into 'init', run when calling script
  3. add_action( 'init', 'spotabot' );
  4.  
  5. /**
  6.  * checks if visitor is a bot
  7.  *
  8.  * This method checks the http_user_agent string
  9.  * to see if the visitors is a non-essential bot
  10.  *
  11.  * @param void
  12.  * @return void
  13.  */
  14.  
  15. /*
  16.    if(IS_A_BAD_BOT) {}
  17. */
  18. function spotabot()
  19. {
  20.     $bot_list = array("Teoma", "betaBot", "alexa", "froogle", "Gigabot", "inktomi",
  21.     "looksmart", "URL_Spider_SQL", "Firefly", "NationalDirectory",
  22.     "Ask Jeeves", "TECNOSEEK", "InfoSeek", "WebFindBot", "girafabot",
  23.     "crawler", "www.galaxy.com", "Googlebot", "Scooter", "Slurp",
  24.     "msnbot", "appie", "FAST", "WebBug", "Radian6", "Spade", "ZyBorg", "rabaz",
  25.     "Baiduspider", "Feedfetcher-Google", "TechnoratiSnoop", "Rankivabot",
  26.     "Mediapartners-Google", "Sogou web spider", "WebAlta Crawler");
  27.  
  28.     $bot_allowed = array("Googlebot", "Feedfetcher-Google", "Mediapartners-Google", "Slurp", "Baiduspider", "msnbot");
  29.  
  30.     foreach($bot_list as $bot) {
  31.         if(strpos(strtolower("x".$_SERVER['HTTP_USER_AGENT']), strtolower($bot))>0)
  32.         {
  33.             foreach($bot_allowed as $okbot) {
  34.                  if($okbot==$bot) {
  35.                     define("IS_A_BAD_BOT", false);
  36.                     return;
  37.                  }
  38.            
  39.             define("IS_A_BAD_BOT", true);
  40.             return;
  41.             }
  42.         }
  43.     }
  44.    
  45.     define("IS_A_BAD_BOT", false);
  46.     return;
  47. }

In templates and functions i can use some simple code to run stuff conditional :

  1. if (defined('IS_A_BAD_BOT')) {
  2.    if(IS_A_BAD_BOT)
  3.    {
  4.     echo "hi bot<br />";
  5.                                 run_time_consuming_overhead_tasks();
  6.                                 and_omit_the_sidebar();  
  7.    } else {
  8.     echo "hello wonderful visitor<br />";
  9.    }
  10.   }
  11. //if it is not defined it is not a bot or the function ain't present,
  12. //I am lazy and sloppy and don't want a code-break

It would be nice if Wordpress built in a switch to run plugins conditional.

one related smart plugin is the chennai central plugin that sends 304 not modified headers on conditional GETs, so crawlers don’t fetch the page. That can save some bandwidth and serverload.

Comments
3 Comments »
Categories
optimisation, wordpress
Tags
optimisation, wordpress
Comments rss Comments rss
Trackback Trackback

quick note : serp plugin beta

juust | 02/09/2009

I am developing a new google serp plugin for wordpress. I developed it over the weeknd on wp2.8, it should run on anything with a ’shutdown’ hook and jquery. I will be adding some other stuff to it over the next few weeks, but I thought I’d put a beta on the blog. If you try it, I hope it runs, send some feedback.

If you have some ideas about cool features drop me a note, I might put it in there. I was pondering on a rest service to pool sem-data and develop some competitor analysis functionality, but that is just a vague idea.

Comments
3 Comments »
Categories
juust, serp, wordpress
Tags
plugin, serp, wordpress
Comments rss Comments rss
Trackback Trackback

wordpress : fun with pluggable classes

juust | 31/07/2009

I was checking some idea i had about writing a small user class with an option to ‘plug in’ functions for wordpress.

This page covers most of it :
dynamically add functions to php classes @ www.gen-x-design.com. The class construct at the end of the comment thread, Martin Pietschmann’s contribution, is rather useful. This pattern revolves around importing functionality from pluggable classes and exposing it through one object instance (the ‘decorator’ pattern mentioned is mostly used for writing extended classes, different objects with the same base data and functionality, ‘views’ sort of).

I can include the file with base, import and user class into function.php, and on making a user object have it read the directory and import functions modules (or load functionality conditional based on user role/authorization).

For this example I used the wordpress options table. I write a new functions class

  1. class UserBogusPlugin extends MI_Importable
  2. {
  3.  public function the_anchor() {
  4. //user_url and user_nicename are exposed through the user class
  5. //I plug the functions class into, i can use the $this reference
  6. //as if i am writing code in the user class
  7.                 if($this->user_url<>'') {
  8.                     return '<a href="'.$this->user_url.'">'.$this->user_nicename .'</a>, ' . $this->first_name;
  9.                 }
  10. //no url, no anchor….  
  11.   return $this->user_nicename;
  12.  }
  13. }

…store the added class name in the options table…

  1. //load $arr from options table
  2. $modules = get_option('usermodules');
  3. if($modules) $usermodules = json_decode($modules);
  4. //add module
  5. $usermodules[] = 'UserBogusPlugin';
  6. //store back in options
  7. add_option('usermodules', json_encode($usermodules));

…and load the plugin classes when instantiating the user object :

  1. class User extends MI_Base
  2. {
  3.  public function __construct($ID) {
  4.             $modules = get_option('usermodules');
  5.             if($modules) {
  6.                 $usermodules = json_decode($modules);
  7.                 foreach($usermodules as $module) {
  8.                   if(class_exists($module)) $this->import(new $module);
  9.                  //or..
  10.                  //include('plugclass/'.$module.'.class.php');
  11.                  //$this->import(new $module);
  12.                 }
  13.             }
  14.             $this->ID=$ID;
  15.  }
  16. }

In the wordpress template i can use the added functionality through the user instance :

  1.       $my_user = new User($user_id);
  2.       …
  3.       echo $my_user->the_anchor();

Fun with classes :)

add. 3-8 (qed) :
a function to load a directory with plugin files, i add a header /* plugin pluginfilename */ and check the files if there is a header.

  1.        public function getPlugins()
  2.         {
  3.             $plugdir = TEMPLATEPATH .'/plug';
  4.             if ($handle = opendir($plugdir)) {
  5.                 $retval = array();
  6.                 while (false !== ($file = readdir($handle))) {
  7.                     if (($file <> ".") && ($file <> "..")) {
  8.                         $fh = fopen($plugdir.'/'.$file, 'r');
  9.                         $contents = '';
  10.                           $contents .= fread($fh, 1024);
  11.                         fclose($fh);
  12.                         if(preg_match('/lugin/', $contents))
  13.                         {   //check for header : plugin, grab pluginname
  14.                             $a = strpos($contents, 'lugin');
  15.                             $a += 6;
  16.                             $b = strpos($contents, ' ', $a);
  17.                             $plugname = substr($contents, $a, $b-$a);
  18.                             $retval[] = array($plugdir.'/'.$file, $plugname);
  19.                 }  }  }
  20.                 closedir($handle);
  21.             }
  22.             return $retval;
  23.         }
  24.  
  25.  
  26.  public function __construct($id) {
  27. //get he array with plugin files
  28.             $usermodules = $this->getPlugins();
  29.             if($usermodules) {
  30.                 foreach($usermodules as $module) {
  31.  
  32.                   //use require to load and import to add the function
  33.  
  34.                    require_once($module[0]);
  35.                    $this->import(new $module[1]);    
  36.                 }
  37.             }
  38.             $this->ID=$id;  
  39.  }
Comments
No Comments »
Categories
php, wordpress
Tags
wordpress
Comments rss Comments rss
Trackback Trackback

« Previous Entries Next Entries »