Blacknorati

For our next exercise, dark ones, I present :

Blacknorati

Give us your xml-rpc endpoint, password and username, chose your tags or your favorite author, and we’ll take it from there, Sparky!

A xml-rpc client wired to a streaming blog pipe, 400.000 posts a day. I could build an aggregator with
that, Technorati II, or what I like more, wire it to a blog-farm and reproduce the entire stream, Blacknorati !

I reckon one should program a ‘frame’, front to end, and once you got a working concept you go tune the components. So how does the frame look ?

  • 1 take a streaming blog pipe
  • 2 parse out the feed fragments with php
  • 3 store em in a mysql database
  • 4 run entries through a synonym table
  • 5 re-post with xml-rpc

That doesn’t seem too hard ?

1 take a blog pipe

I did some work on the sixapart atom stream a few months ago : a constant updating stream of blogposts in Atom format, you can pick up on the details on their page on the sixapart atom format from their website. Sixapart use a basic Atom format with an added time notification and a header.

2 parse out the feed fragments

I got the pipe, now I need a pipe reader. With my meager mental facilities I fantasized a new stream client together, this time with the SimpleXml parser.

What does it do ?

  • log onto the pipe
  • strip the initial header
  • store the output in a string
  • on /feed (end tag) parse out time notifications
  • add a standard xml-header
  • hand the string to the simplexml parser.
  • (store in database)
  • continue

Using php stream_set_blocking() commands makes it 1:1 with the pipe, don’t ask me how it works, it just does that, simply accepting that makes it easier to handle the stream ;)

  1.  
  2.  $fnr = fopen('http://updates.sixapart.com/atom-stream.xml', "r");
  3.  $mytime = time();
  4.  $inStream = 0;
  5.  
  6.  while(1) {
  7.   $buffer='';
  8.   stream_set_blocking($fnr, FALSE);
  9.   $buffer = fread($fnr, 1600);
  10.   stream_set_blocking($fnr, TRUE);
  11.  
  12.   if(trim($buffer)=='') {
  13.   } else {
  14.    if($inStream<1) {
  15.     $tpos = strpos(trim($buffer),'–>');
  16.     if($tpos>0) {
  17.      $buffer=substr($buffer, $tpos+4);
  18.      $inStream=1;
  19.     }
  20.    }
  21.  
  22.    if($inStream==1) {
  23.      $string = $string.$buffer;
  24.      flush();
  25.      if(strpos($string,'')>0) {
  26.       $mystring=substr($string,0,strpos($string,''));
  27.      
  28.       $remstring=substr($string, strpos($string,'')+7);
  29.       $string=$remstring;
  30.       $tstring = preg_replace('/<time (.*?)\/time>/', '', $mystring);
  31.       $xstring="< ?xml version='1.0' encoding='utf-8' ?>".$tstring.'';
  32.  
  33.  
  34.       $xml = simplexml_load_string($xstring);
  35.       if(!$xml) { } else {
  36.        echo '<strong>has xml</strong>';
  37.        foreach($xml->entry as $e){
  38.         echo $e->title;
  39.         echo $e->link['href'];
  40.         echo $e->published;
  41.         echo $e->updated;
  42.         //echo $e->content;
  43.        }
  44.       flush();
  45.       unset($xml);
  46.       unset($mystring);
  47.       unset($xstring);
  48.       unset($tstring);
  49.       unset($upto);
  50.       }
  51.      }
  52.     }
  53.    }
  54.   }
  55.  
  56. </time>

(tune, tune, tune….) okay that seems to work… so now we’re gonna add a database where we store author-url, title, content :

3 store em in a database

We make one mysql table for the feed data

  1. CREATE TABLE `feed` (
  2.   `id` bigint(11) NOT NULL auto_increment,
  3.   `date` varchar(12) NOT NULL,
  4.   `url` varchar(120) NOT NULL,
  5.   `entries` varchar(3) NOT NULL,
  6.   PRIMARY KEY  (`id`)
  7. ) ENGINE=MyISAM

I’m not using that yet, that’s for later on…

and one mysql table for the feed entries

  1.  CREATE TABLE `entry` (
  2. `id` BIGINT( 11 ) NOT NULL ,
  3. `title` VARCHAR( 150 ) NOT NULL ,
  4. `feedid` BIGINT( 11 ) NOT NULL ,
  5. `url` VARCHAR( 150 ) NOT NULL ,
  6. `content` BLOB NOT NULL ,
  7. `tags` VARCHAR( 150 ) NOT NULL ,
  8. `datep` VARCHAR( 12 ) NOT NULL ,
  9. `dateu` VARCHAR( 12 ) NOT NULL ,
  10. PRIMARY KEY ( `id` )
  11. ) ENGINE = MYISAM

Then we cut and paste a simple connect function

  1. function connect_data() {
  2.   $DB_USER =  "";
  3.   $DB_PASSWORD = "";
  4.   $DB_HOST = "";
  5.   $DB_DATA = "";
  6.   $link =  mysql_connect($DB_HOST, $DB_USER, $DB_PASSWORD) or $error = mysql_error();
  7.   if (!$link) {
  8.       return $error;
  9.   }  
  10.      mysql_select_db($DB_DATA, $link) or $error = mysql_error();
  11.   return $link;
  12.  }

and make a query and store the stuff from the simplexml $xml object in the database :

  1.  $entrylink=connect_data();
  2.  foreach($xml->entry as $e){
  3.                 $myqry="INSERT INTO entry (`feedid`, `title`,`url`,`datep`,`dateu`,`content`) VALUES (
  4.  '0',
  5.  '".htmlentities($e->title, ENT_QUOTES, 'UTF-8')."',
  6.  '".htmlentities($e->link['href'], ENT_QUOTES, 'UTF-8')."',
  7.  '".htmlentities($e->published, ENT_QUOTES, 'UTF-8')."',
  8.  '".htmlentities($e->updated, ENT_QUOTES, 'UTF-8')."',
  9.  '".htmlentities($e->content, ENT_QUOTES, 'UTF-8')."')";
  10.      $myresult=mysql_query($myqry, $entrylink) or die ('entry error '.mysql_error());
  11.  }
  12.  mysql_close($entrylink);

Wow dude, ten posts a second (don’t forget to put a timer in the routine this one got no breaks)

4 run it through a synonym table

Then we get a synonym database, and make a routine to edit a post (3 out of 100 words) and generate unique new content.

nice : mister John Watson made a synonym database with an API, 10.000 requests a day for free, halleluja. There is WordWeb and FreeThesaurus.net if you are into scraping, but I’ll just take the one with the API.

(not interesting for now, first I wanna make a post, if I cannot post why bother with the rest ?)

5 bombs away! long range xml-rpc payload

Let’s pick a remote blog and fire the post with XML-RPC (if that is installed).
(code from nickycakes.com)

  1. function wpPostXMLRPC($title,$body,$rpcurl,$username,$password,$categories=array(1)){
  2.     $categories = implode(",", $categories);
  3.     $XML = "<title>$title</title>"."<category>$categories</category>".$body;
  4.     $params = array('','',$username,$password,$XML,1);
  5.     $request = xmlrpc_encode_request('blogger.newPost',$params);
  6.  
  7.     $ch = curl_init();
  8.     curl_setopt($ch, CURLOPT_POSTFIELDS, $request);
  9.     curl_setopt($ch, CURLOPT_URL, $rpcurl);
  10.     curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  11.     curl_setopt($ch, CURLOPT_TIMEOUT, 1);
  12.     curl_exec($ch);
  13.     curl_close($ch);
  14. }

my ‘target’ : pagerank.livejournal.com.
woops, oh dear, no php xmlrpc installed, what now ? Darn lets do it ‘donkey’-style, lets just code an xml payload the old fashioned way : short and sweet :

  1.  
  2.     $ch = curl_init('http://www.livejournal.com/interface/blogger');
  3.     curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: text/xml'));
  4.     curl_setopt($ch, CURLOPT_POSTFIELDS, getmyxml());
  5.     curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  6.     curl_setopt($ch, CURLOPT_TIMEOUT, 1);
  7.     $return = curl_exec($ch);
  8.     curl_close($ch);
  9.     print_r($return);
  10.  
  11. function getmyxml() {
  12. $myxml='< ?xml version="1.0" encoding="UTF-8"?>';
  13. $myxml.='<methodcall>';
  14. $myxml.='<methodname>blogger.newPost</methodname>';
  15. $myxml.='<params>';
  16. $myxml.='<param>';
  17. $myxml.='<value><string>blogname</string></value>';
  18. $myxml.='</param>';
  19. $myxml.='<param>';
  20. $myxml.='<value><int></int></value>';
  21. $myxml.='</param>';
  22. $myxml.='<param>';
  23. $myxml.='<value><string>blogname</string></value>';
  24. $myxml.='</param>';
  25. $myxml.='<param>';
  26. $myxml.='<value><string>password</string></value>';
  27. $myxml.='</param>';
  28. $myxml.='<param>';
  29. $myxml.='<value><string>This is not a lovesong</string></value>';
  30. $myxml.='</param>';
  31. $myxml.='<param>';
  32. $myxml.='<value><int>1</int></value>';
  33. $myxml.='</param>';
  34. $myxml.='</params>';
  35. $myxml.='</methodcall>';
  36. return $myxml;
  37. }

LiveJournal Is Your Friend! One line posted successful on the LiveJournal bloggy.

Now let us wire it to the database :)

  1. function connect_data() {
  2.   $DB_USER =  "";
  3.   $DB_PASSWORD = "";
  4.   $DB_HOST = "";
  5.   $DB_DATA = "";
  6.   $link =  mysql_connect($DB_HOST, $DB_USER, $DB_PASSWORD) or $error = mysql_error();
  7.   if (!$link) {
  8.       return $error;
  9.   }  
  10.      mysql_select_db($DB_DATA, $link) or $error = mysql_error();
  11.   return $link;
  12.  }
  13.  
  14. $mylink = connect_data();
  15. $myrst = "SELECT * FROM entry";
  16. $mytbl = mysql_query($myrst, $mylink);
  17.  
  18. while($row=mysql_fetch_assoc($mytbl)) {
  19.  
  20.     $ch = curl_init('http://www.livejournal.com/interface/blogger');
  21.  curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: text/xml'));
  22.     curl_setopt($ch, CURLOPT_POSTFIELDS, getmyxml($row['content']));
  23.     curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  24.     curl_setopt($ch, CURLOPT_TIMEOUT, 1);
  25.  
  26.  
  27.     $ret = curl_exec($ch);
  28.     curl_close($ch);
  29.  print_r($return);
  30.  flush();
  31. }
  32.  
  33. function getmyxml($content) {
  34.  
  35. $myxml='< ?xml version="1.0" encoding="UTF-8"?>';
  36. $myxml.='<methodcall>';
  37. $myxml.='<methodname>blogger.newPost</methodname>';
  38. $myxml.='<params>';
  39. $myxml.='<param>';
  40. $myxml.='<value><string>the_blogid</string></value>';
  41. $myxml.='</param>';
  42. $myxml.='<param>';
  43. $myxml.='<value><int></int></value>';
  44. $myxml.='</param>';
  45. $myxml.='<param>';
  46. $myxml.='<value><string>the_bloguser</string></value>';
  47. $myxml.='</param>';
  48. $myxml.='<param>';
  49. $myxml.='<value><string>the_password</string></value>';
  50. $myxml.='</param>';
  51. $myxml.='<param>';
  52. $myxml.='<value><string>'.$content.'</string></value>';
  53. $myxml.='</param>';
  54. $myxml.='<param>';
  55. $myxml.='<value><int>1</int></value>';
  56. $myxml.='</param>';
  57. $myxml.='</params>';
  58. $myxml.='</methodcall>';
  59. return $myxml;
  60. }

Wow, dude, massive online blog reproduction, we got a live one…

That ends today’s discourse on how to build a basic massive xml-rpc real time care free blog-farming tool. Now over the next few weeks I will tune the system and found and one day you shall witness the Rise of Blacknorati.

So what’s on the menu ?

* synonyms on tags
* excluding non-english posts
* acquiring new blogs
* adding some nice anchors here and there
* replace the blogger.newPost with MetaWeblog.newPost
* etcetera.

I got the xml-rpc remote post snippet from nickycakes

I got the basic ideas on pipes from Chris Chabot (don’t ask me how?)

I stumbled on Joseph Scott’s blog, seems one of the developers of the Wordpress XML-RPC module.

Tags
, , ,
Comments rss
Comments rss
Trackback
Trackback

Leave a comment

You can use these tags : <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

908 confirmed spam kills