juust ~ php oddities

RedHat Seo : scraper auto-blogging

juust | 26/12/2008

Just give us your endpoint and we’ll take it from there, sparky!

I was going to make one of these tools to scrape google and conjur a full blog out of nowhere, as Christmas special, RedHat Seo. The rough sketch has arrived , far from perfect, but it does produce a blog and don’t even look too shabby. I scraped a small batch of posts off of blogs, keeping the links intact and adding a tribute links. I hope they will pardon me for it.

structure

I use three main classes,

BlogMaker	the application
Target	the blogs you aim for
WPContent	the scraped goodies

…and two support classes

SerpResult	scraped urls
Custom_RPC	a simple rpc-poster

Target blogs have three texts,

file	contents	maintenance
blog categories	category you post under	manual
blog tags	tags you list on the blog	manual
blog urls	urls already used for the blog	system

routine

The BlogMaker class grabs a result list (up to 1000 urls per phrase) from Google, extracts the urls and stores them in SerpResult, scrapes the urls and extracts the entry divs, stores div-entries in the WPContent class (that has some basic functions to sanitize the text), and uses the BlogTarget-definitions to post it up blogs with xml-rpc.

usage

My highlighter tends to mess up text with div markers in it, copying off the blog may not work,
the full text source (about 500 lines) is overhere. Underneath I’ll list the main program loop :

//make main instance
$Blog = new BlogMaker("keyword");
//define a target blog, you can define multiple blogs and refer with code
//then add rpc-url, password and user
//and for every target blog three text-files
$T=$Blog->AddTarget(
'blogcode',
'http://my.blog.com/xmlrpc.php',
'password',
'user',
'keyword.categories.txt',
'keyword.tags.txt',
'keyword.urls.txt'
);
//read the tags, cats and url text files stored on the server
//all retrieved urls are tested, if the target blog already has that
//scraped url, it is discarded.
$T->CSV_GetTags();
$T->List_GetCats();
$T->ReadURL();
//grab the google result list
//use params (pages, keywords) to specify search
$Blog->GoogleResults();
$a=0;
foreach($Blog->Results as $BlogUrl) {
$a++;
echo $BlogUrl->url;
//see if the url isnt used yet
if($T->checkURL(trim($BlogUrl->url))!=true) {
echo '…checking ';
flush();
//if not used, get the source
$BlogUrl->scrape();
//check for divs marked "entry", if they arent there, check "post"
//some blogs use other indications for the content
//but entry and post cover 40%
$entries = $BlogUrl->get_entries();
if(count($entries)<1) {
echo 'no entries…';
flush();
$entries = $BlogUrl->get_posts();
if(count($entries)<1) {
echo 'no posts either…';
//if no entry-post div, mark url as done
$T->RegisterURL($BlogUrl->url);
}
}
$ct=0;
foreach($BlogUrl->WpContentPieces as $WpContent) {
//in the get_entries/get_post function the fragments are stored
//as wpcontent
$ct++;
if($WpContent->judge(2000, 200, 5)) {
$WpContent->tribute(); //add tribute link
$T->settags($WpContent->divcontent); //add tags
$T->postCustomRPC($WpContent->title, $WpContent->divcontent, 1); //1=publish, 0=draft
$T->RegisterURL($WpContent->url); //register use of url
usleep(20000000); //20 seconds break, for sitemapping
}
}
}
}

notes

xml-rpc needs to be activated explicitly on the wordpress dashboard under settings/writing.
categories must be present in the blog
url file must be writeable by the server (777)

It seems wordpress builds the sitemap as background process, the standard google xml sitemap plugin wil attempt to build in the cache (takes anywhere between 2 and 10 seconds), and apart from building a sitemap the posts also get pinged around. Giving the install 10 to 20 seconds between posts allows for all the hooked in functions to be completed.

period

That’s about all,
consider it gpl, I added some comments in the source but I will not develop this any further. A mysql backed blogfarm tool (euphemistically called ‘publishing tool’) is more interesting, besides, I am off to the wharves to do some painting.

if you use it, send some feedback,
merry christmas dogheads

Comments: 1 Comment »
Categories: google, seo, seo tips and tricks, tool, wordpress, xml-rpc
Tags: google, scrape, seo, seo tips and tricks, tool, wordpress, xml-rpc
Comments rss
Trackback

google trends III

juust | 25/12/2008

How to get the urls and snippets from the Google Trends details page. The news articles on the details page are listed with an ‘Ajax’ call, they are not sent to the browser in the html source. No easy way to scrape that.

The blog articles are pretty straight forward : first the ugly fast way :

$mytitle='manuel benitez';
$mydate=''; //2008-12-24
$html=file_get_contents('http://www.google.com/trends/hottrends?q='.urlencode($mytitle).'&date=&sa=X');
$start = strpos($html, '<div class="gsc-resultsbox-visible">');
$end = strpos($html, '<div class="gsc-trailing-more-results">');
$content = substr($html, $start, $end-$start);
echo $content;
</div></div>

That returns the blog snippets, ugly. The other way : regular pattern matching : you can grab the divs that each content item has, marked with

div class=”gs-title”
div class=”gs-relativePublishedDate”
div class=”gs-snippet”
div class=”gs-visibleUrl”

from the html-source and organize them as “Content” array, after which you can list the content items with your own markup or store them in a database.

//I assume $mytitle is taken from the $_GET array.
//array 'Content' with it's members
Class Content {
var $id;
var $title;
var $pubdate;
var $snippet;
var $url;
public function __construct($id) {
$this->id=$id;
}
}
//grab the source from the google page
$html=file_get_contents('http://www.google.com/trends/hottrends?q='.urlencode($mytitle).'&date=&sa=X');
//cut out the part I want
$start = strpos($html, '<div class="gsc-resultsbox-visible">');
$end = strpos($html, '<div class="gsc-trailing-more-results">');
$content = substr($html, $start, $end-$start);
//grab the divs that contain title, publish date, snippet and url with regular pattern match
preg_match_all('!<div class=\”gs-title\”>.*?< \/div>!si', $html, $titles);
preg_match_all('!<div class=\”gs-relativePublishedDate\”>.*?< \/div>!si', $html, $pubDates);
preg_match_all('!<div class=\”gs-snippet\”>.*?< \/div>!si', $html, $snippets);
preg_match_all('!<div class=\”gs-visibleUrl\”>.*?< \/div>!si', $html, $urls);
$Contents = array();
//organize them under Content;
$count=0;
foreach($titles[0] as $title) {
//make a new instance of Content;
$Contents[] = new Content($count);
//add title
$Contents[$count]->title=$title;
$count++;
}
$count=0;
foreach($pubDates[0] as $pubDate) {
//add publishing date (contains some linebreak, remove it with strip_tags)
$Contents[$count]->pubdate=strip_tags($pubDate);
$count++;
}
$count=0;
foreach($snippets[0] as $snippet) {
//add snippet
$Contents[$count]->snippet=$snippet;
$count++;
}
$count=0;
foreach($urls[0] as $url) {
//add display url
$Contents[$count]->url=$url;
$count++;
}
//leave $count as is, the number of content-items with a 0-base array
//add rel=nofollow to links to prevent pagerank assignment to blogs
for($ct=0;$ct< $count;$ct++) {
$Contents[$ct]->url = preg_replace('/ target/', ' rel="nofollow" target', $Contents[$ct]->url);
$Contents[$ct]->title = preg_replace('/ target/', ' rel="nofollow" target', $Contents[$ct]->title);
}
//its complete, list all content-items with some markup
for($ct=0;$ct< $count;$ct++) {
echo '<h3>'.$Contents[$ct]->title.'';
echo '<p><strong>'.$Contents[$ct]->pubdate.'</strong>:<em>'.$Contents[$ct]->snippet.'</em></p>';
echo $Contents[$ct]->url.'<br />';
}
</div></div></div></div></div></div>

It ain’t perfect, but it works. the highlighter I use gets a bit confused about the preg_match_all statements containing unclosed div’s, so copying the code of the blog may not work, a text file with the source code is on trends.trismegistos.net. I added it that snippet to trendinfo.php, works fine.

Comments: 5 Comments »
Categories: google
Tags: google, scrape, trends
Comments rss
Trackback

google trends II

juust | 22/12/2008

I wanted to reply to a question elsewhere on the site, but a ‘comment’ box isn’t fit for it so I’ll put the reply here. The question was about creating ’search engine friendly’ descriptive URL’s based on keywords from the Google Trends atom feed, listing pages a graph of the trend.

I hacked a quick example together on a subdomain over at trends.trismegistos.net, just to be sure it works.

You can get a site to list http://domain.com/trend_title.html type url’s by using mod_rewrite, an apache module.

In the server directory of the application you can use an .htaccess file to set rules for file access in these folders. When the server gets request from browsers or servers it applies any rewriting rules you define in .htaccess to these requests.

I tried this one :

<ifmodule mod_rewrite.c>
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*).html /trendinfo.php?title=$1
</ifmodule>

RewriteEngine On
sets the rewrite mechanism on

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
tell the apache server that rewriteconditions apply to file-requests that are not an existing file (F) or directory (D). If the requested filename is anywhere in the servers file table, the server dishes out that file, otherwise it will try to apply a RewriteRule. Applying the rule generates a new request, if that returns anything, the server dishes that out, otherwise it returns an htpp-404 ‘file not found’.

The actual url rewrite rule is :
RewriteRule ^(.*).html /trendinfo.php?title=$1
which means :

if any filename is requested that satisfies the mask ^(.*).html then

take everything before .html
add that as variable $1 to trendinfo.php?title=$1
see if it sticks

If the browser requests http://domain.com/bob+bowersox.html, the server will assert it is not a file or directory on the server, and test the available rules. When it notices it the requested file ends with .html, it applies the rewrite rule and tries to access http://domain.com/trendinfo.php?title=bob+bowersox.

A browsing user does not notice a thing.

In trendinfo.php I wrote some code to handle the ‘new’ request :

if(!isset($_REQUEST['title'])) {
//if there is no $1, added as title, fake a 404 "file not found" message
echo 'the emptiness…';
} else {
//get the title from the request
$mytitle=htmlentities($_REQUEST['title'], ENT_QUOTES, "UTF-8");
//put the google trends graph url together
$graphurl = 'http://www.google.com/trends/viz?hl=&q=';
$graphurl .= urlencode($mytitle);
$graphurl .= '&date='; //leave date blank to get the current graph
$graphurl .= '&graph=hot_img&sa=X';
echo "<img class=hotGraph width=280 height=190 src='$graphurl'/>";
}

…that outputs the Google trend graph on the url http://domain.com/bob+bowersox.html

I zipped the trends.trismegistos.net program files, but that might be a bit over the top, the download file contains a class that relies on a mysql table being filled every hour with new trends (by cron.php on an apache cron-job), parsing and storing the atom feed of google trends, and listing it as a cross-table in index.php spanning the past 24 hours.

You can also put this in index.php :

$feed = simplexml_load_file('http://www.google.com/trends/hottrends/atom/hourly');
$children = $feed->children('http://www.w3.org/2005/Atom');
$parts = $children->entry;
foreach ($parts as $entry) {
$details = $entry->children('http://www.w3.org/2005/Atom');
$dom = new domDocument();
$html=$details->content;
@$dom->loadHTML($html);
$anchors = $dom->getElementsByTagName('a');
foreach ($anchors as $anchor) {
$url = $anchor->getAttribute('href');
$urltext = $anchor->nodeValue;
echo '<a href="'.urlencode($urltext).'.html" target="_blank">'.$urltext.'</a> ';
}
}
unset($dom);
unset($anchors);
unset($parts);
unset($feed);

That lists the current 100 google trends with a link. If you use the .htaccess rewrite rules, the server reroutes all the links to trendinfo.php with descriptive urls.

I hope that helps.