juust ~ php oddities » trends

about the trackback thing

juust — Fri, 24 Apr 2009 03:43:49 +0000

The question about the trends script with trackbacks was wether a few hundred backlinks was worth the trouble, and it wasn’t. I wrote a second routine to grab the most common significant words from excerpts, and do a second search to grab better results and up to five trackbacks per page.

So I put that online, it grabbed 4000 backlinks in an hour and overloaded the host server.

Baidu, radian6 and google had stepped up indexing after I added sitewide tags and that didnt show up in analytics, the site got the trackback validations and crawlers and the server went haywire. It is a shared host, the resources are too limited to run that kind site on. I put it on hold till I find a solution for the hosting,

Google of course penalised the site with PR0 and dropped the domain from the serp on its main keywords, but in Yahoo it ranks about 20 out of 360 million result pages and in MSN it ranks no 1. I was thinking about adding a translator plugin and see if I can get some traffic from Baidu.

trackbacks

juust — Mon, 02 Mar 2009 17:51:31 +0000

Trackbacks are brilliant stuff. I programmed a trackback module into the trends script yesterday just to see what it yields. As long as you don’t use it to spam and stick to common standards, it’s the fastest deep link building method available. I noticed another trends script is also using trackbacks.

GTrends lists an average 600 different searches per day, that makes 200K pages a year. If you put five blog excerpts with a link on a page you have 1000K backlink opportunities a year, automated, if you use trackbacks.

I got 50% success rate in the first tests, so I put it on a cronjob and it seems to level out at 30% successful links. That seemed a bit much, so I checked the PingCrawl plugin Eli (bluehatseo) and joshteam put together for Wordpress. They claim a 80% success rate using Eli’s result scraper, I guess 30% is not aberrant.

For trends, I can’t narrow my search down too much. I need the most recent blogs for the trends buzz. Too narrow searches might exclude the recent news and the script would lose it’s usability. Besides, I figure 10% trackbacks would already be more than enough, a few hundred lines of code with a css template for 100K backlinks a year ain’t bad.

I don’t actually have anything to blog about today, so that’s it.

[added 3-3] ****ing brilliant, 65% trackbacks are accepted, increasing traffic, bots come crawling, finally something that works. Now add proxies.

[added 3-3] bozo style “the script got 4 uniques yesterday!”

Can I be honest ? Dude over at seounderworld gave me a vote of confidence on the trends script and I felt embarrased as the demo looks like shit and didn’t do anything. For scraper basics fine, but it lacked seo potential.

So I added some CSS, validated the source, added caching, gzip, rss-feed, sitemap, and the trackback module. It got 300 uniques yesterday and 400 uniques this morning on its first day out, so it performs better now and I don’t feel so embarrassed anymore.

(nice impression of the trends audience by the way)

I’ll add some proxies to prevent bans and some other stuff, once that’s done I’ll refresh the download.

google trends III

juust — Wed, 24 Dec 2008 23:53:14 +0000

How to get the urls and snippets from the Google Trends details page. The news articles on the details page are listed with an ‘Ajax’ call, they are not sent to the browser in the html source. No easy way to scrape that.

The blog articles are pretty straight forward : first the ugly fast way :

$mytitle='manuel benitez';
$mydate=''; //2008-12-24
$html=file_get_contents('http://www.google.com/trends/hottrends?q='.urlencode($mytitle).'&date=&sa=X');
$start = strpos($html, '
');
$end = strpos($html, '
');
$content = substr($html, $start, $end-$start);
echo $content;
div>div>

That returns the blog snippets, ugly. The other way : regular pattern matching : you can grab the divs that each content item has, marked with

div class=”gs-title”
div class=”gs-relativePublishedDate”
div class=”gs-snippet”
div class=”gs-visibleUrl”

from the html-source and organize them as “Content” array, after which you can list the content items with your own markup or store them in a database.

//I assume $mytitle is taken from the $_GET array.
//array 'Content' with it's members
Class Content {
var $id;
var $title;
var $pubdate;
var $snippet;
var $url;
public function __construct($id) {
$this->id=$id;
}
}
//grab the source from the google page
$html=file_get_contents('http://www.google.com/trends/hottrends?q='.urlencode($mytitle).'&date=&sa=X');
//cut out the part I want
$start = strpos($html, '
');
$end = strpos($html, '
');
$content = substr($html, $start, $end-$start);
//grab the divs that contain title, publish date, snippet and url with regular pattern match
preg_match_all('!
\”gs-title\”>.*?< \/div>!si', $html, $titles);
preg_match_all('!
\”gs-relativePublishedDate\”>.*?< \/div>!si', $html, $pubDates);
preg_match_all('!
\”gs-snippet\”>.*?< \/div>!si', $html, $snippets);
preg_match_all('!
\”gs-visibleUrl\”>.*?< \/div>!si', $html, $urls);
$Contents = array();
//organize them under Content;
$count=0;
foreach($titles[0] as $title) {
//make a new instance of Content;
$Contents[] = new Content($count);
//add title
$Contents[$count]->title=$title;
$count++;
}
$count=0;
foreach($pubDates[0] as $pubDate) {
//add publishing date (contains some linebreak, remove it with strip_tags)
$Contents[$count]->pubdate=strip_tags($pubDate);
$count++;
}
$count=0;
foreach($snippets[0] as $snippet) {
//add snippet
$Contents[$count]->snippet=$snippet;
$count++;
}
$count=0;
foreach($urls[0] as $url) {
//add display url
$Contents[$count]->url=$url;
$count++;
}
//leave $count as is, the number of content-items with a 0-base array
//add rel=nofollow to links to prevent pagerank assignment to blogs
for($ct=0;$ct< $count;$ct++) {
$Contents[$ct]->url = preg_replace('/ target/', ' rel="nofollow" target', $Contents[$ct]->url);
$Contents[$ct]->title = preg_replace('/ target/', ' rel="nofollow" target', $Contents[$ct]->title);
}
//its complete, list all content-items with some markup
for($ct=0;$ct< $count;$ct++) {
echo '
'.$Contents[$ct]->title.'';
echo '
'.$Contents[$ct]->pubdate.':'.$Contents[$ct]->snippet.'
';
echo $Contents[$ct]->url.'
';
}
div>div>div>div>div>div>

It ain’t perfect, but it works. the highlighter I use gets a bit confused about the preg_match_all statements containing unclosed div’s, so copying the code of the blog may not work, a text file with the source code is on trends.trismegistos.net. I added it that snippet to trendinfo.php, works fine.

google trends II

juust — Mon, 22 Dec 2008 01:50:38 +0000

I wanted to reply to a question elsewhere on the site, but a ‘comment’ box isn’t fit for it so I’ll put the reply here. The question was about creating ’search engine friendly’ descriptive URL’s based on keywords from the Google Trends atom feed, listing pages a graph of the trend.

I hacked a quick example together on a subdomain over at trends.trismegistos.net, just to be sure it works.

You can get a site to list http://domain.com/trend_title.html type url’s by using mod_rewrite, an apache module.

In the server directory of the application you can use an .htaccess file to set rules for file access in these folders. When the server gets request from browsers or servers it applies any rewriting rules you define in .htaccess to these requests.

I tried this one :

<ifmodule mod_rewrite.c>
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*).html /trendinfo.php?title=$1
ifmodule>

RewriteEngine On
sets the rewrite mechanism on

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
tell the apache server that rewriteconditions apply to file-requests that are not an existing file (F) or directory (D). If the requested filename is anywhere in the servers file table, the server dishes out that file, otherwise it will try to apply a RewriteRule. Applying the rule generates a new request, if that returns anything, the server dishes that out, otherwise it returns an htpp-404 ‘file not found’.

The actual url rewrite rule is :
RewriteRule ^(.*).html /trendinfo.php?title=$1
which means :

if any filename is requested that satisfies the mask ^(.*).html then

take everything before .html
add that as variable $1 to trendinfo.php?title=$1
see if it sticks

If the browser requests http://domain.com/bob+bowersox.html, the server will assert it is not a file or directory on the server, and test the available rules. When it notices it the requested file ends with .html, it applies the rewrite rule and tries to access http://domain.com/trendinfo.php?title=bob+bowersox.

A browsing user does not notice a thing.

In trendinfo.php I wrote some code to handle the ‘new’ request :

if(!isset($_REQUEST['title'])) {
//if there is no $1, added as title, fake a 404 "file not found" message
echo 'the emptiness…';
} else {
//get the title from the request
$mytitle=htmlentities($_REQUEST['title'], ENT_QUOTES, "UTF-8");
//put the google trends graph url together
$graphurl = 'http://www.google.com/trends/viz?hl=&q=';
$graphurl .= urlencode($mytitle);
$graphurl .= '&date='; //leave date blank to get the current graph
$graphurl .= '&graph=hot_img&sa=X';
echo "";
}

…that outputs the Google trend graph on the url http://domain.com/bob+bowersox.html

I zipped the trends.trismegistos.net program files, but that might be a bit over the top, the download file contains a class that relies on a mysql table being filled every hour with new trends (by cron.php on an apache cron-job), parsing and storing the atom feed of google trends, and listing it as a cross-table in index.php spanning the past 24 hours.

You can also put this in index.php :

$feed = simplexml_load_file('http://www.google.com/trends/hottrends/atom/hourly');
$children = $feed->children('http://www.w3.org/2005/Atom');
$parts = $children->entry;
foreach ($parts as $entry) {
$details = $entry->children('http://www.w3.org/2005/Atom');
$dom = new domDocument();
$html=$details->content;
@$dom->loadHTML($html);
$anchors = $dom->getElementsByTagName('a');
foreach ($anchors as $anchor) {
$url = $anchor->getAttribute('href');
$urltext = $anchor->nodeValue;
echo '.urlencode($urltext).'.html" target="_blank">'.$urltext.' ';
}
}
unset($dom);
unset($anchors);
unset($parts);
unset($feed);

That lists the current 100 google trends with a link. If you use the .htaccess rewrite rules, the server reroutes all the links to trendinfo.php with descriptive urls.

I hope that helps.

the value of google trends

juust — Wed, 12 Nov 2008 00:50:07 +0000

I wondered what the value of Google Trends was. I list it on the serp tool and seriously doubted it’s potential, so I did an experiment and decided to use the webmaistro blog, which was doing 1 hit a day, excellent for a test :). Webmaistro is a blogspot blog and Google own blogspot, generally what I publish on it is indexed fast.

I took a ’spicy’ Google Trend with a very characteristic term ‘ocean of glass’, did a search and picked a page gambling that it might be what people are looking for. I published parts of it on the blog, added some links to the source article and to Flickr and posted it.

Using the search phrase ‘ocean of glass‘ as post title and linking to other resources put the post on the details page of Google Trends for the search phrase.

The next hour I got 30 hits on the blog. Checking the referrers shows them all coming from the Google Trends detail page.

After an hour the traffic stopped. It seems to me Google Trends has it’s search engine marketing value, as traffic source.

parsing the google trends atom feed

juust — Fri, 07 Nov 2008 21:03:13 +0000

Short and sweet : how to grab the google trends atom feed and parse out the links,
in 15 lines.

$feed = simplexml_load_file('http://www.google.com/trends/hottrends/atom/hourly');
$children = $feed->children('http://www.w3.org/2005/Atom');
$parts = $children->entry;
foreach ($parts as $entry) {
$details = $entry->children('http://www.w3.org/2005/Atom');
$dom = new domDocument();
@$dom->loadHTML($details->content);
$anchors = $dom->getElementsByTagName('a');
foreach ($anchors as $anchor) {
$url = $anchor->getAttribute('href');
$urltext = $anchor->nodeValue;
echo 'Link: . $url . '" title="' . $urltext . '">' . $urltext . ' ';
}
}

Requires php with simplexml and dom xml. You could use it for a blogfarm script but that’s about all I can think of.

edited 18-12-08 :
$dom->loadHTML((string) $details->content);
to
@$dom->loadHTML($details->content);