I actually want the crawlers to come around, so I will make an array with bots and allowed_bots. Whatever is not on the white-list gets a meager page with overhead jobs attached to it, the rest (iow visitors and the big search engines) get the standard page.

There are truckloads of bots (see crawltrack), for my purposes a few regulars will do.

In templates and functions i can use some simple code to run stuff conditional :

It would be nice if WordPress built in a switch to run plugins conditional.

one related smart plugin is the chennai central plugin that sends 304 not modified headers on conditional GETs, so crawlers don’t fetch the page. That can save some bandwidth and serverload.

I use the Incutio xml-rpc library on both hosts, to handle the tedious stuff (xml formatting and parsing). That leaves only some snippets to send and receive table data and store it on a mysql database.

First : how to handle the table data on the sending end:

• I take an associative array from a mysql query
• I make an array to hold the records
• I add each row as array
• I make an IXR-client.
• I add some general parameters
• I hand these and the entire table array to my IXR-client.
• send…

I consider some additional fields necessary for basic integrity checks : I add “ID” as key field, so on the receiving end the server knows which field is my table’s auto-increment field. Other fields are a username, password, tablename and the batch recordcount.

The IXR_Client then generates a tangled mess of xml-tags holding the entire prodecure call and data. (you can put the client on ‘debug’, then it dumps the generated xml to the screen).

The first part of the xml file contains the single parameters :

• username
• password
• tablename
• recordcount
• id-field

<methodCall>
<methodName>serp.putTable</methodName>
<params>
<param><value><string>user</string></value></param>
<param><value><string>pass</string></value></param>
<param><value><string>serp_tags_keys</string></value></param>
<param><value><int>91</int></value></param>
<param><value><string>id</string></value></param>

Then the entire table is sent as one parameter in the procedure call.

That parameter is built from an array containing the table rows as ‘struct’. If I want to use the routine for any table, I need the fieldname-value pairs to compose a standard mysql insert statement. A struct type allows me to use key-value pairs in the xml-file that can be parsed back into an array.

<param><value><array>

<data>

<value><struct>
<member><name>id</name><value><string>4</string></value></member>
<member><name>tag</name><value><string>ranking</string></value></member>
<member><name>cat</name><value><string>alexa ranking seo internet ranking internet positi</string></value></member>
<member><name>date</name><value><string>200901</string></value></member>
</struct></value>

<value><struct>
<member><name>id</name><value><string>94</string></value></member>
<member><name>tag</name><value><string>firm</string></value></member>
<member><name>cat</name><value><string>firm seo</string></value></member>
<member><name>date</name><value><string>200901</string></value></member>
</struct></value>

</data>

</array></value></param>

That was the last of the param holding the table, so the entire tag-mess is closed :

</params&gt</methodCall&gt

Then the second part : on the receiving end the Incutio class parses the whole tag-mess, and hands an array of the param sections as input to my function putTable.

$table is a straightforward array holding as items an array ($t) created from the struct with the pairs of fieldname-value. I turn the recordsets key-value struct into a mysql INSERT query :
$query = “INSERT INTO `”.$tname.”` (” field, field… “) VALUES (” fieldvalue, fieldvalue “)”;

All I have to do is add the fieldnames and fieldvalues to the mysql insert query.

that generates mysql queries like
INSERT INTO `serp_tags_keys` (`tag`, `cat`, `date`) VALUES (‘ranking’, ‘alexa ranking’, ’200901′) and copies the entire table.

That is how I handle the table data.

Of course I have to define two custom classes to process the serp.putTable procedure itself, using the Incutio class.

First the class for the sending script, which is pretty straight forward :

• make an IXR_Client instance
• hand the record set to it
• have it formatted and sent

I use it in the snippets above with :

Then, on the receiving end, my program has to know how to handle the xml containing the remote procedure call.

I define an extension on IXR_server and pass serp.putTable as new ‘method’ (callback function).

That’s all. I am not going to list a cut-and-paste version. You have to build some stuff with it, then you will come up with lots of stuff you can do with it.

WordPress and iPhone built a plugin that receives pictures from iPhone. WordPress uses Incutio so you can ‘piggyback’ on that and have an iPhone plugin for your own website in two days flat using an ajax lightbox gallery script. Or go monetize small websites with some seo oriented ‘optimisation’ functions like ChangeFooterLinks(array($paidurl, $anchortext)) :) or whatever… boring, isn’t it ?

Lucky for me, mister John Watson’s synonym database comes with a free 10.000 request a day API and boy is it sweet!

API Requests are straightforward :
http://words.bighugelabs.com/api/2/[apikey]/[keyword]/xml

A number of return formats are supported but xml is easiest, either for parsing with simplexml or regular pattern matching.

It returns on request :
black (slightly shortened)
an xml file like :
<words>
<w p=”adjective” r=”syn”>bleak</w>
<w p=”adjective” r=”syn”>sinister</w>
<w p=”adjective” r=”sim”>dark</w>
<w p=”adjective” r=”sim”>angry</w>
<w p=”noun” r=”syn”>blackness</w>
<w p=”noun” r=”syn”>inkiness</w>
<w p=”verb” r=”syn”>blacken</w>
<w p=”verb” r=”syn”>melanize</w>
</words>

…which is easiest handled with preg_match_all :

practically applying it,
I take a slab of stale old content and…

• strip tags
• do a regular match on all alphanumeric sequences dropping other stuff
• trim the resulting array elements
• (merge all blog tags, categories, and a list of common words)
• excluding common terms from the array with text elements
• excluding words smaller than N characters
• set a percentage words to be synonimized
• attempt to retrieve synonyms for remaining terms
• replace these words in the original text, keep count
• when I reach the target replacement percentage, abort
• return (hopefully) a revived text

Nothing fancy, a straightforward search-replace routine. A 1200 word text has about 150 candidates and for 3% synonyms I need to replace 36 words, it can do that. If I were to use it for real I would build a table with non-returning terms, and store often used terms, that would speed up the synonimizing, allow the use of preferences and take a load of the api use.

Why use a cache ?

Most people skip sites with long load-times, speeding up load times with a cache is a ‘must-have’ if you plan to develop and optimize a site aimed at getting your traffic through a search engine

I installed Google Analytics a week ago and that affirms it, the rate of people that skipped this site after one page was 85% compared to 15% on another domain with normal load times. Pages took 4 seconds or more to load without a cache, and 1.5 seconds or less with a cache. Average page per visitor was 1.14 for this site and 5.8 for the other.

For advertising purposes, long loading time means a site is unattractive to advertisers. You don’t bind visitors and the ‘bad rep’ your site gets also reflects on advertisers, especially with banner ads that work more visual/emotional (image and experience are linked and the poor performance of the site reflects on the advertisers).

Webslug keep a list on slow loading sites, some of them really extreme with 5 minutes load times. Some sites use a five second boundary, sites with higher loading times are considered unacceptable for advertising programs.

A cache also decreases server-load (which isn’t the foremost reason for wordpress bloggers to install it) and allows for a higher concurrent traffic volume, especially if you become popular with a few dozen visitors at the same time, your server overloads.

Loads of reasons to install a cache.

——–

added 14-09

One other reason is MySql, which can handle 2500 concurrent connections but only about 50 concurrent users, problems arise when you for instance have a large file table or registered user table, that for indexing builds a temporary index table (the ‘digg’-effect where a sudden massive rush on the site makes mysql opt out).

Using any caching mechanism diminishes the load on the mysql server which  effectively solves most problems with the site slowing down. For wordpress, removing unnecessary plug-ins helps, as does a page-cache.But neither solves the actual problem, the mysql database interaction.

WordPress-MU might in time deliver the goods, a stable mysql query caching mechanism, and for now it seems the ‘ease-of-use’ and the simplicity of the source-code dictate a page-cacher will have to do for the standalone version.

So depending on your plans with a website, where WordPress is the ideal tool for a site with up to, say, 1000 pages, if you plan on a larger business strength site use a CMS that does have backend mysql query-caching.

I cached my monkeys, so it don’t take so long to retrieve pictures every time, and added a size switch. where can you retrieve the width of the sidebar the widget instance is in ?

Anyways, when I figured the cache thing out, i reckoned hey, lets cache the serp, so then I cached the serp, now it keeps a ‘current’ file and an archive. works half.

at least my pages don’t take an aeon to load anymore.

tomorrow : more monkey business.

…that is a cute ape

tag links

SeoQuake indicates my second blog has 52 inside links and 14 outbound, 66 links, but my blog is not ranking.

I ran a spider on the site that counts all the anchors (also double links), that reports 120 anchors.

Google possibly exclude anything with 100+ anchors.

I just counted the links manually and found about 50 links, so i wondered ‘where are the other 70?’. So i counted the tag-cloud and realized ‘damn.. the tags…’. What causes it are the tags per post (the main cause) and the number of posts listed on the frontpage. If i use 10 posts per page I easily have (10*4) 40 tag-links, and if google counts ‘em all for onr link and ditches my page I loose the page linking the two blogs. The whole second blog becomes a closed ring with no way out so the entire blog is excluded from page ranking.

I turned the posts per page down to max 3 posts (settings, reading), and removed the ‘recent posts’ widget from the sidebar. Now its down to about 80 links.

If that is all true then tomorrow or by sunday both blogs should start ranking again.

A quick spidering shows the main blog is on 2.20, pages on 1.80, posts on 0.60, and the second blog on 0.60, only the rss feed still has the feed url on ‘follow’ but that’s a minor detail.

That should score quite nicely.

Design and content guidelines (7-2008)

• Make a site with a clear hierarchy and text links. Every page should be reachable from at least one static text link.
• Offer a site map to your users with links that point to the important parts of your site. If the site map is larger than 100 or so links, you may want to break the site map into separate pages.
• Create a useful, information-rich site, and write pages that clearly and accurately describe your content.
• Think about the words users would type to find your pages, and make sure that your site actually includes those words within it.
• Try to use text instead of images to display important names, content, or links. The Google crawler doesn’t recognize text contained in images.
• Make sure that your TITLE tags and ALT attributes are descriptive and accurate.
• Check for broken links and correct HTML.
• If you decide to use dynamic pages (i.e., the URL contains a “?” character), be aware that not every search engine spider crawls dynamic pages as well as static pages. It helps to keep the parameters short and the number of them few.
• Keep the links on a given page to a reasonable number (fewer than 100).

[ad#banner]

Quality – basic principles

• Make pages primarily for users, not for search engines. Don’t deceive your users or present different content to search engines than you display to users, which is commonly referred to as cloaking.
• Avoid tricks intended to improve search engine rankings. A good rule of thumb is whether you’d feel comfortable explaining what you’ve done to a website that competes with you. Another useful test is to ask, “Does this help my users? Would I do this if search engines didn’t exist?”
• Don’t participate in link schemes designed to increase your site’s ranking or PageRank. In particular, avoid links to web spammers or “bad neighborhoods” on the web, as your own ranking may be affected adversely by those links.
• Don’t use unauthorized computer programs to submit pages, check rankings, etc. Such programs consume computing resources and violate our Terms of Service. Google does not recommend the use of products such as WebPosition Gold™ that send automatic or programmatic queries to Google.

[ad#banner]

Here’s the “bad-things” list :

• Avoid hidden text or hidden links.
• Don’t use cloaking or sneaky redirects.
• Don’t send automated queries to Google.
• Don’t load pages with irrelevant keywords.
• Don’t create multiple pages, subdomains, or domains with substantially duplicate content.
• Don’t create pages with malicious behavior, such as phishing or installing viruses, trojans, or other badware.
• Avoid “doorway” pages created just for search engines, or other “cookie cutter” approaches such as affiliate programs with little or no original content.
• If your site participates in an affiliate program, make sure that your site adds value. Provide unique and relevant content that gives users a reason to visit your site first.

[post is on noindex, nofollow]