contact us google

I really wonder how Matt and company figure that one in terms of relevance to the query.

Before we start :

google scraping is bad !

Good People use the Google Adwords API : 25 cents for 1000 units, 15++ units for keyword suggestion so they pay 4 or 5 dollar for 1000 keyword suggestions (if they can find a good programmer which also costs a few dollars). Or they opt for SemRush (also my preference), KeywordSpy, Spyfu, and other services like 7Search PPC programs to get keyword and traffic data and data on their competitors but these also charge about 80 dollars per month for a limited account up to a few hundred per month for seo companies. Good people pay plenty.

We tiny grey webmice of marketing however just want a few estimates, at low or better no cost : like this :

Suggestion autocomplete is AJAX, it outputs XML :

< ?xml version="1.0"? >
   <toplevel>
     <CompleteSuggestion>
       <suggestion data="senior quotes"/>
       <num_queries int="30000000"/>
     </CompleteSuggestion>
     <CompleteSuggestion>
       <suggestion data="senior skip day lyrics"/>
       <num_queries int="441000"/>
     </CompleteSuggestion>
   </toplevel>

Using SimpleXML, the PHP routine is as simple as querying g00gle.c0m/complete/search?, grabbing the autocomplete xml, and extracting the attribute data :

I made a quick php script that outputs the terms as a list of new queries so you can walk through the suggestions :

The source is as text file up for download overhere (rename it to suggestit.php and it should run on any server with php5.* and simplexml).

the feed has three namespaces (atom, opensearch and dxp/analytics), a simple way is accessing the ENTRY tags (from the Atom namespace), in that tag is one DXP: line and that has the answer to the question.

<dxp:metric confidenceInterval=’0.0′ name=’ga:pageviews’ type=’integer’ value=’755′/>

Important is using the (string) typecast, normally simplexml returns a simplexml object, when you force a string type, it gives the actual metric ga:pageview value attribute as number.

An actual google analytics api, and I missed out on it. This api is already a month old and i havent read anything on the blogs about it.

I found it half an hour ago, I havent checked it completely but it looks promising. Here is the first bit, basic authentication with php and curl.

somewhere in the garbled mess that curl returns is the Authorization token, starts with auth=.

I put that token in the header of the next calls and google assumes I am kosher : time to get the accounts feed :

And there it is : a whole list with weird codes, my account list :) seems easier than the other gData api’s.

note : the google code curl example does not show the ” auth=” part of the token, they assume you use the entire line “auth=…” as token.

Once I have my spectacular visitor count in a sidebar widget I’ll blog another post on this one.

It was, indeed, delightful to walk about in the country. In a sunny spot stood a pleasant old forum close by a portal, and from the forum down to the web grew great landing pages, so tall, that under the tallest of them a little child could stand upright. In this snug retreat sat a white-hat on her nest, watching for her young brood to hatch; she was beginning to get tired of her task, for the little ones were a long time coming out of their shells, and she seldom had any visitors. The other white-hats liked much better to surf about in the seomoz than to climb the slippery banks, and have a gossip with her.

At length one shell cracked, and then another, and from each egg came a living creature that lifted its head and cried, “peep, peep.” “Quack, quack,” said the mother, and then they all were all quacks, and quacked as well as they could, and looked about them on every side at the large adsense slots.

Their mother allowed them to look as much as they liked, because green is good for the eyes. “How large the cyberspace is,” said the young white-hats, when they found how much more room they now had than while they were inside the egg-shell. “Do you imagine this is the whole cyberspace?” asked the mother; “Wait till you have seen the google; it stretches far beyond that to the Baidu, but I have never ventured to such a distance. Are you all out?” she continued, rising; “No, I declare, the largest egg lies there still. I wonder how long this is to last, I am quite tired of it;” and she seated herself again on the nest.

“Well, how are you getting on?” asked an old white-hat, who paid her a visit. “One egg is not hatched yet,” said the white-hat, “it will not break. But just look at all the others, are they not the prettiest little white-hatlings you ever saw? They are the image of Big Daddy, who is so unkind, he never comes to see.”

“Let me see the egg that will not break,” said the white-hat; “I have no doubt it is a spammer’s egg. I was persuaded to hatch some once, and after all my care and trouble with the young ones, they were afraid of the web. I quacked and clucked, but all to no purpose. I could not get them to venture in. Let me look at the egg. Yes, that is a spammer’s egg; take my advice, leave it where it is and teach the other children to kiss matt’s butt.”

“I think I will sit on it a little while longer,” said the white-hat; “as I have sat so long already, a few days will be nothing.” “Please yourself,” said the old white-hat, and she went away.

At last the large egg broke, and a young one crept forth crying, “PHP, PHP.”

It was very large and black. The white-hat stared at it and exclaimed, “It is very large and not at all like the others. I wonder if it really is a hacker. We shall soon find it out, however when we go to the web. It must go in, if I have to push it myself.”

On the next day the weather was delightful, and the sun shone brightly on the green burdock leaves, so the mother white-hat took her young brood down to the web, and jumped in with a splash. “Quack, quack,” cried she, and one after another the little white-hatlings jumped in. The web closed over their heads, but they came up again in an instant, and swam about quite prettily with their legs paddling under them as easily as possible, and the black-hatling was also in the web swimming with them.

“Oh,” said the mother, “that is not a spammer; how well he uses his legs, and how upright he holds himself! He is my own child, and he is not so very black after all if you look at him properly. Quack, quack! come with me now, I will take you into grand society, and introduce you to the Dmoz, but you must keep close to me or you may be trodden upon; and, above all, beware of the Aaron.”

When they reached the Dmoz, there was a great disturbance, two families were fighting for a toolbar, which, after all, was carried off by the Aaron. “See, children, that is the way of the world,” said the mother white-hat, whetting her beak, for she would have liked the toolbar herself. “Come, now, use your legs, and let me see how well you can behave. You must bow your heads prettily to that old white-hat yonder; she is the highest born of them all, and has Stanford blood, therefore, she is well off. Don’t you see she has a blue flag tied to her leg, which is something very grand, and a great honor for a white-hat; it shows that every one is anxious not to lose her, as she can be recognized both by man and beast. Come, now, don’t turn your toes, a well-bred white-hatling spreads his feet wide apart, just like his father and mother, in this way; now bend your neck, and say ‘quack.’”

The white-hatlings did as they were bid, but the other white-hat stared, and said, “Look, here comes another brood, as if there were not enough of us already! and what a weird looking object one of them is; we don’t want him here,” and then one flew out and bit him in the neck.

“Let him alone,” said the mother; “he is not doing any harm.”

“Yes, but he is so big and black,” said the spiteful white-hat “and therefore he must be turned out.”

“The others are very pretty children,” said the old white-hat, with the blue rag on her leg, “all but that one; I wish his mother could improve him a little.”

“That is impossible, your grace,” replied the mother; “he is not pretty; but he has a very good disposition, and swims as well or even better than the others. I think he will grow up pretty, and perhaps be smaller; he has remained too long in the egg, and therefore his figure is not properly formed;” and then she stroked his neck and smoothed the feathers, saying, “It is a drake, and therefore not of so much consequence. I think he will grow up strong, and able to take care of himself.”

“The other white-hatlings are graceful enough,” said the old white-hat. “Now make yourself at home, and if you can find a toolbar, you can bring it to me.”

And so they made themselves comfortable; but the poor black-hatling, who had crept out of his shell last of all, and looked so black, was bitten and pushed and made fun of, not only by the white-hats, but by all the googlebots. “He is too big,” they all said, and the DP admin, who had been born into the world with spurs, and fancied himself really an emperor, puffed himself out like a vessel in full sail, and flew at the black-hatling, and became quite red in the head with passion, so that the poor little thing did not know where to go, and was quite miserable because he was so black and laughed at by the whole dmoz. So it went on from day to day till it got worse and worse. The poor black-hatling was driven about by every one; even his brothers and sisters were unkind to him, and would say, “Ah, you black creature, I wish the Aaron would get you,” and his mother said she wished he had never been born. The white-hats pecked him, the chickens beat him, and the girl who fed the poultry kicked him with her feet. So at last he ran away, frightening the little birds in the hedge as he flew over the palings.

“They are afraid of me because I am black,” he said. So he closed his eyes, and flew still farther, until he came out on a large moor, inhabited by wild scripters. Here he remained the whole night, feeling very tired and sorrowful.

In the morning, when the wild scripters rose in the air, they stared at their new comrade. “What sort of a scripter are you?” they all said, coming round him.

He bowed to them, and was as polite as he could be, but he did not reply to their question. “You are exceedingly black,” said the wild scripter, “but that will not matter if you do not want to marry one of our family.”

Poor thing! he had no thoughts of marriage; all he wanted was permission to lie among the rushes, and drink some of the water on the moor. After he had been on the moor two days, there came two wild rappers, or rather youtubbies, for they had not been out of the egg long, and were very saucy. “Listen, friend,” said one of them to the black-hatling, “you are so black, that we like you very well. Will you go with us, and become a bird of passage? Not far from here is another moor, in which there are some pretty wild rapstresses, all unmarried. It is a chance for you to get a wife; you may be lucky, black as you are.”

“Pop, pop,” sounded in the air, and the two wild rappers fell dead among the rushes, and the web was tinged with blogreviews.

< snip (the part where all rappers get shot, google: 50cents if you're into that) >

It would be very sad, were I to relate all the misery and privations which the poor little black-hatling endured during the hard winter; but when it had passed, he found himself lying one morning in a dot org, amongst the PLR blogs. He felt the warm sun shining, and heard the lark singing, and saw that all around was beautiful spring.

Then the young bird felt that his wings were strong, as he flapped them against his sides, and rose high into the air. They bore him onwards, until he found himself in a large forum, before he well knew how it had happened. The threads were in full blossom, and the fragrant profiles bent their long iTraders down to the frontpage which wound round a smooth lounge. Everything looked beautiful, in the freshness of early spring. From a thicket close by came three beautiful webmasters, rustling their feathers, and swimming lightly over the smooth webr. The black-hatling remembered the lovely birds, and felt more strangely unhappy than ever.

“I will POST to those royal birds,” he exclaimed, “and they will blacklist me, because I am so black, and dare to approach them; but it does not matter: better be blacklisted by them than pecked by the white-hats, cursed by the copywriters, pushed about by Mad Butts, or starved with hunger in the winter.”

Then he logged onto the web, and surfed towards the beautiful webmasters. The moment they espied the stranger, they rushed to meet him with outstretched wings. “Blacklist me,” said the poor bird; and he bent his head down to the surface of the web, and awaited banishment.

But what did he see in the clear stream below? His own image; no longer a dark, gray bird, black and disagreeable to look at, but a graceful and beautiful webmaster.

To be born in a white-hat’s nest, in the google, is of no consequence to a bird, if it is hatched from a webmaster’s egg.

He now felt glad at having suffered sorrow and trouble, because it enabled him to enjoy so much better all the pleasure and happiness around him; for the great webmasters swam round the new-comer, and stroked his neck with their beaks, as a welcome.

Presently came some little children, and threw comments and backlinks into the web. “See,” cried the youngest, “there is a new one;” and the rest were delighted, and ran to their father and mother, dancing and clapping their hands, and shouting joyously, “There is another webmaster come; a new one has arrived.”

Then they threw more comments and backlinks into the web, and said, “The new one is the most beautiful of all; he is so young and xhmtl-compliant.” And the old webmasters bowed their heads before him.

Then he felt quite ashamed, and hid his head under his wing; for he did not know what to do, he was so happy, and yet not at all proud. He had been persecuted and despised for his blackness, and now he heard them say he was the most beautiful of all the birds. Even the elder-tree bent down its bows into the web before him, and the sun shone warm and bright. Then he rustled his feathers, curved his slender neck, and cried joyfully, from the depths of his heart, “I never dreamed of such happiness as this, while I was a black white-hatling.”

——-
original story

Just give us your endpoint and we’ll take it from there, sparky!

I was going to make one of these tools to scrape google and conjur a full blog out of nowhere, as Christmas special, RedHat Seo. The rough sketch has arrived , far from perfect, but it does produce a blog and don’t even look too shabby. I scraped a small batch of posts off of blogs, keeping the links intact and adding a tribute links. I hope they will pardon me for it.

structure

I use three main classes,

…and two support classes

Target blogs have three texts,

routine

The BlogMaker class grabs a result list (up to 1000 urls per phrase) from Google, extracts the urls and stores them in SerpResult, scrapes the urls and extracts the entry divs, stores div-entries in the WPContent class (that has some basic functions to sanitize the text), and uses the BlogTarget-definitions to post it up blogs with xml-rpc.

usage

notes

• xml-rpc needs to be activated explicitly on the wordpress dashboard under settings/writing.
• categories must be present in the blog
• url file must be writeable by the server (777)

It seems wordpress builds the sitemap as background process, the standard google xml sitemap plugin wil attempt to build in the cache (takes anywhere between 2 and 10 seconds), and apart from building a sitemap the posts also get pinged around. Giving the install 10 to 20 seconds between posts allows for all the hooked in functions to be completed.

period

That’s about all,
consider it gpl, I added some comments in the source but I will not develop this any further. A mysql backed blogfarm tool (euphemistically called ‘publishing tool’) is more interesting, besides, I am off to the wharves to do some painting.

if you use it, send some feedback,
merry christmas dogheads

The blog articles are pretty straight forward : first the ugly fast way :

That returns the blog snippets, ugly. The other way : regular pattern matching : you can grab the divs that each content item has, marked with

• div class=”gs-title”
• div class=”gs-relativePublishedDate”
• div class=”gs-snippet”
• div class=”gs-visibleUrl”

from the html-source and organize them as “Content” array, after which you can list the content items with your own markup or store them in a database.

It ain’t perfect, but it works. the highlighter I use gets a bit confused about the preg_match_all statements containing unclosed div’s, so copying the code of the blog may not work.