juust ~ php oddities » serp

how to get bing api search results in MsOffice VBA

admin — Sun, 25 Dec 2011 23:27:21 +0000

For the loyal VBA fans a quick snippet to grab search results from Bing in Access or Excel. There are few articles on the web about that, and that is a shame. So lets add some content on it to the web : first in MsAccess VBA and then in Excel, with a sample workbook added.

Working with the BING API requires an Application Id you can get at the Bing website.

Accessing the Bing Api in MsAccess VBA

In the the VBA Ide, through the menu Tools References I add a reference to the XML object library that contains the XMLHTTPRequest object, enabling me to make HTTP requests from VBA in the MsAccess database.

The simple part is the http request string :

Dim MyKeyword As String
MyKeyword = "seo"
Dim requestString As String
requestString = "http://api.bing.net/xml.aspx?"
Dim AppId As String
AppId = "(get one at the bing website)"
' Common request fields (required)
requestString = requestString & _
"AppId=" & AppId & "&Query=" & MyKeyword & "&Sources=Web"
' Common request fields (optional)
requestString = requestString & _
"&Version=2.0" _
& "&Market=en-us" _
& "&Adult=Moderate" _
& "&Options="
' Web-specific request fields (optional)
requestString = requestString & _
"&Web.Count=10" _
& "&Web.Offset=0" _
& "&Web.Options=DisableHostCollapsing+DisableQueryAlterations"

Then comes the working part, sending the actual request and receiving the XML response (objSvrHTTP.responseText) containing the search data. I use a DomDocument object and XPath to get at the XML data.

First sending the request

objSvrHTTP.Open "GET", requestString, False
objSvrHTTP.send requestString

then creating a domdocument and receiving the response in it

'Create the DomDocument Object
Dim oDoc As MSXML2.DOMDocument
Set oDoc = CreateObject("MSXML2.DOMDocument")
oDoc.async = False
oDoc.validateOnParse = False
'Load the response in the DomDocument Object
Dim fSuccess As Boolean
fSuccess = oDoc.loadXML(objSvrHTTP.responseText)
If Not fSuccess Then
MsgBox "failed"
Exit Sub
End If

The response uses two namespaces and the data we actually want is in the “web” namespace, so before we start selecting nodes in the xml-tree we first indicate what namespace we want to access :

oDoc.SetProperty "SelectionNamespaces", "xmlns:web='http://schemas.microsoft.com/LiveSearch/2008/04/XML/web'"

The total search result pages is a top level node in the namespace :

Set xmlnode = oDoc.selectSingleNode("//web:Total")
Debug.Print xmlnode.Text

…but the good stuff is in the “WebResult” nodes in the xml tree, I can use an XPath reference //web:Webresult to access that node-collection and with a simple for-next iterate through the collection, selecting the single nodes containing the actual data.

Set oChildren = oDoc.selectNodes("//web:WebResult")
For Each oResult In oChildren
Debug.Print oResult.selectSingleNode("./web:Title").Text
Debug.Print oResult.selectSingleNode("./web:Description").Text
Debug.Print oResult.selectSingleNode("./web:Url").Text
Debug.Print oResult.selectSingleNode("./web:DisplayUrl").Text
Debug.Print vbCrLf
Next

I output to the Immediate window aka Debug (under View, Immediate Window or Ctrl+G)

What can we do with it ? Let’s make a table for date, keyword, url, title, description, position

and add some code to store the results we retrieve

Dim table1 As DAO.Recordset
Set table1 = CurrentDb.OpenRecordset("Table1", dbOpenTable)
'counter for position
i = 0
Set oChildren = oDoc.selectNodes("//web:WebResult")
For Each oStruct In oChildren
i = i + 1
With table1
.AddNew
'data from bing
!TITLE = oStruct.selectSingleNode("./web:Title").Text
!Description = oStruct.selectSingleNode("./web:Description").Text
!url = oStruct.selectSingleNode("./web:Url").Text
'and some additional data
!MYDATE = Now()
!POSITION = i
!KEYWORD = MyKeyword
'store the record
.Update
End With
Next
table1.Close

There now that scratches the seo itch :

MsAccess stuff is a bit hard to put on the web as sample, so I’ll make a quick one in Excel and put it up for download.

Accessing the Bing Api in Excel VBA

there you go : on sheet 1 you can fill in your AppID and keyword, click the butt(on…

…and on sheet 2 you get the top 10 search results

It uses the cells B1 and B2 as named ranges for MyKeyword and AppID, and has the code in the enclosed module. It also contains one of my AppId’s so no querying nasty shit, please.

quick note : serp plugin beta

juust — Wed, 02 Sep 2009 17:47:40 +0000

I am developing a new google serp plugin for wordpress. I developed it over the weeknd on wp2.8, it should run on anything with a ‘shutdown’ hook and jquery. I will be adding some other stuff to it over the next few weeks, but I thought I’d put a beta on the blog. If you try it, I hope it runs, send some feedback.

If you have some ideas about cool features drop me a note, I might put it in there. I was pondering on a rest service to pool sem-data and develop some competitor analysis functionality, but that is just a vague idea.

bing

juust — Fri, 21 Aug 2009 02:46:06 +0000

for completeness : php bing serp scraping :

$query = 'serp';
$page = 1;
$start = ($page-1)*10;
$url = 'http://www.bing.com/search?q='.urlencode($query)."&first=".($start+1);
$curl_handle = curl_init();
curl_setopt($curl_handle,CURLOPT_URL, $url);
curl_setopt($curl_handle,CURLOPT_CONNECTTIMEOUT,2);
curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, 1);
$return = curl_exec($curl_handle);
curl_close($curl_handle);
$parts = split('
', $return);
for($j=1;$j<count ($parts);$j++)
{
$p=$parts[$j];
preg_match('#\s+.*?href=[\'"]([^\'"]+)[\'"]\s*(?:title=[\'"]([^\'"]+)[\'"])?.*?>((?:(?!).)*)#i', $p, $urls);
echo "position: ".($start +$j)." url: ".$urls[1]." title: ".$urls[3].'
';
}
count>h3>

serp tool

juust — Mon, 03 Nov 2008 22:20:49 +0000

I was putting together a serp tool with a database and an emailer on a cronjob, mainly because I am lazy, I always get cranky when I have to type in these keywords again, I ain’t a teletubby.

I wanted an automated one with a history that sends me an email every day, so I started putting on together but I got distracted because the gethost server was shut down and my moms Mary Chapel event blog was on it. My old site was also on it, with my crappy blog, but losing that ain’t half as bad as losing yer moms Mary Chapel blog, that is bad karma.

So I got her a domain and put it on one of my accounts, installed a new blog and made sure it ranks number one in google again. Apart from that I was putting together a scraper and still needed some cron jobs for the scraper scripts.

I found a nice free cron job site, standard 5 jobs a day. It’s free and it works flawless, no hassles. Very nice.

Once I had that I remembered I still had to finish that serp thing as well, so I finished the basic routines today. Still needs some testing, and it could use the yahoo and msn serps. I’ll grab them from a wordpress widget and add some language options, after that it should be a fine tool.

I’ll put the scripts up for download in two or three weeks or something.

serp tool 2008

juust — Sun, 21 Sep 2008 23:55:09 +0000

I never got around to building a serp tool with a mysql backend, one on a cronjob with an email option.
This weekend I got an OSWD free template, then it at least looks like a website. I develop a lot easier when it has a template. Filling in the blanks.

Often if you program a rough sketch, in time it sort of develops itself. If you don’t start nothing gets done.

I am developing it on juust.org/serp/, once its finished i’ll put one on trismegistos.net and put the source up for download.

It’s flimsy but the idea of a serp minisite with emailer option is commercially attractive.

It’s also important for the blog to have a ‘serp’ page in the menu, and a serp page should have serp tools, in abundance, all kinds of them.

wordpress serp widget serpent

juust — Mon, 21 Jul 2008 03:37:30 +0000

28-7
the permutation serp creates too much of a load and the results are 90% general so not very usefull per page.

replaced it with a straight top-50 serp on the msn, yahoo, google engines
using post_tags
added a link to the search engine result pages themselves
added a cache and a timer on 6000seconds, it requeries every 1.5 hours when a page is opened and in the meantime dishes out cached results, to minimize queries and page load times
added before_widget and after_widget

it needs

specific options per engine (language etc.)
a dig option up to 1000 result
css hooks
max three keywords constraint
page keywords list (currently only posts’ tags are used)
a mysql backend (I don’t want the nonsense in the wordpress db)
a counter for used queries (otherwise you get ‘no result’)
an archive report to see how the serp results develop

but it’s a start

the results are accurate but i need a once a day routine to grab the keyword of the index page and other pages (the ‘keywords’ section is often dynamic and changes every day, most sites use plugins to either grab all tags of listed posts and put them in the meta-section as page-keywords, or use headspace to override it with a fixed set.)

querying the wordpress database and for all non-post page use get_meta_tags, put url and tags in a cached list and serp the lot once a day would solve that.

something like that.

once I got that fixed i’ll ask some people to test it.

serp pagerank php seo tool

juust — Wed, 09 Jul 2008 01:23:02 +0000

So how did I do with a few days search engine optimisation ? not bad at all, on php+serp nicely on second spot behind shoemoney.com and on serp+php also front page, and on the 5 keys overall 13th. not bad for a PR0 domain.

5-key serp : serp pagerank php seo tool

count	hits	domain
7.8	8	www.juust.org
16	serp pagerank	http://www.juust.org/	0.2
8	serp php	http://www.juust.org/	1
9	serp php	http://www.juust.org/index.php/php-serp-scripts/2008/07/	1
99	serp tool	http://www.juust.org/	0.2
42	pagerank serp	http://www.juust.org/	0.2
3	php serp	http://www.juust.org/	3
4	php serp	http://www.juust.org/index.php/php-serp-scripts/2008/07/	2
70	seo serp	http://www.juust.org/	0.2

and who are the competition :

points	results	domain
20.2	13	www.seochat.com
16.6	21	www.seocompany.ca
15.6	13	sitening.com
14.6	10	www.webconfs.com
13.4	21	forums.digitalpoint.com
11.4	11	www.prchecker.info
11	4	en.wikipedia.org
10	19	www.webmasterworld.com
9	4	www.cristiandarie.ro
8.8	8	www.seroundtable.com
8.4	5	www.shoemoney.com
8.4	5	www.google.com
7.8	8	www.juust.org
7.6	6	tools.seobook.com
7.2	9	www.jumptags.com
7.2	14	www.seomoz.org
7	7	www.seoserp.com
6.4	14	forums.seochat.com
6.4	10	link.ezer.com
6.4	4	www.ljfind.com
5.8	7	www.iwebtool.com

Now, for the seo+tool niche… an online trackback spider…