I love keyword research. That’s a good thing because as a Search Marketing Professional, I do a ton of it. One thing that’s important for keyword research is using a variety of sources to keep your results fresh. Another benefit of having several keyword research sources is having a diverse audience.

When performing keyword research for Paid Search, estimating potential traffic volume is less important than for search engine optimization. As I’ve posted about several times before, my favorite style of keyword research (when search volume estimates don’t matter) is scraping search/keyword suggestion tools. I’ve been using Blekko for about a year now, both as a search engine but also for leveraging their search suggest functionality. It’s only fitting that I show a simple command for returning Blekko keyword suggestions using a Bash shell.

Blekko Keyword Suggestions

Here’s the minimum command to run to get back formatted keyword suggestions from Blekko.

curl -s "http://blekko.com/autocomplete?query=keyword+research" | sed 's/.*\[//g;s/\].*//g;s/","/\n/g;s/"//g'

The only portion of the command above that you’ll need to modify is the keyword+research text, which is equal to the root seed keyword (with spaces replaced with a plus sign). Running the command above produces these results.

keyword research
keyword research tips
keyword research tool
keyword research services
keyword research seo tool
keyword research pro
keyword research software
keyword research seo
keyword research 2
keyword research service

Blekko’s search suggest tool is limited to 10 results at a time, which is very common. If you’ve read my other posts on how this is done in Google, Yahoo, Bing, etc. then you’ll know that I commonly query several search suggestion sources with one command to make this research and comparison even more simple. Below is this Blekko code integrated with my suggestion tool which combines Google, Amazon, Yahoo, Bing and now Blekko.

#!/bin/bash

q=$(echo "$1" | sed 's/ /%20/g')

clear
echo -e "\nGetting Suggestions for \"$1\""

echo -e "\nGoogle:"
curl -s "http://www.google.com/s?sugexp=pfwl&cp=15&q=$q" | sed 's/\[/\n\[/g' | cut -d'"' -f2 | tail -n +4

echo -e "\nAmazon:"
curl -s "http://t1-completion.amazon.com/search/complete?method=completion&q=$q&search-alias=aps&client=amazon-search-ui&mkt=1&x=updateISSCompletion&sc=1" | sed 's/,\[{".*//g;s/,/\n/g' | cut -d'"' -f2 | grep -v '\[\|\]\|\{\|\]' | tail -n +2

echo -e "\nYahoo:"
y_1=$(curl -s "http://sugg.us.search.yahoo.net/gossip-us-ura?droprotated=1&output=sd1&command=$q&nresults=10")
# this way uses our keyword as a suffix rather than prefix. usually there are some duplicates with the first method but those are removed later.
y_2=$(curl -s "http://sugg.us.search.yahoo.net/gossip-us-ura?droprotated=0&output=sd1&command=$q&nresults=10")
echo $y_1 $y_2 | sed 's/{/\n{/g' | grep '"k"' | cut -d'"' -f4 | sort -u

echo -e "\nBing:"
curl -s "http://api.bing.com/qsonhs.aspx?FORM=ASAPIW&mkt=en-US&type=cb&cb=sa_inst.apiCB&q=$q&cp=13&bq=$q" | sed 's/{/\n{/g' | grep '"Txt"' | cut -d'"' -f4
echo

echo "Blekko:"
curl -s "http://blekko.com/autocomplete?query=$q" | sed 's/.*\[//g;s/\].*//g;s/","/\n/g;s/"//g'
echo; echo

If you save the script as sugg.sh, you’ll be able to run it like this.

./sugg.sh "keyword tool"

That will produce the following results:

Getting Suggestions for "keyword tool"

Google:
keyword tool
keyword tool free
keyword toolbox
keyword tool youtube
keyword tool estimator
keyword tool dominator
keyword tool by google
keyword tool traffic estimator
keyword tool seo
keyword tool reviews

Amazon:
keyword tool

Yahoo:

Bing:
keyword tool bing
keyword tool google
keyword tool
keyword tool external
keyword tools free
keyword tool dominator
keyword tool adwords
keyword tool wordtracker

Blekko:
keyword tool
keyword tool api
keyword tool search
keyword tool software
keyword tool download
keyword tool 2
keyword tool review
keyword tool ppc
keyword tool crack
keyword tool training

Pretty cool right! Unfortunately, Yahoo doesn’t seem to have any knowledge of what a “keyword tool” is. I can’t say I’m surprised.

Keyword Suggestion Enumeration

I mentioned above that Blekko limits the keyword suggestion results to 10 phrases and that it’s very common to do so. I’ve also posted about how it’s trivial to start with a root phrase, like “keyword tools”, and then add each letter of the alphabet on to the end to see what results get returned. Doing that for every letter of the alphabet, removing duplicates and then sorting alphabetically is the next natural step in the evolution of any keyword suggestion based keyword research tool.

So, here’s the code for my keyword suggestion enumeration tool with the Blekko code added in. This takes a bit longer to run because it’s sending many, many HTTP GET requests and has to wait for the search engine’s responses. This version reaches out to 5 different search suggestion services and makes a total of 6 different queries per letter of the alphabet, which equals 156 requests. Think about how long that would take you to manually place those searches in your browser and then note the results. Waiting 10 or 15 seconds for a result doesn’t seem so bad now!

Anyway, here is the code.

#!/bin/bash

q=$(echo "$1" | sed 's/ /%20/g')
tmp=tmp.txt

echo "" > $tmp
for suffix in {a..z}
do
    curl -s "http://www.google.com/s?sugexp=pfwl&cp=15&q=$q%20$suffix" | sed 's/\[/\n\[/g' | cut -d'"' -f2 | tail -n +4 >> $tmp
    curl -s "http://t1-completion.amazon.com/search/complete?method=completion&q=$q%20$suffix&search-alias=aps&client=amazon-search-ui&mkt=1&x=updateISSCompletion&sc=1" | sed 's/,\[{".*//g;s/,/\n/g' | cut -d'"' -f2 | grep -v '\[\|\]\|\{\|\]' | tail -n +3 >> $tmp
    curl -s "http://sugg.us.search.yahoo.net/gossip-us-ura?droprotated=1&output=sd1&command=$q%20$suffix&nresults=10" | sed 's/{/\n{/g' | grep '"k"' | cut -d'"' -f4 >> $tmp
    # this way uses our keyword as a suffix rather than prefix. usually there are some duplicates with the first method but those are removed later.
    curl -s "http://sugg.us.search.yahoo.net/gossip-us-ura?droprotated=0&output=sd1&command=$q%20$suffix&nresults=10" | sed 's/{/\n{/g' | grep '"k"' | cut -d'"' -f4 >> $tmp
    curl -s "http://api.bing.com/qsonhs.aspx?FORM=ASAPIW&mkt=en-US&type=cb&cb=sa_inst.apiCB&q=$q%20$suffix&cp=13&bq=$q" | sed 's/{/\n{/g' | grep '"Txt"' | cut -d'"' -f4 >> $tmp
    curl -s "http://blekko.com/autocomplete?query=$q" | sed 's/.*\[//g;s/\].*//g;s/","/\n/g;s/"//g' >> $tmp
    echo >> $tmp
done

sed '/^$/d' $tmp | sort | uniq

If you save it as “enum_sugg.sh”, here’s how you would run it.

./enum_sugg.sh "keyword tools"

And here’s the result you would get (well, at least this is what I get right now).

adwords keyword tools external
best free keyword tools
best keyword tools 2012
free keyword tools online
google adwords keyword tools external
google keyword tool kit
google keyword tools google searches
google keyword tools uk
google keyword tool youtube
keyword density tools
keyword discovery tools
keyword lookup tool
keyword niche tools
keyword optimization tools
keyword phrase tool
keyword preview tool
keyword question tool
keyword research tools 2012
keyword research tools free
keyword research tools review
keyword search tool yahoo
keyword selector tool yahoo
keyword suggestion tool youtube
keyword tool adsense
keyword tool analyzer
keyword tool api
keyword tool average search volume
keyword tool bar
keyword tool beta
keyword tool bing
keyword tool book
keyword tool comma seperated
keyword tool copy list free
keyword tool density
keyword tool discover
keyword tool dominator
keyword tool download
keyword tool estimator
keyword tool external adwords
keyword tool external google
keyword tool finder
keyword tool find little competion warrior
keyword tool for app store
keyword tool for niche ideas
keyword tool generator
keyword tool golf instruction
keyword tool google adwords
keyword tool google external
keyword tool google too many searches
keyword tool in english
keyword tool in google
keyword tool in spanish
keyword tool kit
keyword tool microsoft word
keyword tool multilple
keyword tool multiple
keyword tool old
keyword tool old version
keyword tool overture
keyword tool plr
keyword tool previous interface
keyword tool previous months
keyword tools accuracy
keyword tools accurate
keyword tools adsense
keyword tools adwords
keyword tools adwords google
keyword tools affiliate
keyword tools alexa
keyword tools analysis
keyword tools articles
keyword tools available
keyword tools based
keyword tools beginners
keyword tools best
keyword tools beta
keyword tools bing
keyword tools blog
keyword tools by google
keyword tools by microsoft
keyword tools campaign
keyword tools canada
keyword tools chinese
keyword tools commercial intent
keyword tools compared
keyword tools comparison
keyword tools competition
keyword tools competitors
keyword tools content
keyword tools cpc
keyword tools de adwords
keyword tools dictionary
keyword tools digitalpoint
keyword tools directory
keyword tools disadvantages
keyword tools download
keyword tools dreamweaver
keyword tool search
keyword tool search by city
keyword tool search google
keyword tool search on yahoo google and msn
keyword tools ebay
keyword tools english
keyword tool seo
keyword tool seo organic
keyword tools estimator
keyword tools exact match
keyword tools excel
keyword tools external
keyword tools external google
keyword tools external suggestion tool
keyword tools for adsense
keyword tools for adwords
keyword tools for analysis
keyword tools for bing
keyword tools for ebay
keyword tools for mac
keyword tools for seo
keyword tools for yahoo
keyword tools free
keyword tools free in google
keyword tools from google
keyword tools generator
keyword tools google
keyword tools google adsense
keyword tools google adwords
keyword tools google old version
keyword tools google search global monthly
keyword tools google uk
keyword tools google vs wordtracker
keyword tools guide
keyword tools help
keyword tools home
keyword tools html
keyword tools html code
keyword tools hubpages
keyword tools ideas
keyword tool simple
keyword tools in english
keyword tools in google
keyword tools internet
keyword tools in yahoo
keyword tools ireland
keyword tools kei
keyword tools keywordspy
keyword tools link
keyword tools link building
keyword tools linux
keyword tools list
keyword tools local searches
keyword tools location
keyword tools long tail
keyword tools mac
keyword tools mac free
keyword tools malaysia
keyword tools market
keyword tools market samurai
keyword tools microsoft
keyword tools misspell
keyword tools moneyword matrix
keyword tools msn
keyword tools music
keyword tools nichebot
keyword tools niche finder
keyword tool software
keyword tools old interface
keyword tools omniture
keyword tools on google
keyword tools online
keyword tools online local
keyword tools on the internet
keyword tools on yahoo
keyword tools organic seo
keyword tools overture
keyword tools page optimization
keyword tools paid search
keyword tools pay per click
keyword tools pdf
keyword tools photos
keyword tools ppc
keyword tools price
keyword tools rank
keyword tools rapid
keyword tools rating
keyword tools remove duplicates
keyword tools research
keyword tools resume
keyword tools review
keyword toolss
keyword tools samurai
keyword tools search
keyword tools search engine optimization
keyword tools seo
keyword tools seobook
keyword tools site
keyword tools software
keyword tools south africa
keyword tools spy
keyword tools statistics
keyword tools techniques
keyword tools that work
keyword tools tips
keyword tools to compete
keyword tools tracker
keyword tools traffic
keyword tools traffic estimator
keyword tools traffic travis
keyword tools trends
keyword tools tricks
keyword tools twitter
keyword tools uk
keyword tools url
keyword tools warez
keyword tools webmaster
keyword tools websites
keyword tools wiki
keyword tools word
keyword tools wordpress
keyword tools wordtracker
keyword tools wordze
keyword tools work
keyword tools worth
keyword tools yahoo
keyword tools yahoo panama
keyword tools youtube
keyword tool target geographic location
keyword tool that shortens urls
keyword tool tips
keyword tool traffic estimator
keyword tool uk
keyword tool volume
keyword tool word document
keyword tool yahoo
keyword tool yuri arcurs
lsi keyword tools
paid keyword tools
seo tools keyword density
seo tools keyword list generator

I dare you to find a better keyword research tool or method than this. I doubt you will, but if you do, I’d love to know about it and be proven wrong!

Anyway, I hope you enjoyed the post. Please feel free to “steal” my code, use it, modify it, etc.

As always, happy automation!

I just came across a service called Ubersuggest.org. If you’re familiar with Google’s keyword suggestion tools then you’re already familiar with what the Ubersuggest service is after. Basically their service automatically scrapes the Google Suggest service using any seed keyword or phrase of your choosing. In addition to simply running one query for the keyword you provide, it enumerates through alphabet and numbers 1-9 to gather all results with that letter or number as the start of the next word in the phrase.

It’s an interesting service. Such an interesting service that before I had run across it, I wrote a tool of my own to combine the suggest services of Google, Yahoo, Bing and Amazon to do the same automated keyword suggest scraping and enumeration. It’s written in Bash and I’ve posted the full source code on my blog. The cool thing is, you can see that it isn’t all that complicated to write a tremendously helpful tool like that.

Keyword Suggest Sources

Like I mentioned above, my tool uses Google, Yahoo, Bing and Amazon, which I thought was a good start. Ubersuggest doesn’t reveal their sources so I cannot be sure where their data comes from. One thing is for sure, my script is very comparable (if not more diverse).

I don’t plan to stop there either. I want to add more sources to my suggest scraping. Here’s my list for future versions.

1. Nextag.com
2. Shopping.com
3. Pricegrabber.com
4. TheFind.com
5. Blekko.com

If you have others in mind I should add to the list or add to the top of the list, please leave a comment! It may motivate me to do it this weekend!

- Update - Blekko.com Keyword Suggest Tool -

I just made a post about how to integrate Blekko’s search suggest functionality into a bash script. Check out the full post for more details.

Happy automation!

I’ve previously talked a bit about why I perform keyword research in the manner I do. I also shared several short Bash scripts that allow you to scrape search suggestion tools provided by Google, Yahoo, Bing and Amazon. It’s now time to show you how to take those tools and combine them into a thoroughly beastly keyword research tool.

The Starting Point

Here is the version of the script we ended with in the last post.

#!/bin/bash

q=$(echo "$1" | sed 's/ /%20/g')

clear
echo -e "\nGetting Suggestions for \"$1\""

echo -e "\nGoogle:"
curl -s "http://www.google.com/s?sugexp=pfwl&cp=15&q=$q" | sed 's/\[/\n\[/g' | cut -d'"' -f2 | tail -n +4

echo -e "\nAmazon:"
curl -s "http://t1-completion.amazon.com/search/complete?method=completion&q=$q&search-alias=aps&client=amazon-search-ui&mkt=1&x=updateISSCompletion&sc=1" | sed 's/,\[{".*//g;s/,/\n/g' | cut -d'"' -f2 | grep -v '\[\|\]\|\{\|\]' | tail -n +2

echo -e "\nYahoo:"
y_1=$(curl -s "http://sugg.us.search.yahoo.net/gossip-us-ura?droprotated=1&output=sd1&command=$q&nresults=10")
# this way uses our keyword as a suffix rather than prefix. usually there are some duplicates with the first method but those are removed later.
y_2=$(curl -s "http://sugg.us.search.yahoo.net/gossip-us-ura?droprotated=0&output=sd1&command=$q&nresults=10")
echo $y_1 $y_2 | sed 's/{/\n{/g' | grep '"k"' | cut -d'"' -f4 | sort -u

echo -e "\nBing:"
curl -s "http://api.bing.com/qsonhs.aspx?FORM=ASAPIW&mkt=en-US&type=cb&cb=sa_inst.apiCB&q=$q&cp=13&bq=$q" | sed 's/{/\n{/g' | grep '"Txt"' | cut -d'"' -f4
echo

If you run the script like this…

./script.sh "galaxy nexus"

then you’ll get the following results:

Getting Suggestions for "galaxy nexus"

Google:
galaxy nexus
galaxy nexus release date
galaxy nexus review
galaxy nexus verizon
galaxy nexus dock
galaxy nexus sprint
galaxy nexus accessories
galaxy nexus extended battery
galaxy nexus at\u0026t
galaxy nexus specs

Amazon:
galaxy nexus
galaxy nexus case
galaxy nexus dock
galaxy nexus extended battery
galaxy nexus battery
galaxy nexus screen protector
galaxy nexus accessories
galaxy nexus unlocked
galaxy nexus verizon
galaxy nexus car dock

Yahoo:
galaxy nexus
galaxy nexus 4g lte
galaxy nexus accessories
galaxy nexus prime
galaxy nexus release date
galaxy nexus review
galaxy nexus sprint
galaxy nexus update
galaxy nexus verizon
galaxy nexus vs iphone 4s
samsung galaxy nexus

Bing:
galaxy nexus

This is great, but there are many keyword opportunities that we are missing out on. Each of these sites return a maximum quantity of suggestions at one time. Each engine will choose the 10 “best” results. We can a different set of 10 results for each by just appending a space and then one letter to the end of our root keyphrase like you’re starting to type another word on to the end of your query. You can see why I mean by trying it yourself in Google.

Since Google will evaluate each keystroke you type and send a GET request back to their servers to return updated suggestions, we can use this to our advantage to greatly expand the list of results returned. Let’s rerun the script above, but this time we’ll append an a to the original keyphrase…

./script.sh "galaxy nexus a"

and you’ll get this:

Getting Suggestions for "galaxy nexus a"

Google:
galaxy nexus accessories
galaxy nexus at\u0026t
galaxy nexus amazon
galaxy nexus at\u0026t release date
galaxy nexus apps
galaxy nexus accessories verizon
galaxy nexus armband
galaxy nexus accessories dock
galaxy nexus adb driver
galaxy nexus amazonwireless

Amazon:
galaxy nexus accessories
galaxy nexus armband
galaxy nexus att
galaxy nexus android smartphone
galaxy nexus anti glare
galaxy nexus android case
galaxy nexus androidified
galaxy nexus assesories
galaxy nexus android
galaxy nexus anti glare screen protector

Yahoo:
galaxy nexus accessories
galaxy nexus ad mix up
galaxy nexus amazon
galaxy nexus and facebook
galaxy nexus android
galaxy nexus and verizon
galaxy nexus apps
galaxy nexus at
galaxy nexus at&t
galaxy nexus availability

Bing:

Bing is the only engine which doesn’t offer additional suggestions (maybe Microsoft doesn’t like the Galaxy Nexus?). Anyway, you get the idea. You could just iterate through the alphabet by rerunning the script for each letter. But, why would we want to do that? We want to automate everything!

This is the perfect opportunity to use a for loop as provided by Bash. There are many ways to use them, so I recommend running through the examples provided here if you want to get a better understanding of how they work.

Keyword Suggest Script w/ Iteration

#!/bin/bash

q=$(echo "$1" | sed 's/ /%20/g')
tmp=tmp.txt

echo "" > $tmp
for suffix in {a..z}
do
    curl -s "http://www.google.com/s?sugexp=pfwl&cp=15&q=$q%20$suffix" | sed 's/\[/\n\[/g' | cut -d'"' -f2 | tail -n +4 >> $tmp
    curl -s "http://t1-completion.amazon.com/search/complete?method=completion&q=$q%20$suffix&search-alias=aps&client=amazon-search-ui&mkt=1&x=updateISSCompletion&sc=1" | sed 's/,\[{".*//g;s/,/\n/g' | cut -d'"' -f2 | grep -v '\[\|\]\|\{\|\]' | tail -n +3 >> $tmp
    curl -s "http://sugg.us.search.yahoo.net/gossip-us-ura?droprotated=1&output=sd1&command=$q%20$suffix&nresults=10" | sed 's/{/\n{/g' | grep '"k"' | cut -d'"' -f4 >> $tmp
    curl -s "http://api.bing.com/qsonhs.aspx?FORM=ASAPIW&mkt=en-US&type=cb&cb=sa_inst.apiCB&q=$q%20$suffix&cp=13&bq=$q" | sed 's/{/\n{/g' | grep '"Txt"' | cut -d'"' -f4 >> $tmp
done

sed '/^$/d' $tmp | sort | uniq
rm $tmp

This script will run the previous commands one time per letter of the alphabet for each site per keyword. That means we’ve gone from running the previous script 104 times, to running this script once. It also does the tedious work of removing duplicate suggestions and then sorts the results alphabetically. By using "galaxy nexus" as my root phrase again, I get a total of 551 unique keyword suggestions.

I urge you to try this yourself, but don’t abuse it or Google will just end up giving you the dreaded CAPTIA page. Of course I recommend reviewing the results and filtering the lists down further.

I bet this method of keyword research will replace your previous method very quickly. If not, prove it by sharing your method in the comments below or by contacting me through Google+.

Happy automation!