Automate Everything w/ Bash, Linux & Command Line

Pinterest Pin It Button/Bookmarklet Hybrid

Thu, 21 Feb 2013 22:08:00 -0500

AKA - Pinterest “Pin It” Button Implementation Revisit

I wrote a post about a year ago about how to install the Pinterest Pin It button. Why would I have to do that? Well, because the code generated in the developer docs was incomplete, confusing and led to many many broken implementations on sites. I know that’s true because of how popular that post was.

Read the old post here.

Pin It - A Year Later

Well, I’m writing this post because many of the same issues still exist. They have done a better job of making sure that people understand how to install the code, but they’ve still allowed it to be easy for people to install it incorrectly on their sites. The biggest problem is when the media parameter isn’t tagged properly in the button href.

The original version would just fail if an image wasn’t specified. The new version allows you to pin the page, with a broken image. How much sense does that make?

The Pin It Bookmarklet Works…

What makes all this worse is that the Pinterest bookmarklet that they’ve had since the very beginning has more functionality in it than the button does. Ever notice that? It’s really mind-boggling.

Why wouldn’t they just mimic the functionality of the bookmarklet with the Pin It button? When you use the bookmarklet it gives you a gallery of images for you to choose from to use in your pin. That makes the bookmarklet usable on many different types of pages with no configuration required. Whereas the button has to have one image hard coded to it. This makes zero sense.

So I thought… Why not just rewrite the button functionality so that it calls the Javascript from the bookmarklet instead of the code the button would normally run? Would that work? YES IT DOES!

The Pin It Button - Pin It Bookmarklet Hybrid

The implementation is really simple. Just start out by adding the following code, as recommended by Pinterest, on the page where you want the button to appear. The one change is that I’ve added an ID to a surrounding <span> tag.

<span id="pin-it"><a href="//pinterest.com/pin/create/button/"><img src="//assets.pinterest.com/images/pidgets/pin_it_button.png" alt="image" /></a></span>

Then, just add the following script to your page. Make sure you have the jQuery library on the page as well.

<script type="text/javascript">
    jQuery.getScript('//assets.pinterest.com/js/pinit.js', function() {
        jQuery('#pin-it a').attr('href', '#');
        jQuery('#pin-it a').attr('onclick', "javascript:void((function(d){e=d.createElement('script');e.setAttribute('type','text/javascript');e.setAttribute('charset','UTF-8');e.setAttribute('src','//assets.pinterest.com/js/pinmarklet.js?r='+Math.random()*99999999);d.body.appendChild(e);}(document)));");
    });
</script>

That’s it. You should now get the bookmarklet action when the button is clicked, which will show a gallery of images on the page to choose from to use in your pin.

Here’s an example implemented right here:

Anyway… Happy pinning!

Big Data, Privacy & Ad Targeting

Fri, 14 Dec 2012 14:03:26 -0500

I think it’s important to talk about my perspective and potential bias. I’m a digital marketer. It’s what I do. But, sometimes I feel like Harvey Dent.

{Image Source}

I use cookie based tracking data every day for website optimization, ad targeting and business intelligence. There’s no way to deny how powerful an understanding of the information available, even with a vanilla install of Google Analytics, can be to your business. There are often very enlightening stories about what website users want, need or don’t understand that will help you improve your product. The structure, quantity, organization and analysis of this data is getting better at a fanatic pace. I feel like the big gap between how this data is used and how the general population speculates about it’s use is just too wide.

Or, Popular Are Privacy Concerns Right On Target?

I continually ask myself that question. As a consumer, I quit using Facebook. It wasn’t all about privacy but it had a role in my decision. But Facebook isn’t the only offender. Not by a long shot.

Recently there have been a few time where I’ve really had to stop, think and question my perspective on ethics surrounding online tracking technology. A couple of those prompted this post.

De-Anonymizing Tracking Data

What does that mean? Well, all the tracking data provided in common analytics packages like Google Analytics is anonymous until the visitor fills out a form and tells you who they are. This level of privacy is by design.

There are companies springing up that seek to change that. They pool data together from sites across their network partners that track the behavior of a visitor. When that visitor identifies him/herself on site A, then all the partner sites will know who that anonymous visitor was without the visitor actually telling site B, C or D.

Here’s the post I read that explains this much more effectively than I can. It’s creepy and I think it’s unethical.

It isn’t just new emerging businesses or technologies that can be creepy/evil. In fact, the bigger data players like Google have the ability to make these connections now. But are they?

When Ads Are Too Personal

Until now, the news of my wife and I separating hasn’t been public. The decision was made about 1 month ago. My communication of that with friends, family and co-workers has been limited to word of mouth, GTalk and SMS.

Imagine my surprise when I saw this ad at the top of my GMail account last night…

If you cannot imagine - I was creeped the fuck out.

After the shock of it wore off I started to think about how it was possible for this ad to be so spot on. To be honest, I have no way of knowing. And because I don’t know I’m creeped out. I don’t like it. Instead of ads being relevant, useful and persuasive, this ad gave me chills (not the good kind).

So, how? One can only speculate. Perhaps the add was just a coincidence. Maybe GTalk conversations are monitored for content based targeting just like GMail messages are. There could be any number of reasons.

The point of this post is that it doesn’t matter how tracking and targeting works online. I believe people prefer ads that are relevant to their interests, but they need to understand how and why ads can be so accurate at times. They also need to have the tools necessary to opt-in or out of this targeting. Without the transparency people will be left to speculate and become more upset about how their privacy is being abused online.

IBM Model M Keyboard Collection - On Sale

Wed, 03 Oct 2012 21:49:00 -0400

It’s time. I’m going to part ways with my small collection of buckling spring IBM Model M keyboards (the original clicky keyboard).

Why? Well, because I have too many keyboards and I have an upcoming project and want to replenish my PayPal account. I’m going to list them on eBay but I thought I’d list them here just because. So, here they are…

IBM Model M, Part No 1391401 - Produced on April 8, 1988

Bid on this keyboard

This is considered a “white” label IBM Model M keyboard and is fully functional. It has very low yellowing and types very nicely. I’ve used it as my main keyboard for the past two plus years and it’s been very reliable. It has the curly style removable cable and is in great condition. The feet on the back are also in great condition, which is more rare because they’re known to be a weak point. I don’t use them so I haven’t added any wear during my time using it.

They buyer of this keyboard is going to get a very well taken care of keyboard that is very likely to out last the machine he/she will use it on.

Here are a couple of images:

An image of the label - part number and production date.

I will post back a link to the auction as soon as it’s up. If you’re interested in purchasing it outside of the auction then please comment below or reach out to me on Google+ and we’ll see if we can work out a deal.

IBM Model M, Part No 52G9685 - Produced on January 4, 1996

Bid on this keyboard

This keyboard is also technically a IBM Model M, but in the later years these keyboards were produced by Lexmark for IBM, as you’ll see in the images of the back label below. With that said, this keyboard works very well. There are three main differences between this keyboard and the older “white” label Model Ms:

The key caps are not removable. Most of the time this isn’t a big deal, in fact it makes the caps much more difficult to lose. It also makes them a lot harder to clean.
The cable isn’t detachable. This really only becomes a problem if the cable it’s self breaks.
While the keypress is still very clicky, it’s also a bit more soft. This is either appealing or not. It really just depends on your personal typing preference.

While this keyboard is a bit less rare than the other two I’m selling, it is bound to have fewer keystrokes and is almost a decade newer. It’s a very reliable keyboard and all the keys feel consistent type nicely. I’ve used this keyboard for the past year and a half at work. The one thing that’s wrong with it is that one of the feet is broken. It never bothered me because as I mentioned above, I don’t use the feet.

Here are a couple pictures:

Again, I’ll post a link to it once it’s up on eBay.

IBM Model M, Part No 1391401 - Produced on June 7, 1987

Bid on this keyboard

This is the final IBM Model M I have to offer. It’s the oldest one and share many of the same features of the first one posted above. It’s considered a “white” label and is almost as old as I am…

It has the curly detachable cable that’s in great shape. It has fully removable key caps for easy cleaning. It has strong buckling spring keys that are very clicky. It’s exterior is in great condition. The only blemish on this one is that the enter key on the number pad isn’t balanced properly because one of the support pieces is missing. It’s very likely that it can be repaired later, but even in it’s current state it can be used. Other than that, this keyboard is in great shape despite it being 25 years old!

I haven’t used this one much at all. For the past six months or so it has just been sitting in my closet. It’s time to go. This keyboard was meant to be used.

Here are a couple images:

Once again, I’ll post the link when it’s up on eBay. If you’d like to reach out and make an offer ahead of time please feel free to contact me on Google+. There’s a link in the left navigation of this site.

Please let me know if you have any questions about any of the keyboards listed here.

Magento: Automatically Apply a Coupon Code w/ jQuery

Mon, 17 Sep 2012 20:09:52 -0400

I started working on planning a promotion a couple of days ago that would require a discount to be applied automatically. The site runs on Magento, which has very sophisticated coupon code and discounting tools. But this time, none of the built in options quite suited my needs.

tl;dr; - I needed to apply a coupon code to a customer’s cart, before they actually had anything in their cart and without the visitor doing anything more than visiting a specific landing page. I was able to accomplish my goal with a little creativity, jQuery and determination. And, I don’t know PHP so this is a front end solution using jQuery and cookies. Read on for the code and how it works…

How to Setup a Discount in Magento

The first thing you’ll need to do is setup a coupon code with an associated discount of some type in Magento. It doesn’t really matter what type of coupon it is, you just need something to test with. I’m not going to explain how to do that here but for demonstration’s sake, I’ll assume that the coupon code you’ve chosen is testcouponcode and it gives the customer 20% off.

Don’t be confused. Even though we’re setting up a coupon code, we’re not going to require any action from the visitor. We’re just going to use the back-end coupon capabilities of Magento to do the discounting in the shopping cart.

Before we go any further, make sure you manually test the coupon code you setup by going into the cart and applying it the old fashioned way. Trust me that if you made a mistake, it’s much easier to discover and correct it at this stage.

A Few More Notes About Coupons in Magento

When the customer enters a coupon code and clicks the apply coupon button, the code makes a HTTP POST to /checkout/cart/couponPost/ with the following data - remove=0&coupon_code=testcouponcode.
A coupon code cannot be applied to the shopping cart if there are no items in the visitor’s cart.
In my case, if the customer doesn’t checkout during the same session then I don’t want the discount to apply.

With those three things in mind, let’s dive into how I’ve solved this problem.

“Apply” Discount on the Landing Page, well Kinda…

Remember the introduction - I want to apply a discount based on the landing page. However, in Magento, this is technically impossible. You cannot apply the coupon code to trigger the discount right away on the landing page. Since it’s the first page your visitor will have hit, the visitor will have zero items in their cart and that’s the problem.

No worries though, there’s a way around that. All you have to do is create a session cookie that can be referenced if the visitor does end up putting items into his/her cart. The trigger will be the same (the landing page), but applying the discount will be technically deferred.

To make the order of events easy to understand, here’s a list:

Visitor hits the special landing page.
Javascript on the page sets a special session cookie, signifying that a discount should be applied once an item has been added to the cart. This cookie will automatically be deleted when the session ends.
Once the visitor views his/her cart (and there are one or more items in it), Javascript on the shopping cart page will make a POST to apply the coupon code that was specified back on that special landing page. This POST will happen in the background and then refresh the page quickly.

Pretty simple right? Based on those three events, there will be two different blocks of code needed to make this work. One needs to be on the landing page and the other needs to be on the shopping cart view page.

jQuery for the Landing Page

Remember that the only goal for the code on this page is to set a session cookie that can be read and referenced later. I’m a newb when it comes to Javascript and for most things I use jQuery. Unfortunately there’s no built in way to write or read cookies in jQuery. The good news is that there’s a really great plugin for it and using the plugin is as easy as downloading it and including it in the HTML of your site. You can get it here, hosted on Google Code.

Here’s the code:

<script type="text/javascript" src="/path/to/js/jquery.cookies.2.2.0.min.js"></script>
<script type="text/javascript">
    jQuery.cookies.set('nameOfCookie', 'testcouponcode', '/');
</script>

It’s really simple. All you have to do is name the cookie, give it a value and set the path that the cookie should be valid for. Setting a path of / means that it can be read or modified anywhere on the domain. I recommend making the value of the cookie equal to the actual coupon code. It’s most simple that way and I don’t think there’s any reason to try to hide it.

Once you’ve downloaded the cookies plugin and included this code on your page, it’s a good idea to load the page and inspect your cookies to make sure the cookie was actually set. If it was, move on to the next step.

jQuery for the Shopping Cart View Page

This is the code you’ll need to put on the cart view page. The simplest way to implement this is to put it just before the closing <body> tag on the cart view page.

<script type="text/javascript">
    var emptyCart = jQuery('.cart-empty').text();
    if ( (emptyCart == "") ) {
        jQuery.getScript('/path/to/js/jquery.cookies.2.2.0.min.js', function() {
            var couponCode = jQuery.cookies.get('nameOfCookie');
            if ( (/../i.test(couponCode)) && (!!couponCode) ) {
                var postData = "remove=0&coupon_code=" + couponCode;
                jQuery.post("/checkout/cart/couponPost/", postData, function() {
                    jQuery.cookies.del('nameOfCookie');
                    document.location = '/checkout/cart/';
                });
            };
        });
    };
</script>

If you cannot add code to this page, you can place it in a file and include it in the head of the page. If you do that though, you’ll want to slightly modify it to execute after the page load is complete. You’ll also want to make sure that it only executes on the /checkout/cart/ page. It would look something like this…

window.onload = function() {
    var emptyCart = jQuery('.cart-empty').text();
    if ( (/\/checkout\/cart\//i.test(page)) && (emptyCart == "") ) {
        jQuery.getScript('/path/to/js/jquery.cookies.2.2.0.min.js', function() {
            var couponCode = jQuery.cookies.get('nameOfCookie');
            if ( (/../i.test(couponCode)) && (!!couponCode) ) {
                var postData = "remove=0&coupon_code=" + couponCode;
                jQuery.post("/checkout/cart/couponPost/", postData, function() {
                    jQuery.cookies.del('nameOfCookie');
                    document.location = '/checkout/cart/';
                });
            };
        });
    };
};

Here’s a quick explanation of what the code does:

If you remember, a coupon code cannot be applied to a cart if there aren’t any items in the cart. The first part of the script grabs the text from the headline with a class equal to .cart-empty and it’s stored in the variable emptyCart.
The following IF statement checks to make sure the emptyCart variable is empty. If it isn’t empty then that means the cart is empty and the rest of the code should not execute. If you used the second version, there’s also a condition that makes sure that the current document is the cart view page.
Next, we need to make sure that the jQuery cookies plugin has been loaded.
After the cookie plugin has loaded, the variable of couponCode gets set to the value of the coupon code from the landing page.
Then there’s another IF statement that checks to see if a coupon code exists.
If the coupon code does exist, then the next part makes a POST to /checkout/cart/couponPost/ with a query string equal to remove=0&coupon_code=testcouponcode which will actually activate the discount for the customer’s shopping cart.
If that POST is successful, the cookie gets deleted and the page refreshes. The refresh is needed to let the customer see the discount in the sub-total. It’s unfortunate that the page refresh is necessary. I just haven’t been able to figure out a way to get around it. If you know how, please feel free to comment.

I hope this has been helpful. Now, go forth and rockout some new landing pages!

Fixing Magento's Order Tracking in Customer Accounts

Mon, 20 Aug 2012 08:57:11 -0400

Fixing may be too strong of phrasing. There’s nothing “broken” about how order tracking works inside customer account in Magento. However, it also isn’t very usable for several reasons. Let me explain.

The Issues

Here are the issues I have with how Magento presents tracking information in customer accounts.

The link for the tracking number isn’t very well presented. If I placed an order and I’m logging into my account to check on it, I’m going to either be checking on the status or I’m trying to track the package. Presenting those two pieces of data should be top priority.
When you find the link to get your tracking number and click on it, you’ll see that another pop up window opens just to show you the shipping carrier, method and tracking number. It seems like a big waste of a click to me. Why not show that information on the order detail page? Or, if you want to have a link to it, why not link directly out to the shipper’s tracking page?

I set out to fix these to issues on sites that I manage.

Uh Oh, I Don’t Know PHP or Magento

It was very important to me to be able to fix this but I don’t know a thing about PHP or the core functionality of Magento. I wasn’t going to let that stop me though. I tried looking around in the interwebs to see if anyone else had already solved this problem and I didn’t find much. I also looked for extensions on Magento Connect, but no luck there either.

After reviewing the problem and thinking about the existing skills that I posses, I decided it didn’t matter. All the data I need to solve those problems was available to me on the client side. And, since that’s true, I knew I could get what I needed by using jQuery.

Using jQuery on Magento Sites

I’ve been learning jQuery for about 6 months or so. I learn a little more each time I have a project or excuse to use it. Since I’ve started to use Magento, there has been a few circumstances where I wanted to use jQuery. The problem is that by default jQuery conflicts with Prototype (another Javascript framework). It took me a while to figure out what was going on but in the end the solution was simple.

If Prototype.js is in use on a site, write your jQuery selectors like this jQuery('#someId').method() rather than $('#someId').method(). The reason is that both Prototype and jQuery use that same $ as a variable.

orderTracking.js

First thing’s first - here’s the script:

jQuery(document).ready(function () {
    if ( jQuery('h2.table-caption a').length > 0 ) {
        var onClickActionObject = jQuery('h2.table-caption a').attr("onclick") + "";
        var onClickAction = JSON.stringify(onClickActionObject);
        var URLstripped = onClickAction.replace(/\"function.*\/\/www/, "https://secure").replace(/','.*/, "");
        jQuery.ajax({
            url: URLstripped,
            type: 'GET',
            dataType: 'text',
            success: function ( code ) {
                var method = jQuery(code).find('.label').text();
                var trackingNumber = jQuery(code).find('.value').text();
                if (method.replace(/ .*/,"") == "USPS") {
                    var carrierURL = "https://tools.usps.com/go/TrackConfirmAction!execute.action?formattedLabel=";
                    var trackingURL = carrierURL + trackingNumber;
                };
                if (method.replace(/ .*/,"") == "UPS") {
                    var carrierURL = "http://wwwapps.ups.com/WebTracking/track?track=yes&trackNums=";
                    var trackingURL = carrierURL + trackingNumber;
                };
                jQuery('h2.table-caption span, h2.table-caption a').remove();
                jQuery('.col-2 .box .box-content:first').after('<a id="external-tracking-link" target="_blank" style="color:#973C3E; font-weight:bolder;" href="' + trackingURL + '">Track Your Shipment</a>');
            }
        });
    };
});

Save the script and then include it in the <head> of your Magento site.

Now, let me explain what’s going on step by step:

The first step is to make sure the script doesn’t do anything until the page content has loaded. The first line in the script performs that task. Here’s the snippet jQuery(document).ready(function () { ... });.
Since this script won’t serve any purpose except for when the customer is logged into their account, we don’t want the full script to execute on every page. There are multiple ways of doing this. One way would be to only include the script on the account pages but since I don’t know anything about the Magento’s core files I couldn’t go that route. I decided to include a simple if statement that would check to make sure that it doesn’t execute if not on the order details page. This is the test I used for my store if ( jQuery('h2.table-caption a').length > 0 ) { ... };.
The “Track your order” link in the customer’s account has an onclick attribute assigned to it. When the customer clicks it, the popup window presents the carrier, shipping method and tracking number. We’ll need that information so we need to record that URL into a variable. In the script, that URL gets assigned to the URLstripped variable.
Now, we need to get that tracking information by executing a AJAX request in the background and parse out the pieces we need. Lines 6-20 do the heavy lifting there. It’s important to note that we ship using either USPS or UPS. Therefore this script will only work for those two shipping carriers. You should be able to add in others pretty easily.
The last step is to take the tracking number and rewrite the tracking link, style it and put it in a different spot on the page. That’s handled by lines 21 and 22.

Now, when the tracking link is clicked on the carrier’s website will open in a new tab/window with the tracking information for the customer’s package already displayed.

Let me know if you have any questions or have ideas for accomplishing this task in a better way.

Thanks for reading.

Fixing Google Analytics Self Referrals in Magento Enterprise

Sat, 18 Aug 2012 09:16:04 -0400

If you have a website and it runs well, your next priority should be to make sure your analytics setup is sound. This was a challenge I just had to conquer myself and sometimes it isn’t as easy as it seems like it should be. Here’s my tale…

Setting up Google Analytics in Magento Enterprise

Configuring Google Analytics in Magento couldn’t be easier. There’s a config page, you put in your profile number, activate it and then you’re done. Except in my case we weren’t.

We have a bit of a unique setup. We have a sub-domain that we need to track as well and for us, there’s no need to have the data separated into a separate profile because the sub-domain is only used for the cart. We have the main http://www. sub-domain as well as the https://secure. sub-domain.

What is a self referral?

It’s pretty simple. When your visitor traffic is properly tracking source by source but you also have traffic and conversions tracking under the same domain name as what you’re trying to track. Because the only different sub-domain we have is in our cart, this meant that only our conversion data was wrong. Fortunately (or so I thought), Google provides special code to use when you need to track sub-domains.

Google’s Recommended Code for Sub-Domains

This is the code that Google recommends you use when tracking a site that has one or more sub-domains.

<script type="text/javascript">
    var _gaq = _gaq || [];
    _gaq.push(['_setAccount', 'UA-123456-7']);
    _gaq.push(['_setDomainName', 'domain.com']);
    _gaq.push(['_trackPageview']);

    (function() {
        var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
        ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
        var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
    })();
</script>

This is the newest asynchronous Google Analytics tracking code. The only part that’s different from the standard install is _gaq.push(['_setDomainName', 'domain.com']);. However, in Magento, there’s no way to define that variable without modifying core PHP files. But that’s exactly what we had to do to get that setup properly.

After this was installed and we verified that it was rendering properly on both sub-domains, we waited a day and reviewed our Google Analytics account. Still, this didn’t fix our problem. We were still tracking self referrals and there was no change at all.

At this point, I went back to the drawing board which pretty much means that I went back to Google searches.

I found a few other forum threads talking about another snippet that could be added that will help fix these issues. That’s when I decided that Google’s recommendation for code was going out the window.

The Fix

This is what worked for me. If you’re facing similar problems with self referrals, I hope it works for you too. However, the only real way to get things working is trial and error. Trial and error never fails forever.

The code:

<script type="text/javascript">
    var _gaq = _gaq || [];
    _gaq.push(['_setAccount', 'UA-123456-7']);
    _gaq.push(['_setDomainName', 'domain.com']);
    _gaq.push(['_trackPageview']);
    _gaq.push(['_addIgnoredRef', 'domain.com']);

    (function() {
        var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
        ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
        var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
    })();
</script>

The addition is _gaq.push(['_addIgnoredRef', 'domain.com']);.

Honestly, that’s all it took. It all seems simple now but at the time it was incredibly frustrating. As soon as the code went live and I inspected the GET requests for __utm.gif I knew that it was working properly and the problem had been solved.

Verifying Referral Data in Firebug/Chrome

In all reality, if you know what to look for you don’t have to wait a day to check Google Analytics to see if your data was tracked properly. All you really have to do is inspect the GET requests that the Google Analytics Javascript makes at page load. Here’s a screenshot of what you need to watch out for (screenshot of this blog)…

Click on the GET request for __utm.gif and then you’ll get the detail for that request. Pay special attention to the section under “Query String Parameters”, specifically the parameter called utmcc. That parameter is what recalls the tracking data stored in cookies. Here’s the value from that parameter:

_utma=189990958.53866207.1339112813.1345157959.1345242492.53;+__utmz=189990958.1345242492.53.25.utmcsr=assets.tumblr.com|utmccn=(referral)|utmcmd=referral|utmcct=/iframe.html;

Within that value, check out the parameter utmcsr, which is equal to the referral source. When you’re evaluating this on your own domain it’s important to look at this on the first page you visit when transitioning from one sub-domain to the other. That’s where the cookie information will get all screwed up. If the referrer is equal to your domain, then you still have a problem. If it is equal to the original referrer like direct, organic, etc, then you’ve solved the problem.

If I’m being honest I still do not understand why the Google Analytics code that’s recommended by Google in my account wasn’t enough. It’s really frustrating. If anyone has additional insight I’d love to understand.

I hope this has been informative. Please leave a comment if you have a question and I’ll do my best to answer or provide ideas.

NexTag.com Keyword Suggest Scraper

Thu, 16 Aug 2012 13:59:52 -0400

In writing the short script for this post I learned that NexTag.com is the holy grail of keyword suggest results. Most sites limit there suggestions to about 10, where as NexTag.com returns 100! That’s truly amazing and great for us.

Let’s cut to the chase. Here’s the script:

#/bin/bash

# This script will scrape NexTag.com's keyword suggestion tools.

query=$1
f_query=$(echo "$query" | sed 's/ /%20/g')
curl -s "http://www.nextag.com/buyer/opensearch.jsp?suggest=complete&perpage=100&search=$f_query" | sed 's/\[//g;s/\]//g;s/","/\n/g;s/"//g'
echo

You run it like this:

bash nextag.com.sh "shoes"

Then enjoy the results!

If you have any questions or feedback please leave a comment. Otherwise, happy automation.

Shopping.com Keyword Suggest Scraper

Wed, 15 Aug 2012 20:45:00 -0400

This is a really quick post about another keyword tool. As the title suggests, this is a tool to scrape keyword suggest on Shopping.com. The script is very simple and is just an adaptation to the tools discussed here, here, here and here.

This is the full script…

#/bin/bash

# This script will scrape Shopping.com's keyword suggestion tools.

query=$1
f_query=$(echo "$query" | sed 's/ /%20/g')
curl -s "http://www.shopping.com/ajaxSearchAssistant?q=$f_query" | grep "<S>" | cut -d'>' -f2 | cut -d'<' -f1
echo

Then you run it like this…

bash shopping.com.sh "keyword"

Using niche sites like Shopping.com (or any other shopping engine) is a good idea when doing keyword research specifically for campaigns trying to sell very common consumer products. Shopping engines like Shopping.com can be a very helpful resource.

Happy automation!

Das Keyboard vs. Razer Blackwidow vs. IBM Model M

Tue, 14 Aug 2012 18:00:52 -0400

A colleague introduced me to mechanical keyboards about three years ago. I’m certainly not a grey beard and my first computer was a laptop with Windows XP installed. Needless to say, I didn’t have any experience with mechanical keywords and didn’t have any “good ol days” nostalgic feelings about how keyboards used to be made with quality materials for touch typists. So, when this guy started blabbing about how awesome mechanical keyboards are, it was easy for me to tune him out.

I thought I loved the feeling of typing on a low-profile laptop style keyboard until, on one day, this guy brought in his humongous Silver Label IBM Model M keyboard and let me give it a try. Again, upon first glance I turned up my nose. However, after a couple hours of use I started to understand what all the hype was about. I really enjoyed typing on that keyboard. I felt like I could type faster and I felt my pinky finger reaching for the backspace key less often. It’s worth mentioning that my office mates didn’t appreciate the extra noise very much (I didn’t really care).

After that, it was my mission to get a mechanical keyboard for myself. I quickly discovered that old Model M keyboards aren’t very easy to source on-demand. Long story short, I was able to source two Model M keyboards locally and restore them to perfect working condition. I’ve been using one at work and one at home for the past couple years. They’ve treated me well and have me hooked on mechanical key switches.

My Newest Mechanical Keyboards

The only thing pushing me to shelve my IBM keyboards is the fact that PS/2 connectors aren’t installed on new machines anymore. You can buy a converter to go from PS/2 to USB. I’ve been using one of those to connect to a laptop for the past year and a half, but sometimes it drops keystrokes which is not acceptable. For that reason, I decided to move on and purchase a couple of modern day mechanical keyboards.

Razer Blackwidow Ultimate

The first modern mechanical keyboard I purchased was the Razor Blackwidow Ultimate. I’ve read good things about this keyboard and was attracted to the dedicated macro keys. But, to be honest, the final motivating factor for pulling the purchasing trigger was the fact that I had a Best Buy gift card and needed to spend it. I purchased it for home use with Ubuntu Linux.

Quick note: The dedicated macro keys on that keyboard aren’t recognized by the kernel by default. Some guy wrote a Python script that enables them. You can get the script here. After you download it, run it like this (root permissions are required):

sudo python blackwidow_enable.py

You should now be able to program the macro keys.

After overcoming that little hurdle, this keyboard has been great. It has a shorter keystroke than the old Model M but is has a very similar audible click and smooth tactile click feel. It’s been great.

Das Keyboard Professional S

The second modern mechanical keyboard I purchased was the Das Keyboard. I purchased this one for use at work. This one is much less “gamer” than the Razer, but it truly delivers with performance. It doesn’t have the dedicated macro keys that the Razer does but it has the Cherry MX Blue key switches and that’s what really matters.

The feel is amazing. And, let me say this - the Das has better feel than the Razer Blackwidow and the IBM Model M. I’ve also used Filco keyboards in the past (never owned one) and I would argue that the Das even types nicer than the Filcos. The difference is very subtle and therefore I cannot even describe why it is. Typing on the Das keyboard is just a unique experience that you won’t understand until you do it.

So, do it. Get yourself a modern mechanical keyboard today. If you type all day long you won’t regret investing in the perfect keyboard.

Strange URLs Appearing in Google After Redesign

Fri, 10 Aug 2012 20:31:05 -0400

A couple of weeks ago I wrote a post showing some efficient ways to verify your server side redirects. In that post, I describe several ways to get lists of all the different canonical URLs that Google and other search engines may be aware of so that you can setup redirects ahead of the redesigned site launch.

I wrote that post because I was managing two such relaunches at that time. Last week on Tuesday, those two sites went live. Since then I’ve been monitoring Google Position reports daily to see how my positions and pages have been changing. Since we made the decision to change several of our URLs, it’s also important to make sure the changes are being understood properly. Here are some of the thing’s I’ve been paying attention to:

Individual keyword position swings, like changes of 3 spots or more.
Have the 301 redirects from old URLs to the new URLs been understood? Meaning, when the old URL isn’t positioning anymore, has the new equivalent replaced it?
Are category pages still positioning for my head keywords? Are product pages or content pages still positioning for the longer-tail keywords?
Are all the new URLs being indexed?

Everything is looking great. Our positions are holding through the transition and each day I see a few more positioning URLs transition over from the old URLs to the new ones.

Then, the Google Surprise

Then I started to see something strange. I started to see URLs appear to be positioning that I’ve never seen before. The URL structure looked familiar enough, but the actual URL hasn’t ever shown up in Google’s index before (that I recorded). In each of these instances, the positioning URL was either the second URL for our domain positioning for that keyword.

To recap, the URL that was positioning for the old site has now been replaced with an old canonical version that doesn’t exist on the new site and never positioned when the old site was live. This is really strange behavior, but I’m not really surprised because our old site had ridiculous amounts of canonical URLs. It would be sickening for any knowledgeable SEO.

I found a handful of occurrences like this that fit the same pattern. To double check that I wasn’t mistaken, I double checked the server responses for those pages and found that they are indeed returning 404 (as they should). Again, this doesn’t really bother me because overall SEO performance has been great and each of these instances are just secondary positions. It’s just strange and I haven’t ever seen it before.

Next Steps

I honestly think that things will sort themselves out within a week or two. In the meantime, I’m going to continue with my daily position reports. When I find strange situations like this where new URLs show up, I’ll just create 301 redirects for them to the most relevant page on the new site. It shouldn’t take too long for Google to figure it out.

Fun with TextBelt - Public SMS API

Wed, 08 Aug 2012 21:01:22 -0400

The first time I heard about TextBelt was months and months ago. I tried it out right away and it worked very well. At the time I didn’t really have an excuse to build a tool around it. In fact, I still don’t but I had a few ideas and just happen to think this kind of thing is fun.

So, let’s first take a look at how to use it from the Linux command line.

How to use TextBelt

The text belt has a really clear tutorial that should get you started. All you really need to know is the following command:

curl "http://textbelt.com/text" -d number=1234567890 -d "message=this is where you put the body of the sms"

That’s it. Once you successfully send your text message you should get back the following response in your terminal:

{"success":true}

If you get that message back you should receive the sms very shortly. I did determine that sending these to Google Voice accounts does not work. I’m not sure why and I didn’t really look into it. My guess is that Google Voice silently drops the request because TextBelt reports that the message was successfully sent. You can also get error messages, though I’ve only received one and it is as follows:

{"success":false,"message":"Exceeded quota for this phone number."}

TextBelt does have some pretty tight quotas. You cannot send more than three messages to any one phone number in a three minute period. You are also limited to 75 outgoing messages from any IP address per day. It does say on their site that you can request increased limits if you have a real need for them.

Interesting Ways to use TextBelt

Now that you know how to use TextBelt, lets look at some of the interesting ways to use it.

Watch a log file for changes, SMS yourself the updates

You may or may not know about the tool called tail. Basically tail is a command line tool that writes the last set of lines to the command line. Here’s the most basic way to use it:

tail logfile.txt

When you run tail like this it will exit after printing the last ten lines. You can run it with the -f flag or run the tailf program and then it will stay open and print out each of lines added to the end of the log file as soon as they’re added.

tail -f logfile.txt

So, let’s combine our new-found skills. With some help of the program xargs and this Commandlinefu page, I was able to come up with this little gem:

tail -f file.txt |xargs -IX curl "http://textbelt.com/text" -d number=1234567890 -d "message=X"

Instead of printing the newest items out to the command line, they’ll get sent to your phone number. Pretty cool right? Depending on what type of a file you’re monitoring, you may not want to get every one of the messages sent to your phone. It’s easy enough to filter the messages from the log update by using grep.

Here’s a real world example that would send you a SMS each time someone logs into a server using SSH:

tail -f /var/log/auth.log | grep "sshd.*Accepted" |xargs -IX curl "http://textbelt.com/text" -d number=1234567890 -d "message=X"

You could modify the grep expression to send the failed log on attempts as well.

Send Facebook’s falling stock price every 30 minutes

Maybe you have a similar disdain for Facebook as me and it brings you a small amount of joy each time you see something bad happen to Facebook. It’s easy enough to scrape Yahoo Finance using curl to send the current stock price for any stock through TextBelt. This is the quick example using ‘FB’:

 curl -s 'http://download.finance.yahoo.com/d/quotes.csv?s=fb&f=l1'|xargs -IX curl "http://textbelt.com/text" -d number=1234567890 -d "message=X"

If you don’t enjoy watching Facebook’s stock fall then you can change it to what ever stock you want by replacing the fb in s=fb with the stock of your choosing.

That’ll just send it once and exit. If you want it to be sent every 30 minutes then you have two options. You can set up the command as a cron job or you can execute the command inside a while true loop and ask it to sleep for 1800 seconds before re-running the command. Here’s an example of the while loop:

while true; do curl -s 'http://download.finance.yahoo.com/d/quotes.csv?s=fb&f=l1'|xargs -IX curl "http://textbelt.com/text" -d number=1234567890 -d "message=X"; sleep 1800; done;

That’s all I have for now. I’d love to see some additional uses in the comments. Happy automation!

Facebook, the Coolest Click Fraudster

Mon, 30 Jul 2012 21:23:02 -0400

Yup, but I’m not the first or the only one to say it. It’s very likely that Facebook is participating in planned mass click fraud. This topic is only top of mind because of the post I read a few minutes ago about how 80% of clicks on Facebook ads are bots. Since that was a Facebook post and the author claims to be deleting the page/account soon, I decided to add the text here as well for archiving purposes. This isn’t something I’ve worried about in years primarily because I swore off Facebook ads (except in very rare cases) back when Facebook ads were still in beta.

I wrote about a very similar experience I had just a few months ago with click fraud at Shopzilla. I hate click fraud. If you read my post about Shopzilla, you’ll also get some of my memories about dealing with Yahoo when they used to get a ton of invalid click activity (that’s the PC way of referring to click fraud) from publishers on their network. The good thing about Yahoo was that they used to give refunds if you could prove that the activity was “invalid”, that is if you even payed enough attention to your analytics to notice. Yahoo wouldn’t call it a refund though, that would be an admission of guilt. They’d be very clear when referring to the monies paid as a “courtesy credit” even though, I’ve actually gotten them to send a check as opposed to crediting back to the account balance.

At least Yahoo had the courtesy to refund fraudulent click spend. Facebook & Shopzilla, well, it seems as though they’re in denial mode.

A Different Perspective…

I think that most of the people who write about this in the coming days will be bashing Facebook over this. They probably deserve it. Also, I bet there are a few who will take Facebook’s side and defend them. However, there’s a third perspective that I’d like to share. How digital marketing agencies fit into the mix.

Most large online, digital marketing agencies make their revenue based on a percent of the client’s advertising spend. The more their client spends on advertising, the more the agency brings in for fees. This is logically defensible because if the agency wants to convince their client into spending more money on advertising then the agency must also have to return more sales for that client. You know, provide them a good return on investment.

Here’s the thing about ROI… a good return on investment is different for every business. One of the accounts I manage every day would be doing poorly if the Google Paid Search account returned less than 1000% ROI averaged over a week. However, I’ve managed plenty of accounts over the years where 200% ROI was a frigg’n miracle. The agency trick is convincing the client that 500% ROI is some how better long term, even if 1000% is possible now. Who would deny that a return of $5 dollars for every $1 dollar spent on advertising isn’t an amazing return? If I didn’t know better, I wouldn’t.

If you’re currently working with an agency to takes their fee based on percent of ad spend, keep your ears open for terms/phrases like this:

“Cast a wide net”
“Branding”
“Long conversion cycle”
“Building awareness”
“prospecting keywords”, usually referring to broad match keywords in AdWords.

I’m not saying all those are bullshit, I’m just saying that you need to be able to definitively track and prove results. You owe it to yourself as the client to hold the agency to a high standard/burden of proof. If you don’t care, they won’t either. In my experience, more often than not, those strategies are birthed by pressure to grow client budgets.

Agencies Care More About Their Own ROI

Advertising account managers at agencies are pressured by those above them to grow budgets to increase ROI for the agency. Imaging that. Yeah, I said it.

As long as the client is happy, grow spend, work hard to justify it (consulting), twist the graphs and grow those budgets. What happens if client’s become unhappy, convince them they’re wrong. How do I know this? I’ve been there. I was the guy who cared about my clients and took shit from my superiors for it. I don’t believe that the agency I came from was unique. I took over many accounts from bigger, more well known agencies that were being blatantly negligent with account management. And why, because it benefits them. As a matter of fact, I was just talking with a friend who’s also an ex-agency guy/girl about me writing this post and this is a quote (I cut out a couple of name…):

“I wasted thousands of dollars “testing” FB ads for clients. They don’t work after three months? “Keep testing and trying new ad copy.” Roger that <redacted>”

“FB ads for the most part were the biggest goddamn waste of time and money ever. <redacted> seemed to like them. I think because they burned through available budget.”

I left the agency world (in part) for that reason. I went to work in-house for a company who needed a full time online marketing guy. It was the best move of my life. I’m now truly incentivized to optimize advertising accounts for ROI. I’m rewarded when I kick more ass. I like that arrangement.

Why Throw Agencies Under the Bus?

I think they deserve it because they’re trusted advisers and they should know better. In many cases, it’s like taking candy from a baby. Agencies are the ones who know about click fraud. They’re the people who’re the experts. They see the large data needed to pick out the oddities in analytics that just don’t seem right. Agencies should be advocates against shit bombs like Facebook, Shopzilla and Yahoo (like I was and still am). But, more often, agencies are too busy writing blog posts and whitepapers about how social media is the next big way target prospective buyers. Agencies citing one another’s published “research” about how social media “advertising” will “grow your brand”. These agencies have one interest in common - getting more of your ad budget.

If you’re a smart brand, small business or startup then you should tell agencies to DIAF. Hire a consultant who trains you or someone in your company who has a vested interest in your success. Hire someone who has real experience. Don’t be pushed around by buzzwords and skewed market research.

Take control of your advertising spend and make it return for you.

Verifying 301 Redirects During Site Redesign

Tue, 24 Jul 2012 19:25:00 -0400

I’m currently working on finishing up redesigns on two e-commerce sites at my day job. This is the culmination of many months of work and significant resources. We’re within days of launch. I’m pretty excited, but I decided that I needed to do another review of my 301 redirects to make sure I didn’t miss anything.

This analysis showed me that I had.

When designing and architecting new sites, it’s really easy to only think about how awesome things will be with the new platform/design/whatever. However, it’s projects like organizing 301 redirects for old URLs to new URLs that require you not to forget how bad things once were. For me, this analysis forced me to recall how many problems these sites have had over the years with canonical URLs.

We were able to overcome our internal canonicalization issues by using the rel=canonical tag and fixing internal linking. However, there were years before where several different URLs were used for any one page. Some of those alternate URLs are still haunting us today.

My challenge is to dig up all those old URL versions and make sure we have 301 redirects in place so that when we launch our new sites, visitors and search engines see our 404 page as little as possible. We just need to get a list of all those URLs to get started with.

Sources of URLs

Here’s a list of several sources I used to get together a list of URLs to check. I wanted to make sure I used a variety of sources with the thinking that a wider net is better.

HTML Sitemap
XML Sitemap
PPC Account Landing Pages
Google Webmaster Tools
Blekko In-Bound Link Report (see my bookmarklet for easy data extraction)
Server Logs
Google Analytics

It isn’t important to me to keep URLs separated by source. Once I had extracted URLs from one source, I just dumped them into an Excel worksheet. Later on you can use Excel to remove duplicates.

Checking Server Responses on URLs

Once you have your list of pages you’ll need an efficient (automated) way to check the server response for each. The server response code, i.e. 200, 301, 302, 404, 500, etc. is a part of the HTTP header that sent by the server to the web browser for each page request. I decided to write a short script to extract just the HTTP header.

Here’s the Bash one-liner:

wget --spider -S "$url_to_check" 2>&1;

Now just replace $url_to_check with the actual URL, like http://www.google.com. Here’s an example:

wget --spider -S "http://www.google.com" 2>&1;

This is what you’ll get back (Google sends a rather large header):

Spider mode enabled. Check if remote file exists.
--2012-07-24 16:11:15--  http://www.google.com/
Resolving www.google.com (www.google.com)... 74.125.225.176, 74.125.225.180, 74.125.225.178, ...
Connecting to www.google.com (www.google.com)|74.125.225.176|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Set-Cookie: NID=62=GZrQmPOvK5AyPhgA1RYRKP3KCxVdFL_QZ_GptmYGQrOI2d9nUqQETovH7MhtWroeeFOL_xKGt1w-YffuGhmP5IjF38IcR6IbNlTVBLLU_t35rQwaVZFW7H7jKGVqRIr3; expires=Wed, 23-Jan-2013 20:12:05 GMT; path=/; domain=.google.com; HttpOnly
  Date: Tue, 24 Jul 2012 20:12:05 GMT
  Expires: -1
  Cache-Control: private, max-age=0
  Content-Type: text/html; charset=ISO-8859-1
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com
  Set-Cookie: PREF=ID=d72f33e49b899251:FF=0:TM=1343160725:LM=1343160725:S=B8YfflyvIQvhgjE8; expires=Thu, 24-Jul-2014 20:12:05 GMT; path=/; domain=.google.com
  Set-Cookie: NID=62=d-OSrg2MYi6_7kbY5lHpW3qQ5ASiMMblUeUfBqppHahxDRXQL4qqPuI1nNxbgV3MoQnwxOuD6mPSTRrCK4xZk_ApFTi0KSsyDibrQ6-KHRXKZlPpECBr6AW39QNfhC9G; expires=Wed, 23-Jan-2013 20:12:05 GMT; path=/; domain=.google.com; HttpOnly
  P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657www.google.com (www.google.com)... 74.125.225.176, 74.125.225.180, 74.125.225.178, ...
Connecting to www.google.com (www.google.com)|74.125.225.176|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Set-Cookie: NID=62=GZrQmPOvK5AyPhgA1RYRKP3KCxVdFL_QZ_GptmYGQrOI2d9nUqQETovH7MhtWroeeFOL_xKGt1w-YffuGhmP5IjF38IcR6IbNlTVBLLU_t35rQwaVZFW7H7jKGVqRIr3; expires=Wed, 23-Jan-2013 20:12:05 GMT; path=/; domain=.google.com; HttpOnly
  Date: Tue, 24 Jul 2012 20:12:05 GMT
  Expires: -1
  Cache-Control: private, max-age=0
  Content-Type: text/html; charset=ISO-8859-1
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com
  Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com
  Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com
  Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com
  Set-Cookie: PREF=ID=d72f33e49b899251:FF=0:TM=1343160725:LM=1343160725:S=B8YfflyvIQvhgjE8; expires=Thu, 24-Jul-2014 20:12:05 GMT; path=/; domain=.google.com
  Set-Cookie: NID=62=d-OSrg2MYi6_7kbY5lHpW3qQ5ASiMMblUeUfBqppHahxDRXQL4qqPuI1nNxbgV3MoQnwxOuD6mPSTRrCK4xZk_ApFTi0KSsyDibrQ6-KHRXKZlPpECBr6AW39QNfhC9G; expires=Wed, 23-Jan-2013 20:12:05 GMT; path=/; domain=.google.com; HttpOnly
  P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
  Server: gws
  X-XSS-Protection: 1; mode=block
  X-Frame-Options: SAMEORIGIN
  Transfer-Encoding: chunked
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

All you should really care about is the line that says “HTTP/1.1 200 OK”. That is the server response that we want to verify. To extract just the server response from the output, let’s pipe the HTTP header out to grep to filter it.

Run this…

wget --spider -S "http://www.google.com" 2>&1 | grep "HTTP/"

…and you’ll get…

HTTP/1.1 200 OK

Perfect, except that it would be very tedious to do this one URL at a time. Let’s use a Bash for loop to finish this up. Since you should be doing all of this PRIOR to launching your new site, you’ll need to replace your productiondomain in the URLs with the stage.productiondomain for the development site. sed is perfect for that. Also, update the /path/to/url-file.txt to match the actual path to the file containing URLs. This assumes that there isn’t any other data in the text file except for the URLs you want to check.

 for URL in `cat "/path/to/url-file.txt" | sed 's/productiondomain.com/stage.productiondomain.com/g'`; do echo "$URL" - `wget --spider -S "$URL" 2>&1 | grep "HTTP/"` ; done

For this example, let’s take the following URLs as the example:

http://www.slipxsolutions.com
http://www.slipxsolutions.com/slip-X_NEXT.php
http://www.slipxsolutions.com/product/41/Safety_TreadsTub_Appliqu_and_233s/75/Bath__and__Shower_Appliques/
http://www.slipxsolutions.com/product/41/Safety_TreadsTub_Appliqu_and_233s/
http://www.slipxsolutions.com/product/41/Safety_TreadsTub_Appliqu_and_233s/134/75_Safety_Treads/
http://www.slipxsolutions.com/product/41/Safety_TreadsTub_Appliqu_and_233s/78/145_and_quot_Safety_Treads/
http://www.slipxsolutions.com/product/23/Drain_PlugsDrain_Products/46/Snug_Plug_Drain_Stopper_/
http://www.slipxsolutions.com/product/23/Drain_PlugsDrain_Products/48/StopAClog_Drain_Protector/
http://www.slipxsolutions.com/product/23/Drain_PlugsDrain_Products/126/Bottomless_Bath/
http://www.slipxsolutions.com/product/30/Bath__and__Home_Accessories/113/Shower_Splash_Guard/

Once I run the tool I get this as output:

http://www.slip-xsolutions.com - HTTP/1.1 200 OK
http://www.slip-xsolutions.com/slip-X_NEXT.php - HTTP/1.1 301 Moved Permanently HTTP/1.1 200 OK
http://www.slip-xsolutions.com/product/41/Safety_TreadsTub_Appliqu_and_233s/75/Bath__and__Shower_Appliques/ - HTTP/1.1 301 Moved Permanently HTTP/1.1 200 OK
http://www.slip-xsolutions.com/product/41/Safety_TreadsTub_Appliqu_and_233s/ - HTTP/1.1 301 Moved Permanently HTTP/1.1 200 OK
http://www.slip-xsolutions.com/product/41/Safety_TreadsTub_Appliqu_and_233s/134/75_Safety_Treads/ - HTTP/1.1 301 Moved Permanently HTTP/1.1 200 OK
http://www.slip-xsolutions.com/product/41/Safety_TreadsTub_Appliqu_and_233s/78/145_and_quot_Safety_Treads/ - HTTP/1.1 301 Moved Permanently HTTP/1.1 200 OK
http://www.slip-xsolutions.com/product/23/Drain_PlugsDrain_Products/46/Snug_Plug_Drain_Stopper_/ - HTTP/1.1 301 Moved Permanently HTTP/1.1 200 OK
http://www.slip-xsolutions.com/product/23/Drain_PlugsDrain_Products/48/StopAClog_Drain_Protector/ - HTTP/1.1 301 Moved Permanently HTTP/1.1 200 OK

Now all you have to do is watch and look for server response codes that don’t match your expectations. Notice that curl when using the --spider operator will check each page along the redirect path. If there are 6 redirects, it should follow every one and output the server responses.

Let me know if you have any questions. Happy redirecting.

WTF Is Security?

Tue, 17 Jul 2012 15:51:00 -0400

I’m not really sure how much this is related to automation, but I just have to get this out.

I feel kind of insulted right now. If the speed and force of my typing is any indication of my mood, then I’m sure the whole office knows I’m pissed (I’m typing this on a IBM Model M keyboard). Why am I pissed? Well, at my day job, I’m working with a new vender to help do some e-commerce integration with our accounting package and a set of new websites we’re developing. Again, the experience I’m about to share is pretty close to my first impression of them.

If I Give You Remote Access, Respect It!

This company was trying to troubleshoot a bug and couldn’t figure it out without getting access to our main server in the office. So, with them being hundreds of miles away, naturally they ask for remote access. I agree.

They then send me an email with instructions for allowing the remote connection. I’ll just paste what they sent me, minus the important bits, and let’s see if you can spot what’s so fucked up about what they’re doing…

Hi Adam,

One of our developers requires a remote connection to your machine to investigate the on-premise component of <redacted>.

Please find below instructions for making your computer available for remote connection via GoToMyPC.com:

eMail:              <redacted>
Password:        <redacted>
NickName:        **Company Name**
AccessCode:    <redacted>

If GoToMyPC is already installed on your computer, there should be a little green and white MYPC logo by the clock in the bottom right hand corner.  If it is there, please right click on the icon and choose ‘Register’.  You will be prompted for the information above.

If GoToMyPC is not already installed on your computer, please go to www.gotomypc.com and log in using the eMail address and password provided above.  When you log in, you will see a list of computer with a ‘Add Computer’ button at the bottom.  Please click on the button and allow the GoToMyPC software to install (should only take a minute)

[PLEASE NOTE: When installing GoToMyPC and registering the computer, this must be done on the machine you are physically in front of (not over a network).  This is a built in security feature.]

Once the software is installed, you will be asked to re-start.  This is not necessary for the product to function.

You will then be prompted for the information provided above.

Please let us know when you are available for us to connect.

Thank you and Kind Regards,

<redacted>

<redacted> Help Desk Administrator

Let me just say that I don’t have any real experience with GoToMyPc.com, fortunately for me, I didn’t really need it to be able to tell how ridiculous this is.

This company is providing the login credentials for their main account and asking me to log into it and add my server. Please remember that I’m their client. They don’t know me. I don’t work for them. And, they emailed these credentials.

I logged in. Guess what I see? I see a list of all their clients’ servers who’ve also followed these instructions. Next to each computer on the list is a button that would allow me to connect to them! WTF!. They essentially gave me access to every one of their clients’ servers where their backend accounting software is installed. What the fucking fuck? Are they brain dead? This is where all clients’ customers’ financial and PII data is stored.

I’m not an idiot, I didn’t connect to any of them.

Quit Emailing Credentials

I’m a good guy. I’m not going to do anything with this information besides complain. Yes, I’ll complain to them too. But there are plenty of people in the world who don’t have such good intentions. Seriously, we read about big companies getting breached every week.

Quit emailing passwords and other sensitive information. When you send an email it naturally gets copied several times over. And, more people than your intended recipient have access to the computers where that email is replicated. That’s the reason why secure protocols exist.

How Long Has This Been Going On?

There’s no way to know the answer to that question for sure, but I can make a few guesses based on how people choose passwords. I didn’t include the password above, but let’s just say that it ended in 2011, like the year. It’s currently July, 2012. So my guess is that they’ve been using GoToMyPc.com like this for a year or more and they haven’t changed the password.

It’s GoToMyPc, Not LetMeGoToYourPc

I’m fairly certain that the GoToMyPc.com service wasn’t designed to be used this way. There are client/support style remote access services out there that allow tech support personal to access client computers in the right way (certainly a more secure way).

Other Interesting Bits…

When I logged into their account (per their instructions) there were roughly 20 client computers available for connection, including the Help Desk Administrator that sent me the email! Apparently you can access their clients’ computers as well as several computers within their company network! This isn’t a small company either, which makes this even more scary.

It may be in their best interest to give a shit after all if people can get into their network (probably not though)!

Anyway, I’m done ranting. Even though it probably won’t change their behavior, I’ll let them know that I don’t think this is a good idea. That’s the least I can do.

Updating Google Analytics Code on Many Static Pages

Mon, 02 Jul 2012 09:57:16 -0400

I’m a relative newb. I have been fortunate enough that when I’ve had to work on sites there’s almost always been some sort of templating system in place. If I needed to make a site-wide code update, I could just make the change in the some templated code block and that would push the update throughout the site. It makes sense, and thank bejesus it’s now the norm.

What about when the whole site is static HTML files? If the site is tens of files, no problem. But what do you do when the site is several hundred? Before today, I had a few theories but had never had to test them out. Here’s my story.

Please Read, Very Important

These instructions will not work in every case, they aren’t designed to. This is a hack I put together that was designed to work with my very specific use case. I’m sharing it because I think you could update my code based on your needs. Please be careful and always make back ups. Enjoy!

The Goal

I was asked by a couple of friends to review their site and try to help them figure out why Google Analytics was tracking things strangely. Here were a few symptoms:

Event tracking was registering in the GA account even though everything looked correct in the GET request for utm.gif.
Keywords and query data wasn’t coming in for any search visits. No for Paid Search and not for Organic Search.
No social data.

I checked on a few things and saw that they had an old version of GA installed. This is the code.

<script type="text/javascript"> 
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>
<script type="text/javascript"> 
var pageTracker = _gat._getTracker("UA-123456-12");
pageTracker._trackPageview();
</script>

I didn’t really see anything wrong with it, but I decided to just try the new code on one page and see if it fixed the problem. I got the following code from their GA profile:

<script type="text/javascript">

  var _gaq = _gaq || [];
  _gaq.push(['_setAccount', 'UA-1235526-26']);
  _gaq.push(['_setDomainName', 'rfsystemlab.us']);
  _gaq.push(['_trackPageview']);

  (function() {
    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
  })();

</script>

I uploaded the updated page to their server and opened it in a web browser. By looking at the source, I could tell…

I was getting the new page with the updated tracking code.
The new code was working to track pageviews because I could see the GET requests.
After triggering a few events, I could see that the GET requests for those were sent correctly.

I waited a few minutes and checked their GA account again to see that now the Events were correctly showing up in their account! Awesome…

Except not.

The New Goal

Now that I think I know how to fix their tracking problems, I also have to figure out how to…

Remove all the old GA tracking code.
Add in the new code.
Avoid doing it one page at a time.

I could have just stopped at this point and said, “You do it”, but I wouldn’t do that. Besides, this is fun anyway.

My Tools

It wasn’t exactly easy to do this. Through a bunch of Googling, experimentation and a bit of experience with the command line in Linux, I was able to updates several hundred pages fairly easily.

I used the following standard command line Linux tools to complete this task:

sed
cut
for loops
echo

That’s honestly it… However, I did use many resources to help. Here’s my list:

Stackoverflow - Here, here and here.
DBASpot.com
LinuxQuestions.org

I found each of these by asking Google.

Step 1 - Find all the files containing the old Google Analytics code

First, before you do anything, make a backup and then a backup of that backup. Seriously.

I then opened a command line and moved into the root directory for the local copy of the site on my hard drive. Then I ran the following command:

grep -n -r "var gaJsHost" .

This command will search recursively for the string "var gaJsHost" and then output the file name that was matched along with the full text from the line where the match was found. The -n option tells grep to also print out the line number where the match was found. And, the string we’re searching for is one of the main variable names in the old GA code that we need to remove. (Don’t forget that dot at the end of the command. It’s very important!)

From there, we have our list of files we can operate on. By putting the previous command in a Bash for loop, we can manipulate the results and do perform additional operations one row at a time. Here’s the basic loop:

for match in `grep -n -r "var gaJsHost" .`; do

    # do some stuff here...

done

In a nutshell we’ll need to know the following pieces of information for each match:

file name.
row number where the match is found.
row number where the old GA code starts.
row number where the old GA code ends.

With that data, we should be able to delete the matching rows as an operation in this loop, for each file. This the next set of code in this process, combining what we’ve done above with the loop.

for match in `grep -n -r "var gaJsHost" .`; do
    file=`echo $match | cut -d":" -f1`
    line=`echo $match | cut -d":" -f2`
    start=$(($line-1))
    end=$(($start+7))
done

Each piece of information we need is stored in a separate variable. The file variable is cut from the original grep command output, which is equal to the file name where the match was found. The line variable was also from the grep command and is the line number in the file where the match was made.

We have two remaining variables that are getting values assigned, start and end, which represent the first line in the old GA code and the last line. I know that the string "var gaJsHost" is found in the second line of the old GA code and that the entire code block is seven lines long. So, the variable start can be calculated by subtracting one from the line where the match was found and the end variable can be calculated by adding seven to start!

Now, we just have to use this information to delete those lines. Here’s the command you’d add as the next command within the loop:

sed -i -e "`echo $start`,`echo $end`d" $file

Then your code would look like this…

for match in `grep -n -r "var gaJsHost" .`; do
    file=`echo $match | cut -d":" -f1`
    line=`echo $match | cut -d":" -f2`
    start=$(($line-1))
    end=$(($start+7))
    sed -i -e "`echo $start`,`echo $end`d" $file
done

After putting this into a Bash file and running it, I had successfully removed all the old Google Analytics tracking code from all the static HTML files on my local copy of my buddies’ site. We aren’t done. We still need to add all the new code to the pages.

Step 2 - Adding new code to the head tag

Honestly I’m a bit tired of writing the post at this point. Besides, most of the rest of what needs to happen is done by simple modification of the concepts above. So, here’s the rest of my code:

# set placeholder text before the </head> tag for the code to be replaced with later.

for match in `grep -n -r "</head>" .`; do
    file=`echo $match | cut -d":" -f1`
    line=`echo $match | cut -d":" -f2`
    sed -i "`echo $line`s/.*/NEW_GOOGLE_ANALYTICS_CODE_HERE\n<\/head>/" "$file"
done

# addin the new code..

$GACODE='<\!-- Start Google Analytics Installation --><script type="text\/javascript">var _gaq=_gaq\|\|\[\]\;_gaq.push\(\["_setAccount"\,"UA-123456-12"\]\)\;_gaq.push\(\["_setDomainName"\,"example.com"\]\)\;_gaq.push\(\["_trackPageview"\]\)\;\(function\(\)\{var a=document.createElement\("script"\)\;a.type="text\/javascript"\;a.async=true\;a.src=\("https\:"==document.location.protocol\?"https\:\/\/ssl"\:"http:\/\/www"\)\+".google-analytics.com\/ga.js"\;var b=document.getElementsByTagName\("script"\)\[0\]\;b.parentNode.insertBefore\(a\,b\)\}\)\(\)<\/script><\!-- End Google Analytics Installation -->
'
for match in `grep -n -r "NEW_GOOGLE_ANALYTICS_CODE_HERE" .`; do
    file=`echo $match | cut -d":" -f1`
    sed -i 's/NEW_GOOGLE_ANALYTICS_CODE_HERE/'"$GACODE"'/' "$file"
done

I hope this is helpful and not too confusing. I would love to answer any questions you may have. Leave a comment, contact me on Google+ or Linkedin.

Happy automation!

Dynamic DNS with Bash & Afraid.org

Tue, 19 Jun 2012 21:22:22 -0400

This has nothing to do with search engine marketing, but everything to do with automation. If you’ve ever wanted to host a server at a location that doesn’t have a static IP address then you know just how much of a pain it can be.

Hopefully by the time you read this you’re familiar with what DNS is and what dynamic DNS is. If not, those links should catch you up to speed quick enough. I’ve tried a few different dynamic DNS providers but for the past few years I’ve been using freedns.afraid.org. I don’t host this site (of course), but I do host FTP and a few other services that allow me to get to my machines at home where ever I am.

I’m writing this post today because I just happened to have to update my scripts that keep my DNS up to date when my IP changes today.

The Dynamic DNS Bash Script

This is the basic script I use to keep DNS updated for my domain.

#!/bin/bash

# paste in the info url from your account
get_info_url='http://freedns.afraid.org/api/?action=getdyndns&sha=<paste_your_password_hash_here>'

# get the current ip...
# method 1 - they aren't providing it for scripts anymore...
#ip=`curl -s http://whatismyip.org`

# method 2 - google provides this as an error message for bots...I'll take it.
ip=$(curl -s "https://www.google.com/search?q=what+is+my+ip+address" | grep "Client IP address:" | sed 's/.*Client IP address: //g;s/).*//g')
# get the current dns settings...
for each in `curl -s "$get_info_url"`
do
  domain=`echo "$each" | cut -d"|" -f1`
  dns_ip=`echo "$each" | cut -d"|" -f2`
  update_url=`echo "$each" | cut -d"|" -f3`

  if [ "$ip" != "$dns_ip" ]
  then
          echo -e "$domain - "\\c >> log
          curl "$update_url" >> log
  fi
done

Script Explanation

Really the script is simple but the first thing you have to do is sign up for an account at freedns.afraid.org and choose your domain name. Once you’ve done that, you’re ready to get yourself setup with this script.

Save the script above as dns.sh.
Make sure that the script is set as executable. You can run sudo chmod +x dns.sh to make sure it is.
You now need to get your API key. To do that, go to http://freedns.afraid.org/api/ and it will explain what you need to do. Basically, you need to hash your username and password, but you can find it on that page if you don’t want to hash it yourself.
In the script, replace the string <paste_your_password_hash_here> with the actual hash (don’t forget to save…).
Now you can run the script when ever your IP address updates. Alternatively (my way), you can setup the script as a cron job and have the script check at regular intervals to make sure everything is up to date.

Setting up the Script as a Cron Job

Open a terminal and run crontab -e. If you’ve never done this before, it will probably prompt you to choose an editor. If you don’t know how to use any of the command line editors then you’ll need to do some additional Googleing, I’m not going to explain it. Although, I prefer nano because I don’t understand vi or vim.

Once you have your crontab open, paste the following line somewhere to run the script every 5 minutes.

*/5 * * * * /path/to/script/dns.sh

From now on your DNS information should be up to date. You may have noticed that I’m scraping Google to get your actual current IP address. That was the change I made to the script today. I used to use another site (commented out) but apparently the focus of the site has changed and they don’t really like scripts using it to get the current IP address. What a shame.

Anyway, I hope people find this handy.

Happy automation!

Blekko Inbound Links Extractor Bookmarklet

Mon, 28 May 2012 18:17:00 -0400

Let’s get right to it. Start using the bookmarklet by dragging the following bookmarklet link to your bookmark toolbar.

Blekko - Inbound Links

Go to a Blekko Inbound Links SEO report like: ubuntuforums.com/ /domainlinks
Click on the Blekko - Inbound Links bookmarklet from your toolbar.
Watch for a new <textarea></textarea> to be added to the top of the page containing all the link data from the report.

If you’re pulling data for a popular site like Twitter or Google, be prepared to be patient because there’s a shit ton of scraping going on. Average and/or small sites typically take a second or two. The example provided above takes only 3 or 4 seconds.

On the topic of links…link to me or share it if you like it!

The How and Why

I’m still working to improve my jQuery skills. Recently I needed to automate some tasks relating to scraping data from a site. That particular site was built relying heavily on Javascript and Ajax requests, which meant that I couldn’t use my favorite combination of cURL, grep, cut and Bash to do the scraping. I needed something that would run client-side and execute Javascript.

After thinking about it for a while, I decided I should learn about whether or not I’d be able to create a bookmarklet to do the job. I’ve been learning jQuery over the past few months, so I also wanted to make sure I could pair my developing jQuery skills with this bookmarklet project. I found this great post that provided the exact starting point I was looking for.

After using the framework provided in that post, I was able to complete that project. In fact, the project ended up working so well that I started to wonder about other ways I could use this technique to automate daily tasks.

Blekko’s SEO Inbound Links Report

Very frequently, I use Blekko’s SEO tools to research sites. I’m a big fan of all the free information Blekko provides webmasters and search engine marketers. In particular, I use the Blekko Inbound Links report often. This report shows all the incoming links for any site that Blekko has in its index. It’s really easy to use and navigate. To see this report for yourself, you’ll need to create an account with Blekko (FREE).

After your account is created, you can enter automateeverything.tumblr.com/ /domainlinks to see all the incoming links for this site. My blog is still very young (at the time of writing), so my link profile isn’t very impressive (you can help change that by linking to me!!). Here’s a screenshot of what you get by getting the same report for plus.google.com/ /domainlinks.

So, here’s the rundown of what you’re seeing:

An overall summary of how many incoming links by how many unique domains.
A list of domains, sorted by most authority to least (as judged by Blekko’s “Host Rank”).
The count of how many links there are from each host/domain.

While this is good summary information, I also want to know which pages on the domain the links are coming from, what the anchor text is, what page is being linked to and whether it’s a nofollow link or not. To get that extra detail, you have to click on each of the link counts for each linking domain. That will load a detailed list of links and will show all the information I mentioned above.

Let me say again, I think it’s great that Blekko is providing all this information for free. But…I want more. I want it all in one place. I want it in Excel. I don’t want to try to combine browser addons and/or macros to get all this data. I don’t want to go to each individual from domain links page to get all the to, from and anchor link details. This would take a ton of time to do manually. This time would be far better spent actually analyzing the link data.

At the time of writing, Blekko does not provide an export option. That’s why I’m going to do it for everyone!

The Bookmarklet

This bookmarklet will automate the entire process. In just a second or so, it will scrape all of the information on the page, request each “links” page, scrape all that and then finally output a nicely formatted report in a text box at the top of the page. From there you’ll be able to just copy and paste into Excel to be used in a pivot table, graph or whatever.

I love Ubuntu, so I thought I’d do a demo on UbuntuForums.com to show the power of this bookmarklet.

Like I said, easy and fast! Here’s a few sample lines from the output of the UbuntuForums.com report:

From_Host   Host_Rank   Link_From_URL   Link_From_Anchor    nofollow    Link_To_URL
www.43things.com    468.7   http://www.43things.com/things/view/2140/use-linux      nf  http://ubuntuforums.com/
www.linuxjournal.com    1343.3  http://www.linuxjournal.com/content/post-penguicon-unity-unification-story          http://ubuntuforums.com/
ubuntuforums.org    1167.3  http://ubuntuforums.org/showthread.php?t=520091&page=2351           http://ubuntuforums.com/bump.php?
ubuntuforums.org    1167.3  http://ubuntuforums.org/showthread.php?t=635117 www.ubuntuforums.com        http://www.ubuntuforums.com/
bugs.launchpad.net  653.8   https://bugs.launchpad.net/ubuntu/+source/evolution/+bug/349312     nf  http://ubuntuforums.com/
bugs.launchpad.net  653.8   https://bugs.launchpad.net/ubuntu/+source/fglrx-installer/+bug/545257       nf  http://ubuntuforums.com/
bugs.launchpad.net  653.8   https://bugs.launchpad.net/ubuntu/+source/fglrx-installer/+bug/545257/comments/6        nf  http://ubuntuforums.com/
www.geek.com    927.7   http://www.geek.com/articles/geek-pick/microsoft-expertzone-training-teaches-best-buy-employees-about-linux-inferiority-2009097/        nf  http://www.ubuntuforums.com/
www.ubuntugeek.com  257.5   http://www.ubuntugeek.com/avast-antivirus-for-ubuntu-desktop.html       nf  http://www.ubuntuforums.com/
www.ubuntugeek.com  257.5   http://www.ubuntugeek.com/avast-antivirus-for-ubuntu-desktop.html/comment-page-2        nf  http://www.ubuntuforums.com/
www.piratbyran.org  100.6   http://www.piratbyran.org/index.php?view=forum&a=thread&id=37918&fview=34           http://www.ubuntuforums.com/
www.forevergeek.com 122.7   http://www.forevergeek.com/2005/04/ubuntu_504_hoary_hedgehog_review/    Official Ubuntu forums/suppport site        http://ubuntuforums.com/
www.forevergeek.com 122.7   http://www.forevergeek.com/2005/04/ubuntu_504_hoary_hedgehog_review/?replytocom=21953   Official Ubuntu forums/suppport site    nf  http://ubuntuforums.com/
www.forevergeek.com 122.7   http://www.forevergeek.com/2005/04/ubuntu_504_hoary_hedgehog_review/?replytocom=21959   Official Ubuntu forums/suppport site    nf  http://ubuntuforums.com/
www.forevergeek.com 122.7   http://www.forevergeek.com/2005/04/ubuntu_504_hoary_hedgehog_review/?replytocom=21969   Official Ubuntu forums/suppport site    nf  http://ubuntuforums.com/
www.forevergeek.com 122.7   http://www.forevergeek.com/2005/04/ubuntu_504_hoary_hedgehog_review/?replytocom=21974   Official Ubuntu forums/suppport site    nf  http://ubuntuforums.com/
www.forevergeek.com 122.7   http://www.forevergeek.com/2005/04/ubuntu_504_hoary_hedgehog_review/?replytocom=21985   Official Ubuntu forums/suppport site    nf  http://ubuntuforums.com/www.43things.com    468.7   http://www.43things.com/things/view/2140/use-linux      nf  http://ubuntuforums.com/
www.linuxjournal.com    1343.3  http://www.linuxjournal.com/content/post-penguicon-unity-unification-story          http://ubuntuforums.com/
ubuntuforums.org    1167.3  http://ubuntuforums.org/showthread.php?t=520091&page=2351           http://ubuntuforums.com/bump.php?
ubuntuforums.org    1167.3  http://ubuntuforums.org/showthread.php?t=635117 www.ubuntuforums.com        http://www.ubuntuforums.com/
bugs.launchpad.net  653.8   https://bugs.launchpad.net/ubuntu/+source/evolution/+bug/349312     nf  http://ubuntuforums.com/
bugs.launchpad.net  653.8   https://bugs.launchpad.net/ubuntu/+source/fglrx-installer/+bug/545257       nf  http://ubuntuforums.com/
bugs.launchpad.net  653.8   https://bugs.launchpad.net/ubuntu/+source/fglrx-installer/+bug/545257/comments/6        nf  http://ubuntuforums.com/
www.geek.com    927.7   http://www.geek.com/articles/geek-pick/microsoft-expertzone-training-teaches-best-buy-employees-about-linux-inferiority-2009097/        nf  http://www.ubuntuforums.com/
www.ubuntugeek.com  257.5   http://www.ubuntugeek.com/avast-antivirus-for-ubuntu-desktop.html       nf  http://www.ubuntuforums.com/
www.ubuntugeek.com  257.5   http://www.ubuntugeek.com/avast-antivirus-for-ubuntu-desktop.html/comment-page-2        nf  http://www.ubuntuforums.com/
www.piratbyran.org  100.6   http://www.piratbyran.org/index.php?view=forum&a=thread&id=37918&fview=34           http://www.ubuntuforums.com/
www.forevergeek.com 122.7   http://www.forevergeek.com/2005/04/ubuntu_504_hoary_hedgehog_review/    Official Ubuntu forums/suppport site        http://ubuntuforums.com/
www.forevergeek.com 122.7   http://www.forevergeek.com/2005/04/ubuntu_504_hoary_hedgehog_review/?replytocom=21953   Official Ubuntu forums/suppport site    nf  http://ubuntuforums.com/
www.forevergeek.com 122.7   http://www.forevergeek.com/2005/04/ubuntu_504_hoary_hedgehog_review/?replytocom=21959   Official Ubuntu forums/suppport site    nf  http://ubuntuforums.com/
www.forevergeek.com 122.7   http://www.forevergeek.com/2005/04/ubuntu_504_hoary_hedgehog_review/?replytocom=21969   Official Ubuntu forums/suppport site    nf  http://ubuntuforums.com/
www.forevergeek.com 122.7   http://www.forevergeek.com/2005/04/ubuntu_504_hoary_hedgehog_review/?replytocom=21974   Official Ubuntu forums/suppport site    nf  http://ubuntuforums.com/
www.forevergeek.com 122.7   http://www.forevergeek.com/2005/04/ubuntu_504_hoary_hedgehog_review/?replytocom=21985   Official Ubuntu forums/suppport site    nf  http://ubuntuforums.com/

Notes on Usage

The bookmarklet is cross-browser and cross-platform. If you run into any compatibility issues at all, please drop me a comment with the details of your platform and the query in Blekko.
This bookmarklet works by calling the actual Javascript file from my public Dropbox folder. You can see the full code here: http://dl.dropbox.com/u/56691816/automateeverything/bookmarklets/blekko-inbound-links.js
Since this bookmarklet executes code that’s pulled from an external source, it’s functionality and feature set is subject to change. Not to freak you out but if I decided to do nasty things with the Javascript at some point in the future, the only way you’d know is if you inspected the file each time before running it. But, isn’t that the case with any software that updates its-self?

I hope you enjoy it. Please feel free to let me know if you have feature requests or bugs.

Enjoy and happy automation!

Google Analytics PDF Tracking Made Easy

Sun, 13 May 2012 02:51:00 -0400

Tracking the quantity of PDF downloads is a very common wish for business to business and/or lead generation campaigns. It makes sense right? You understand that if a visitor is requesting a PDF then that visitor wants to learn more and is interested in what they’ve learned from your site so far.

The problem with trying to track PDF downloads as a conversion point or a goal is that a PDF document is different than a normal page. You cannot insert Google Analytics code in that PDF to have it track like you would another page. That means you won’t be able to set it up as a goal in your Analytics profile.

The Good News

The good news is that it’s simple enough to setup Google Analytics Event Tracking. In a nutshell, what this will do it record an “event” each time the PDF link is clicked by a user. Here’s Google’s overview of how to set it up.

If you’re too lazy to read through all the documentation provided by Google, here’s the tl;dr:

Before adding the onClick, the link would look like this:

<a href="/some/pdf/file.pdf">Download this PDF</a>

After adding the onClick, it would look something like this:

<a onclick="_gaq.push(['_trackEvent', 'PDF', 'Download', '/some/pdf/file.pdf']);"" href="/some/pdf/file.pdf">Download this PDF</a>

Then, once the link is clicked, you will record an “Event” in Google Analytics with the category equal to PDF, the action equal to Download and the label equal to /some/pdf/file.pdf. Pretty simple right?

The Bad News

Now it’s time for the bad news. I’m typically in the consultant position. I’m trying to convince a site owner, marketing manager or webmaster why they should go through the trouble to tag each and every “a href” PDF link with this onClick attribute, each of which needs to be customized so we can measure appropriately. This is a lot of work. Typically, the project doesn’t even start. Other times when it tagging these links becomes a project, it rarely is done perfectly and almost never is kept up with. This leads to inconsistent reporting.

The only thing worse than no tracking is inaccurate tracking. There’s nothing worse to a marketing professional (who’s trying to justify one’s own existence) than having to explain that the reporting is invalid because there were errors made in the setup of the analytics. However, IT HAPPENS ALL THE TIME. It’s insanely frustrating.

This blog, after all, is titled “Automate Everything”. Hopefully my solution to this problem won’t disappoint. I’ve been learning Javascript, more specifically JQuery, over the past several months. I was thinking about how to solve this problem. How can I use my new skills to make it insanely easy and efficient for people to add event tracking for Google Analytics to all the PDF links on their site? I’m happy to say, I’ve figured it out.

PDF Event Tracking Solution

My solution will allow you to track every PDF link on your site by just pasting a few lines of JQuery in the footer of your site. Here’s my solution:

<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script>
<script>
    $("a[href$='pdf']").each(function(index) {
      pdfLabel = $(this).attr('href');
      pdfOnClick = "_gaq.push(['_trackEvent', 'PDF', 'Download', '" + pdfLabel + "']);";
      $(this).attr("onClick", pdfOnClick);
    });
</script>

You should be able to place this code in the footer of almost any site and when the page loads, it will go through all the <a href="" tags and look for links with the href ending in .pdf. When it finds one, it simply adds the onClick attribute to perform the event tracking in Google Analytics.

If your site already includes the JQuery framework then there’s no need to repeat it. Simply omit the opening <script> tag and include the rest like so…

<script>
    $("a[href$='pdf']").each(function(index) {
      pdfLabel = $(this).attr('href');
      pdfOnClick = "_gaq.push(['_trackEvent', 'PDF', 'Download', '" + pdfLabel + "']);";
      $(this).attr("onClick", pdfOnClick);
    });
</script>

The only thing you need to keep in mind is that this code needs to be placed as far down on the page as possible. Once this code executes, it will only have an effect on the page elements (DOM elements) that have already been loaded by the browser.

This is a very quick, highly efficient way to enable tracking of your PDF downloads using event tracking in Google Analytics. I’m actually quite surprised that Google doesn’t offer suggestions like this on their help pages.

I hope you’ve found this helpful. Please leave a comment if you have questions and share it if you have found it to be helpful. Thanks for reading.

Happy automation!

Scraping Keyword Suggest on TheFind.com

Thu, 10 May 2012 09:18:00 -0400

This will be a really quick post. If you follow my blog then you’ll already get the gist of this post and the fact that I cannot get enough of keyword research.

Continuing on my quest to scrape every available keyword suggestion tool available for use in my automated keyword research methods, I decided to work on TheFind.com next. I’ve honestly written so many tools like this for many different sites and at this point doing so is really trivial. However, the fact that it’s easy now doesn’t detract from the true value provided.

If this is the first keyword research or keyword suggest scraping related post of mine that you’ve read, please remember to have a look at my post “16 Lines of Bash = Keyword Research Glory”. It will give you a full write up of what’s going on in the video below if it all looks foreign to you.

Video on Scraping TheFind.com’s Keyword Suggest

Please leave a comment if you have questions about the video. I appologize for the lack of audio as I’m currently missing a microphone.

Happy automation!

Blekko Keyword Suggest Scraper

Wed, 09 May 2012 00:19:13 -0400

I love keyword research. That’s a good thing because as a Search Marketing Professional, I do a ton of it. One thing that’s important for keyword research is using a variety of sources to keep your results fresh. Another benefit of having several keyword research sources is having a diverse audience.

When performing keyword research for Paid Search, estimating potential traffic volume is less important than for search engine optimization. As I’ve posted about several times before, my favorite style of keyword research (when search volume estimates don’t matter) is scraping search/keyword suggestion tools. I’ve been using Blekko for about a year now, both as a search engine but also for leveraging their search suggest functionality. It’s only fitting that I show a simple command for returning Blekko keyword suggestions using a Bash shell.

Blekko Keyword Suggestions

Here’s the minimum command to run to get back formatted keyword suggestions from Blekko.

curl -s "http://blekko.com/autocomplete?query=keyword+research" | sed 's/.*\[//g;s/\].*//g;s/","/\n/g;s/"//g'

The only portion of the command above that you’ll need to modify is the keyword+research text, which is equal to the root seed keyword (with spaces replaced with a plus sign). Running the command above produces these results.

keyword research
keyword research tips
keyword research tool
keyword research services
keyword research seo tool
keyword research pro
keyword research software
keyword research seo
keyword research 2
keyword research service

Blekko’s search suggest tool is limited to 10 results at a time, which is very common. If you’ve read my other posts on how this is done in Google, Yahoo, Bing, etc. then you’ll know that I commonly query several search suggestion sources with one command to make this research and comparison even more simple. Below is this Blekko code integrated with my suggestion tool which combines Google, Amazon, Yahoo, Bing and now Blekko.

#!/bin/bash

q=$(echo "$1" | sed 's/ /%20/g')

clear
echo -e "\nGetting Suggestions for \"$1\""

echo -e "\nGoogle:"
curl -s "http://www.google.com/s?sugexp=pfwl&cp=15&q=$q" | sed 's/\[/\n\[/g' | cut -d'"' -f2 | tail -n +4

echo -e "\nAmazon:"
curl -s "http://t1-completion.amazon.com/search/complete?method=completion&q=$q&search-alias=aps&client=amazon-search-ui&mkt=1&x=updateISSCompletion&sc=1" | sed 's/,\[{".*//g;s/,/\n/g' | cut -d'"' -f2 | grep -v '\[\|\]\|\{\|\]' | tail -n +2

echo -e "\nYahoo:"
y_1=$(curl -s "http://sugg.us.search.yahoo.net/gossip-us-ura?droprotated=1&output=sd1&command=$q&nresults=10")
# this way uses our keyword as a suffix rather than prefix. usually there are some duplicates with the first method but those are removed later.
y_2=$(curl -s "http://sugg.us.search.yahoo.net/gossip-us-ura?droprotated=0&output=sd1&command=$q&nresults=10")
echo $y_1 $y_2 | sed 's/{/\n{/g' | grep '"k"' | cut -d'"' -f4 | sort -u

echo -e "\nBing:"
curl -s "http://api.bing.com/qsonhs.aspx?FORM=ASAPIW&mkt=en-US&type=cb&cb=sa_inst.apiCB&q=$q&cp=13&bq=$q" | sed 's/{/\n{/g' | grep '"Txt"' | cut -d'"' -f4
echo

echo "Blekko:"
curl -s "http://blekko.com/autocomplete?query=$q" | sed 's/.*\[//g;s/\].*//g;s/","/\n/g;s/"//g'
echo; echo

If you save the script as sugg.sh, you’ll be able to run it like this.

./sugg.sh "keyword tool"

That will produce the following results:

Getting Suggestions for "keyword tool"

Google:
keyword tool
keyword tool free
keyword toolbox
keyword tool youtube
keyword tool estimator
keyword tool dominator
keyword tool by google
keyword tool traffic estimator
keyword tool seo
keyword tool reviews

Amazon:
keyword tool

Yahoo:

Bing:
keyword tool bing
keyword tool google
keyword tool
keyword tool external
keyword tools free
keyword tool dominator
keyword tool adwords
keyword tool wordtracker

Blekko:
keyword tool
keyword tool api
keyword tool search
keyword tool software
keyword tool download
keyword tool 2
keyword tool review
keyword tool ppc
keyword tool crack
keyword tool training

Pretty cool right! Unfortunately, Yahoo doesn’t seem to have any knowledge of what a “keyword tool” is. I can’t say I’m surprised.

Keyword Suggestion Enumeration

I mentioned above that Blekko limits the keyword suggestion results to 10 phrases and that it’s very common to do so. I’ve also posted about how it’s trivial to start with a root phrase, like “keyword tools”, and then add each letter of the alphabet on to the end to see what results get returned. Doing that for every letter of the alphabet, removing duplicates and then sorting alphabetically is the next natural step in the evolution of any keyword suggestion based keyword research tool.

So, here’s the code for my keyword suggestion enumeration tool with the Blekko code added in. This takes a bit longer to run because it’s sending many, many HTTP GET requests and has to wait for the search engine’s responses. This version reaches out to 5 different search suggestion services and makes a total of 6 different queries per letter of the alphabet, which equals 156 requests. Think about how long that would take you to manually place those searches in your browser and then note the results. Waiting 10 or 15 seconds for a result doesn’t seem so bad now!

Anyway, here is the code.

#!/bin/bash

q=$(echo "$1" | sed 's/ /%20/g')
tmp=tmp.txt

echo "" > $tmp
for suffix in {a..z}
do
    curl -s "http://www.google.com/s?sugexp=pfwl&cp=15&q=$q%20$suffix" | sed 's/\[/\n\[/g' | cut -d'"' -f2 | tail -n +4 >> $tmp
    curl -s "http://t1-completion.amazon.com/search/complete?method=completion&q=$q%20$suffix&search-alias=aps&client=amazon-search-ui&mkt=1&x=updateISSCompletion&sc=1" | sed 's/,\[{".*//g;s/,/\n/g' | cut -d'"' -f2 | grep -v '\[\|\]\|\{\|\]' | tail -n +3 >> $tmp
    curl -s "http://sugg.us.search.yahoo.net/gossip-us-ura?droprotated=1&output=sd1&command=$q%20$suffix&nresults=10" | sed 's/{/\n{/g' | grep '"k"' | cut -d'"' -f4 >> $tmp
    # this way uses our keyword as a suffix rather than prefix. usually there are some duplicates with the first method but those are removed later.
    curl -s "http://sugg.us.search.yahoo.net/gossip-us-ura?droprotated=0&output=sd1&command=$q%20$suffix&nresults=10" | sed 's/{/\n{/g' | grep '"k"' | cut -d'"' -f4 >> $tmp
    curl -s "http://api.bing.com/qsonhs.aspx?FORM=ASAPIW&mkt=en-US&type=cb&cb=sa_inst.apiCB&q=$q%20$suffix&cp=13&bq=$q" | sed 's/{/\n{/g' | grep '"Txt"' | cut -d'"' -f4 >> $tmp
    curl -s "http://blekko.com/autocomplete?query=$q" | sed 's/.*\[//g;s/\].*//g;s/","/\n/g;s/"//g' >> $tmp
    echo >> $tmp
done

sed '/^$/d' $tmp | sort | uniq

If you save it as “enum_sugg.sh”, here’s how you would run it.

./enum_sugg.sh "keyword tools"

And here’s the result you would get (well, at least this is what I get right now).

adwords keyword tools external
best free keyword tools
best keyword tools 2012
free keyword tools online
google adwords keyword tools external
google keyword tool kit
google keyword tools google searches
google keyword tools uk
google keyword tool youtube
keyword density tools
keyword discovery tools
keyword lookup tool
keyword niche tools
keyword optimization tools
keyword phrase tool
keyword preview tool
keyword question tool
keyword research tools 2012
keyword research tools free
keyword research tools review
keyword search tool yahoo
keyword selector tool yahoo
keyword suggestion tool youtube
keyword tool adsense
keyword tool analyzer
keyword tool api
keyword tool average search volume
keyword tool bar
keyword tool beta
keyword tool bing
keyword tool book
keyword tool comma seperated
keyword tool copy list free
keyword tool density
keyword tool discover
keyword tool dominator
keyword tool download
keyword tool estimator
keyword tool external adwords
keyword tool external google
keyword tool finder
keyword tool find little competion warrior
keyword tool for app store
keyword tool for niche ideas
keyword tool generator
keyword tool golf instruction
keyword tool google adwords
keyword tool google external
keyword tool google too many searches
keyword tool in english
keyword tool in google
keyword tool in spanish
keyword tool kit
keyword tool microsoft word
keyword tool multilple
keyword tool multiple
keyword tool old
keyword tool old version
keyword tool overture
keyword tool plr
keyword tool previous interface
keyword tool previous months
keyword tools accuracy
keyword tools accurate
keyword tools adsense
keyword tools adwords
keyword tools adwords google
keyword tools affiliate
keyword tools alexa
keyword tools analysis
keyword tools articles
keyword tools available
keyword tools based
keyword tools beginners
keyword tools best
keyword tools beta
keyword tools bing
keyword tools blog
keyword tools by google
keyword tools by microsoft
keyword tools campaign
keyword tools canada
keyword tools chinese
keyword tools commercial intent
keyword tools compared
keyword tools comparison
keyword tools competition
keyword tools competitors
keyword tools content
keyword tools cpc
keyword tools de adwords
keyword tools dictionary
keyword tools digitalpoint
keyword tools directory
keyword tools disadvantages
keyword tools download
keyword tools dreamweaver
keyword tool search
keyword tool search by city
keyword tool search google
keyword tool search on yahoo google and msn
keyword tools ebay
keyword tools english
keyword tool seo
keyword tool seo organic
keyword tools estimator
keyword tools exact match
keyword tools excel
keyword tools external
keyword tools external google
keyword tools external suggestion tool
keyword tools for adsense
keyword tools for adwords
keyword tools for analysis
keyword tools for bing
keyword tools for ebay
keyword tools for mac
keyword tools for seo
keyword tools for yahoo
keyword tools free
keyword tools free in google
keyword tools from google
keyword tools generator
keyword tools google
keyword tools google adsense
keyword tools google adwords
keyword tools google old version
keyword tools google search global monthly
keyword tools google uk
keyword tools google vs wordtracker
keyword tools guide
keyword tools help
keyword tools home
keyword tools html
keyword tools html code
keyword tools hubpages
keyword tools ideas
keyword tool simple
keyword tools in english
keyword tools in google
keyword tools internet
keyword tools in yahoo
keyword tools ireland
keyword tools kei
keyword tools keywordspy
keyword tools link
keyword tools link building
keyword tools linux
keyword tools list
keyword tools local searches
keyword tools location
keyword tools long tail
keyword tools mac
keyword tools mac free
keyword tools malaysia
keyword tools market
keyword tools market samurai
keyword tools microsoft
keyword tools misspell
keyword tools moneyword matrix
keyword tools msn
keyword tools music
keyword tools nichebot
keyword tools niche finder
keyword tool software
keyword tools old interface
keyword tools omniture
keyword tools on google
keyword tools online
keyword tools online local
keyword tools on the internet
keyword tools on yahoo
keyword tools organic seo
keyword tools overture
keyword tools page optimization
keyword tools paid search
keyword tools pay per click
keyword tools pdf
keyword tools photos
keyword tools ppc
keyword tools price
keyword tools rank
keyword tools rapid
keyword tools rating
keyword tools remove duplicates
keyword tools research
keyword tools resume
keyword tools review
keyword toolss
keyword tools samurai
keyword tools search
keyword tools search engine optimization
keyword tools seo
keyword tools seobook
keyword tools site
keyword tools software
keyword tools south africa
keyword tools spy
keyword tools statistics
keyword tools techniques
keyword tools that work
keyword tools tips
keyword tools to compete
keyword tools tracker
keyword tools traffic
keyword tools traffic estimator
keyword tools traffic travis
keyword tools trends
keyword tools tricks
keyword tools twitter
keyword tools uk
keyword tools url
keyword tools warez
keyword tools webmaster
keyword tools websites
keyword tools wiki
keyword tools word
keyword tools wordpress
keyword tools wordtracker
keyword tools wordze
keyword tools work
keyword tools worth
keyword tools yahoo
keyword tools yahoo panama
keyword tools youtube
keyword tool target geographic location
keyword tool that shortens urls
keyword tool tips
keyword tool traffic estimator
keyword tool uk
keyword tool volume
keyword tool word document
keyword tool yahoo
keyword tool yuri arcurs
lsi keyword tools
paid keyword tools
seo tools keyword density
seo tools keyword list generator

I dare you to find a better keyword research tool or method than this. I doubt you will, but if you do, I’d love to know about it and be proven wrong!

Anyway, I hope you enjoyed the post. Please feel free to “steal” my code, use it, modify it, etc.

As always, happy automation!