Monday, May 13th, 2013

Robots.txt and Invisible Characters – How One Hidden Character Could Cause SEO Problems

How syntax errors in robots.txt can cause SEO problems.

If you’ve read some of my blog posts in the past, then you know I perform a lot of SEO technical audits.  As one of the checks during SEO audits, I always analyze a client’s robots.txt file to ensure it’s not blocking important directories or files.  If you’re not familiar with robots.txt, it’s a text file that sits in the root directory of your website and should be used to inform the search engine bots which directories or files they should not crawl.  You can also add autodiscovery for your xml sitemaps (which is a smart directive to add to a robots.txt file).

Anyway, I came across an interesting situation recently that I wanted to share.  My hope is that this post can help some companies avoid a potentially serious SEO issue that was not readily apparent.  Actually, the problem could not be detected by the naked eye.  And when a problem impacts your robots.txt file, the bots won’t follow your instructions.  And when the bots don’t follow instructions, they can potentially be unleashed into content that should never get crawled.  Let’s explore this situation in greater detail.

A sample robots.txt file:

Sample Robots.txt File

Technical SEO – Cloaked Danger in a Robots.txt File
During my first check of the robots.txt file, everything looked fine.  There were a number of directories being blocked for all search engines.  Autodiscovery was added, which was great.  All looked good.  Then I checked Google Webmaster Tools to perform some manual checks on various files and directories (based on Google’s “Blocked URLs” functionality).  Unfortunately, there were a number of errors showing within the analysis section.

The first error message started with the User-agent line (the first line in the file).  Googlebot was choking on that line for some reason, but it looked completely fine.  And as you can guess, none of the directives listed in the file were being adhered to.  This meant that potentially thousands of files would be crawled that shouldn’t be crawled, and all because of a problem that was hiding below the surface…  literally.

Blocked URLs reporting in Google Webmaster Tools:

Blocked URLs in Google Webmaster Tools

 

Word Processors and Hidden Characters
So I started checking several robots.txt tools to see what they would return.  Again, the file looked completely fine to me.  The first few checks returned errors, but wouldn’t explain exactly what was wrong.  And then I came across one that revealed more information.  The tool revealed an extra character (hidden character) at the beginning of the robots.txt file.  This hidden character was throwing off the format of the file, and the bots were choking on it.  And based on the robots syntax being thrown off, the bots wouldn’t follow the instructions.  Not good.

Invisible Character in Robots.txt

I immediately sent this off to my client and their dev team tracked down the hidden character, and created a new robots.txt file.  The new file was uploaded pretty quickly (within a few hours).  And all checks are fine now.  The bots are also adhering to the directives included in robots.txt.

 

The SEO Problems This Scenario Raises
I think this simple example underscores the fact that there’s not a lot of room for error with technical SEO… it must be precise.  In this case, one hidden character in a robots.txt file unleashed the bots on a lot of content that should never be crawled.  Sure, there are other mechanisms to make sure content doesn’t get indexed, like the proper use of the meta robots tag, but that’s for another post.  For my client, a robots.txt file was created, it looked completely fine, but one character was off (and it was hidden).  And that one character forced the bots to choke on the file.

 

How To Avoid Robots.txt Formatting Issues
I think one person at my client’s company summed up this situation perfectly when she said, “it seems you have little room for error, SEO seems so delicate”.  Yes, she’s right (with technical SEO).  Below, I’m going to list some simple things you can do to avoid this scenario.   If you follow these steps, you could avoid faulty robots.txt files that seem accurate to the naked eye.

1. Text Editors
Always use a text editor when creating your robots.txt file.  Don’t use a word processing application like Microsoft Word.  A text editor is meant to create raw text files, and it won’t throw extra characters into your file by accident.

2. Double and Triple Check Your robots.txt Directives
Make sure each directive does exactly what you think it will do.  If you aren’t 100% sure you know, then ask for help.  Don’t upload a robots.txt file that could potentially block a bunch of important content (or vice versa).

3. Test Your robots.txt File in Google Webmaster Tools and Via Third Party Tools
Make sure the syntax of your robots.txt file is correct and that it’s blocking the directories and files you want it to.  Note, Google Webmaster Tools enables you to copy and paste a new robots file into a form and test it out.  I highly recommend you do this BEFORE uploading a new file to your site.

4. Monitor Google Webmaster Tools “Blocked URLs” Reporting
The blocked urls functionality will reveal problems associated with your robots.txt file under the “analysis” section.  Remember, this is where I picked up the problem covered in this post.

 

Extra Characters in Robots.txt – Cloaked in Danger
There you have it.  One hidden character bombed a robots.txt file.  The problem was hidden to the naked eye, but the bots were choking on it.  And depending on your specific site, that one character could have led to thousands of pages getting crawled that shouldn’t be.  I hope this post helped you understand that your robots.txt format and syntax are extremely important, that you should double and triple check your file, and that you can test and monitor that file over time.  If the wrong file is uploaded to your website, bad things can happen.  Avoid this scenario.

GG

 

Posted by Glenn Gabe @ 8:02 am in bing, google, seo, tools
No Comments »
Friday, April 19th, 2013

AdWords for Twitter – How To Set Up Keyword Targeting Campaigns in Twitter Ads [Tutorial]

Keyword Targeting in Twitter Ads

There was a big announcement on Wednesday in the advertising world.  Twitter finally launched a keyword targeting solution for Twitter Ads.  Yes, that means AdWords-like targeting for Twitter.  Many in the industry (including myself) have thought for a long time that Twitter should launch something like this… and it’s finally here.

As soon as I heard the news, I ran to my Twitter Advertising dashboard to jump in.  Based on setting up my first set of campaigns, I decided to write this post to detail how keyword targeting works in Twitter, how to set up a campaign, explain the targeting options you have, etc.  My hope is that after you read this post, you’ll be off and running with your first AdWords-like Twitter campaign. :)

What is Keyword Targeting in Twitter?
Let’s start with a quick introduction.  With this new release, advertisers can now promote certain tweets based on keywords that users are searching for on Twitter, based on keywords used in their tweets, or based on keywords found in tweets users recently engaged with.  When a match is made, your promoted tweet has an opportunity to win placement either in a user’s Twitter stream or in the search results (for when someone searches Twitter).

A Promoted Tweet looks like this:
Example of Promoted Tweet

 

Creating Your First Campaign
In your Twitter Ads dashboard, you should click “Create campaign” in the upper left-hand corner.

Create a keyword targeting campaign in Twitter Ads

 

Next, click the “Promote your Tweets” button to select a promoted tweets campaign.

 

Select campaign type in Twitter Ads

 

Once you select a promoted tweets campaign, you should click the button labeled “Target by keywords” to create a keyword targeting campaign.

 

Select keyword targeting in Twitter Ads

 

Your Options (Feed and Search)
When you decide to launch a keyword targeting campaign in Twitter, you’ll have to decide if you want to target the search results or users’ timelines.  I like that Twitter forces you to break out campaigns by type, since I would have recommended doing that anyway.   This will enable you to better analyze the effectiveness of your targeting (very similar to breaking out display network and search campaigns in AdWords).

Target search or users' timelines

 

Keyword Match Types
Next, you’ll need to address the keywords you will target for your campaign.  Similar to AdWords or Bing Ads, you have several options when targeting keywords.  You can choose unordered match (which is similar to broad match), phrase match, or exact match.  Unordered match will match your keywords, but those keywords can be in any order.  Phrase match requires the keywords be in the exact order you list, but other keywords can be at the beginning or end of the query.  And exact match requires a perfect match on the keywords you enter – without any other keywords in the query.  Exact match is the most restrictive of the match types and is only available when you target the search results.

Choosing keywords for your Twitter Ads campaign

 

Note, you can also import keywords as comma separated, or line separated (which is a nice option for importing keywords from other platforms).

 

Importing keywords in Twitter Ads

 

Negatives
If you are familiar with paid search advertising, then you already know the power of using negatives.  Well, Twitter Ads enable you to include several types of negatives, including negative broad match, negative phrase match, and negative exact match.  Note, negatives are only available when you target the search results versus users’ timelines.  This makes sense, since most tweets don’t contain a few words (like a search query does).  When you use a negative, it tells Twitter to not show your ad when the negative appears in a search query.

Using negatives in keyword targeted campaigns in Twitter Ads

 

Your Ad, I Mean Tweet
When you use keyword targeting, you are setting up a “Promoted Tweets” campaign.  So, you aren’t setting up ads.  Instead, you are choosing a specific tweet to promote.  That’s important to understand or you can make the mistake of tweeting out an ad-sounding message during the campaign creation process.  When you set up your keyword targeting campaign, you can select a tweet to specifically promote from your stream.  You can also create a new tweet to promote, if that makes more sense for your campaign.  And yes, that makes the most sense, since you will want to tag your destination URL’s so your analytics package can accurately report your campaign statistics.

Selecting a tweet to promote in Twitter Ads

 

Personally, I think Twitter can learn something from Facebook here.  Using Facebook Ads, you can create an unpublished post to promote, and then use various targeting methods to reach your audience.  I would love to have that ability using Twitter Ads.  For example, an advertiser could create specific tweets to promote that wouldn’t necessarily show up in their stream (that all of their followers would see).  That would also enable you to split test your tweets more effectively.  Hey, it’s just an idea. :)
Targeting Options for Keyword Campaigns in Twitter Ads

Location Targeting
Once you choose your keywords, set negatives, and choose your promoted tweet, you can target your audience in a number of ways.  For example, you can use location targeting to limit your promoted tweets to users in a specific geographic region.  As of now, you can target users by country, state, and metro area.  For example, you can target the United States, New Jersey, or the Philadelphia Metro area.  Note, you cannot target at a smaller city or town level (at least yet).

Location Targeting in Twitter Ads

Target by Gender
In addition to using location targeting, you can also target by gender.  There are radio buttons for “any gender”, “male only”, and “female only”.  Twitter infers gender by the tweets shared by users, by their profiles, and by their follow graphs.  Using gender targeting, you can test response rates by gender (by splitting out campaigns by gender).  That’s exactly what I did when setting up my first keyword targeting campaign.

Gender Targeting in Twitter Ads

Device Targeting
Keyword targeting campaigns also enable you to target by device.  For example, you can target desktop and laptops, iOS, Android,  Blackberry, and “other mobile devices”.  You can use this targeting capability to split mobile and desktop campaigns, to target specific platforms, etc.

Device Targeting in Twitter Ads

 

Bid and Budget
At the bottom of the campaign setup screen are fields for campaign budget, daily budget, and maximum bid.  You can also use standard or accelerated delivery for your daily budget, which will either spend your budget as fast as possible starting at midnight, or spread that budget throughout the day.  When setting a maximum CPC, Twitter will supply a suggested bid range.

Setting a bid and budget in Twitter Ads

Once you have entered your keywords, chosen a tweet to promote, set up targeting, and set your bid and budget, then you are ready to rock and roll.  Simply click the “Create Campaign”  button at the bottom of the screen to launch your campaign!

Long Overdue, But Glad Keyword Targeting is Here
Although many in the industry believe this is long overdue, I’m thrilled that Twitter has finally released keyword targeting for Twitter Ads.  Over the past few days, I’ve been setting up various campaigns and testing performance, engagement, etc.  I plan to write more posts in the near future, based on the results of my initial campaigns.  So stay tuned.  In the meantime, I recommend jumping in yourself.  You can set up your first keyword targeting campaign today by following the instructions I included above.  Good luck.

GG

 

 

Posted by Glenn Gabe @ 6:55 am in sem, social-advertising, twitter
No Comments »
Sunday, April 14th, 2013

You Might Be Losing Out – How To Make Sure Sitelink Extensions in Bing Ads Are Tracked Properly [Tutorial]

Bing Ads released sitelink extensions in October of 2012, which enables advertisers to provide additional links in their text ads.  Google AdWords has had ad sitelinks for some time, so this was a great addition by our friends at Bing Ads.  For example, if you were an ecommerce website selling sporting goods, you could provide ad sitelinks for your top categories, like football, baseball, basketball, etc. right beneath your standard text ad.  Sitelink extensions are great usability-wise, while they also provide a nice advantage in the SERPs (since they take up more real-estate).

Here are two examples of sitelink extensions in action (2 Formats):
Example of Sitelink Extensions in Bing Ads for Lucky Jeans

 

Example of Sitelink Extensions in Bing Ads for Adidas

So, let’s say you set up sitelink extensions for some of your campaigns, and you’re basking in the glory of those beautiful ads (and the click through they are getting).  But, maybe your reporting isn’t lining up clicks and visits-wise.  Sure, there are several reasons that could be happening, but maybe it got worse since you launched sitelink extensions.  Well, the reason could very well be the lack of tagging on your ad sitelinks.  If those additional URLs aren’t tagged properly, then your analytics package could very well be reporting that traffic as organic search.  And that would be shame.

In this post, I’m going to walk you through why this could be happening, and how to rectify the situation.  After reading this post, you might just run to Bing Ads today and make changes.  Let’s jump in.

Sitelink Extensions and Tracking Parameters
In Bing Ads, you can include sitelink extensions several ways.  First, you can add them manually via the Bing Ads web UI.  Second, you can use Bing Ads Editor to add them locally, and then upload them to your account.  And third, and possibly the top reason ad sitelinks don’t get tagged, is that you can import them from AdWords via the “Import from Google” functionality.  Note, the import from AdWords functionality is awesome, so don’t get me wrong.  It’s just that it’s easy to import ad sitelinks and not know they are there.  Then you run the risk of uploading untagged sitelink extensions.

How To Create Sitelink Extensions in Bing Ads

So, you need to make sure that your ad sitelinks are tagged properly, based on the analytics package you are using to track campaigns.  For example, if you are using Google Analytics, then you need to make sure that you identify each click coming from your sitelink extensions.  That means you should be appending tracking parameters to your sitelink URLs.  For Google Anlaytics, you can use URL Builder to tag your landing page URLs.

Tagging Sitelink URLs Using URL Builder

 

How To Tag Your Ad Sitelinks in Bing Ads
Again there are various ways to include sitelink extensions in your campaigns, from using the web UI to using Bing Ads Editor to using the “Import from Google” functionality.  I’ll quickly cover each method below to make sure you know where to apply your tracking parameters.

1.  The Bing Ads Web UI
You can currently apply ad sitelinks at the campaign level in Bing Ads.  When you access a campaign, you can click the “Ad Extensions” tab to include ad sitelinks.  Once there, you can click “Create” to add a new sitelink extension.  If you have other sitelink extensions set up across campaigns, you will see them listed (and you can apply those to your campaign if it makes sense).

Creating Sitelink Extensions Using the Bing Web UI

If you want to add a completely new sitelink extension, then click “Create New”.  When adding the sitelink extension, Bing Ads provide a field for link text and then a field for the destination URL.  When you add the URL, make sure your tracking parameters are added!  If not, your visits will show up as “Bing Organic” versus “Bing CPC”.  Good for the SEO team, but not so good for the paid search team.  :)

 

Adding Sitelinks Using the Bing Web UI

 

2. Bing Ads Editor
I love Bing Ads Editor.  It’s an awesome way to manage your campaigns locally and then sync with the Bing Ads web UI.  And as you can guess, there is functionality for adding and editing sitelink extensions in Bing Ads Editor.  You can access your sitelink extensions by clicking the “Ad Extensions” tab for any selected campaign.

Once you click the “Ad Extensions” tab, you can add sitelink extensions by clicking the “Create a Sitelink Extension” button from the top menu.  Then similar to the web UI, you can add the link text and the destination URL.  When adding your destination URLs, make sure your tracking parameters are added.

Adding Sitelinks Using the Bing Ads Editor

 

3. Import from Google (in Bing Ads Editor)
As I explained earlier, I love having the ability to import campaigns, changes, etc. from AdWords directly into Bing Ads Editor.  It makes managing campaigns across both platforms much more efficient.  But, I’ve seen advertisers import campaigns from AdWords that have sitelink extensions, but they don’t realize it.  Then they upload their campaigns to Bing Ads and don’t understand that prospective customers are clicking their sitelinks, visiting their sites, etc., but those visits aren’t being tracked correctly.  Again, those visits will show up as “Bing Organic” in your analytics reporting.

When you go through the process of importing your campaigns, make sure you double check the “Ad Extensions” tab for the newly-imported campaign.  You just might find sitelink extensions sitting there.  And yes, they very well could be left untagged.  Make sure you add your tracking parameters before uploading them to Bing Ads (from Bing Ads Editor).

You can also uncheck the “Ad Extensions” radio button when importing your campaigns from AdWords.  Then you can add your sitelink extensions directly in Bing Ads Editor (via the second method I covered earlier in this post.

Importing Sitelink Extensions in Bing Ads Editor

 

Sitelinks Are Powerful, But Only If They Are Tracked
Sitelinks extensions are a great addition to Bing Ads, and they absolutely can yield higher click through rates.  But, you need to make sure those clicks are being tracked and attributed to the right source – your Bing Ads campaigns!  I recommend checking your campaigns today to make sure your sitelink extensions have the proper tracking parameters appended.  If not, you can quickly refine those links to make sure all is ok.   And when everything is being tracked properly, you just might see a boost in visits, orders, and revenue being attributed to Bing Ads.  And that’s always a good thing.

GG

 

 

Posted by Glenn Gabe @ 7:15 am in adcenter, adwords, bing-ads, google-analytics, sem, seo, tools
No Comments »
Sunday, March 24th, 2013

Facebook Graph Search Update: Facebook Now Passing Keywords to Destination Websites, Can Track in Google Analytics

Facebook Graph Search Results and Not Provided

Facebook Graph Search launched in January, and I was fortunate to have early access.  Upon gaining access, I began heavily testing Facebook’s new social search capabilities.  That research led to my first post covering my initial thoughts about Graph Search, including its impact on local search, privacy, reputation management, etc.  You should check out that post out after reading through this one.

Graph Search and Not Provided
One (unfortunate) point I covered in my post was the apparent use of “not provided” in Graph Search.  If you’re not familiar with “not provided”, last year Google started encrypting searches for users that were logged into a Google account.  As a result, the keywords that users searched for would not be passed in the referrer when they visited your website.  Essentially, webmasters started seeing “not provided” show up in their analytics reporting versus the organic keywords that led to the site. Needless to say, this was a huge problem and the “not provided” numbers have been growing ever since.  For some websites, “not provided” accounts for over 60% of all organic searches.

Now back to Facebook Graph Search and how it handles searches.  When Graph Search launched, I quickly realized that Graph Search falls back to Bing results when it can’t answer your query.  You can read my post about Graph Search to learn about the autocomplete funnel that you get forced down.  So, if you aren’t searching for photos, people, places, or interests, then Facebook will display the “web results” for the query (which is basically the Bing results).

Facebook Graph Search Falls Back to Bing Results

During my testing, I quickly checked the referrer when clicking from the fallback results to websites and noticed that Facebook was not passing the keyword in the referrer!  It was basically Facebook’s version of “not provided”.  When you checked Google Analytics, the visit looked like a typical referring source (a “Social” visit from Facebook).  You couldn’t see the keyword that triggered the visit.  Needless to say, I thought Facebook was making a big mistake.  If Facebook passed the keywords along, business owners, advertisers, SEO’s, etc. would love them for doing so…  We could all start to track the impact of Graph Search (at least any searches that fall back to Bing).   I guess we were out of luck… or were we?

SES NY Session Research and the “Referrer” Moment
SES NY is this upcoming week, and my session covers Facebook Graph Search (based on the research I mentioned earlier).  While I was finalizing my presentation over the past week, I noticed something very interesting.  I started a slide on “not provided”, and then double-checked some searches leading from Facebook to external websites.  What I found surprised me!

I noticed that the keyword was now being passed along in the referrer!   Below is an example of a new referrer.  You can clearly see the q={keyword} being passed now.

Facebook Passing Keywords in the Referrer

That’s awesome, but now we need to track the keywords.  Read on.

Google Analytics and Graph Search
As you can guess, I jumped into Google Analytics to see how this was being picked up.  Since Facebook isn’t an official search engine in GA, it was still showing up as a referring site (without the keyword showing up).  But, since the q= querystring parameter was being passed in the referrer, I knew I could surface those keywords via advanced filters.  So, I quickly set up a new profile and added a filter that would capture graph searches from Facebook.  And it works.  I explain how to quickly set this up below.

How to Track Graph Searches in GA
I’m not going to explain the inner workings of advanced filters in this post (it could easily be a series of posts).  Instead, I’ll show you how to quickly set up an advanced filter that will capture graph search keywords, and then report them in Google Analytics.  Note, you can tackle this several ways, and this is just one solution.  Feel free to tailor your own GA setup as you see fit.

1. Launch Google Analytics and Access Your Website Profiles
You can access your profiles by clicking the “Admin” button in the upper right-hand corner of the GA interface.  The first tab should list all profiles for the website at hand (for the “property” in GA).

Profiles in Google Analytics

 

2. Create a New Profile
Click the “New Profile” button and give it a name.  Then choose your reporting time zone.  Then click the “Create Profile” button.

Create a new profile in Google Analytics.

 

3. Add an Advanced Filter
Once you create your profile, you should click the “Filters” tab so you can add an advanced filter.

Add a new filter in Google Analytics

 

4. Adding a New Filter
Click the “New Filter” button, and then make sure the settings match what is list below.  We will name the filter, create a custom filter, select advanced, and then enter patterns to match in order to capture graph search keywords (and then report them in GA).

How to add an advanced filter in Google Analytics

 

5. Save the Filter
Click “Save” at the bottom of the form and you should see your new filter listed for your profile.

 

6. Check Your Reporting
Now you need to wait for graph searches to come in.  Note, this filter will just pick up graph searches that fall back to Bing and that lead to your site.  If you have access to Graph Search, then you can quickly test it out by searching Facebook for a query that your site ranks for in Bing organic.  Then click through the Facebook Graph Search results and check your reporting later on (the standard GA reporting isn’t real-time, so the reports will lag slightly).

To find the new data, you can click the “Sources” Tab, and then click “All Traffic”.  Then find “facebook.com” and click through.  Then you can add a secondary dimension for “User Defined Value” which will contain your graph search keywords.  When setting the secondary dimension in your reporting, click “Content” and then select “User Defined Value”.  Note, the new advanced filter will capture all q={keyword} combinations.  That’s why you need to drill into “facebook.com” in the sources to isolate Graph Search results.

Adding a secondary dimension in Google Analytics

 

Your reporting will look something like this:

Viewing graph search keywords in Google Analytics

Once you start picking up graph searches, you can tie those searches to site activity, performance, conversion, etc.  In addition, we can start to see how many graph searches are being conducted that drive users off of Facebook.com (meaning those searches can’t be answered yet by Facebook Graph Search).

Summary – Capture Graph Search Keywords
Again, I’m glad to see that Facebook is passing keywords along in the referrer!  For a while, I thought we were going to be stuck with another “not provided” situation.  Although Facebook Graph Search has only been rolled out to a few hundred thousand users, it’s only a matter of time before the 1 billion+ active user base gains access.  And that will be across mobile devices too.  My recommendation is to set up this advanced filter today and start tracking graph searches.  You never know what you are going to find.  :)

Last, but not least, if you are attending SES NY this week, and are interested in learning more about Facebook Graph Search, you should definitely check out my session.  It’s at 2:15 on Thursday, March 28th.  I’ll be covering a range of important findings, based on my research.

GG

 

 

Posted by Glenn Gabe @ 8:43 am in facebook, google, seo, social-media
20 Comments »
Thursday, March 21st, 2013

How To Properly Demote Sitelinks in Google Webmaster Tools

How to remove sitelinks in Google Webmaster Tools

I’ve received several questions recently about how to remove sitelinks in Google.  If you’re not familiar with sitelinks, they are additional links that Google provides under certain search listings.  Sitelinks enable users to drill deeper into a site directly from the search results.  You typically see sitelinks for branded searches.

For example, here are sitelinks for Amazon:
Sitelinks for Amazon.com

 

And here are sitelinks for the Apple iPad:
Sitelinks for Apple iPad


How Google Determines Sitelinks
Google algorithmically determines sitelinks for a given query/url combination.  This is based on a number of factors that Google takes into account.  For example, Google explains that it analyzes a site’s link structure to determine if there are additional links it can provide in the search results that will save users time (by quickly enabling them to link to core pages on your site).  Remember, Google always wants to connect users with the information they are seeking as fast as possible.

No, Google Doesn’t Always Get It Right
If you are checking your rankings and notice strange sitelinks showing up, you can always demote those links via Google Webmaster Tools.  For example, you might see sitelinks that are irrelevant, too granular, or links that could end up sending users to supporting pages that wouldn’t provide a strong user experience.  Whatever the case, you can take action.

For cases like this, you can use the “Sitelinks” section of Google Webmaster Tools to demote specific sitelinks.  Note, if you don’t have Google Webmaster Tools set up for your site, stop reading this post, and set it up NOW.  You can set up your account and verify your site in just a few minutes, and then you’ll receive a boatload of important data right from Google.

Demoting Sitelinks in Google Webmaster Tools
Once you set up a webmaster tools account, you can access the sitemaps section to begin demoting specific sitelinks.  Below is a step by step tutorial for demoting sitelinks that shouldn’t be showing up below your search listings.

1. Access the Sitelinks Section of Webmaster Tools
Access Google Webmaster Tools and click the “Configuration” tab, and then “Sitelinks” to access the demotion form.

How to access sitelinks in Google Webmaster Tools

2. Choose Wisely When Demoting Sitelinks
There are two text fields you need to concern yourself with in the “Sitelinks” section.  The first is labeled, “For this search result:” and it refers to the webpage that shows up in the search results that contains sitelinks.  I know this is where confusion starts to set in, so let me say that again (and show you what I mean).

The first text field is not for the sitelink URL you want to demote.  It’s for the webpage that the sitelinks show up for.  It’s the URL that’s displayed at the top of the search listing.  Note, if you are demoting a sitelink for your homepage, you can leave this field blank.  It’s also worth noting that Google provides the root URL of your site already in the text field, so you just need to worry about the remaining part of the URL, which is called the URI (everything after http://www.yourdomain.com/).

Enter search result when demoting sitelinks.

For example, if you were the VP of Marketing for Apple, and wanted to remove the “Refurbished iPad” sitelink for the iPad page, then you would enter http://www.apple.com/ipad/ in the first field.

How to remove sitelinks for the ipad search result.


3. Demote the Sitelink URL
The second field is where you will enter the URL of the sitelink you want to demote.  Using our Apple example above, you would enter http://store.apple.com/us/browse/home/specialdeals/ipad in the field to demote the “Refurbished” sitelink for the ipad URL.  That’s the refurbished iPad page on Apple’s site (and it’s where the sitelink in the search results points to).

Once you enter the URL, you can click the red “Demote” button.  Once you do, the demoted sitelink will be listed below the form with the search result it applies to, the specific sitelink URL, and a “Remove Demotion” button.  If you ever want to remove the demotion, just access this page again, and click “Remove Demotion”.  Then give Google a few days to apply the changes.

Enter the sitelink url to demote.

 

Misc. Sitelink Information
Based on the questions I have received when helping clients demote sitelinks, I figured I would provide some additional information in this post.

1. How Long Does it Take for Google to Demote Sitelinks?
I’ve seen sitelinks get demoted in just a few days.  That said, it definitely varies per site…  I’ve seen it take a little longer in certain cases.  I recommend monitoring the sitelinks for the page in question for at least a week or two after demoting a sitelink.  If you notice that it’s still showing up, then revisit the form to make sure you demoted the right sitelink for the right search result.

2. How Many Sitelinks Can I Demote?
You can demote up to 100 URL’s via Google Webmaster Tools.  That should be plenty for most webmasters.  Actually, I’d argue that something is very wrong if you are demoting too many sitelinks…  You might want to analyze your internal navigation, including the anchor text, to see if Google is picking up something that it shouldn’t be.

Summary – Demotion Can Be A Good Thing
I hope this tutorial helped you better understand what sitelinks are and how to address the wrong sitelinks showing up in the search results.  If you notice any weird sitelinks showing up in the search results for your site, then visit Google Webmaster Tools and demote those specific sitelinks.  It’s one of the few times that a demotion could be a good thing.

GG

 

Posted by Glenn Gabe @ 9:16 am in google, seo, tools
No Comments »
Wednesday, February 27th, 2013

Enhanced Campaigns and Local Search | How To Use Bid Adjustments with Radius Targeting to Customize Bids Per Location [Tutorial]

Enhanced Campaigns in Google AdWords

On February 6th, Google AdWords released a major update to its platform called Enhanced Campaigns.  Advertisers can upgrade to enhanced campaigns now (as an option), but all campaigns will be upgraded by mid-2013.  Enhanced campaigns provide a radically different way to target by device, adjust bids for various targeting options, set up versatile ad extensions, etc.  Therefore, if you’re running paid search campaigns in AdWords, you should get up to speed sooner than later.

The major changes to AdWords with enhanced campaigns include the ability to combine targeting methods in one single campaign versus having to break out separate campaigns.  For example, in the past, it was optimal to separate your desktop and mobile campaigns.  This would enable you to tailor creative, drive users to mobile optimized pages, bid differently, etc.

In addition, if you were targeting various locations, you would also break those out to different campaigns.  For example, you might have had one campaign targeting New York and another New Jersey (in separate campaigns).  Using enhanced campaigns, you can use bid adjustments to control bidding per location versus having to break them out to additional campaigns.

Those are just two of the changes that enhanced campaigns bring, and you should definitely read up on the rest.  Again, you will be forced to upgrade at some point, and you don’t want to frantically get familiar with enhanced campaigns at the last minute.

Local Businesses and Radius Targeting
So, how do enhanced campaigns impact location targeting for local businesses?  Well, if you are targeting users located near your physical location, then enhanced campaigns provide a great way to control bids by location.  Specifically, you can use radius targeting to adjust bids based on the distance from your store, office, etc.

Radius Targeting in Google AdWords
Radius Targeting in Google AdWords

For example, let’s say you ran a plumbing service in the Princeton Area.  Since you need to travel to customer locations, and many need help immediately, you know that prospective customers located within 15 miles of your office are highly valuable targets.  But, you also know that prospective customers as far out as 30 miles can work for your business too.  But, time equals money, and 30 miles out isn’t ideal when you factor in travel time.  Again, this is all hypothetical, but you get the picture.

So, based on this situation, you could use enhanced campaigns to target users 15 miles out and 30 miles out, and set bids accordingly (using bid adjustments).  A bid adjustment lets you increase or decrease your bids, based on certain criteria.  In this case, you might set a standard bid that your 15 mile radius targeting could use, and then decrease your bid for users located up to 30 miles away.  Then AdWords would dynamically adjust your bids based on the location of the person searching for your services.

Below, I’m going to show you how to set this up in an enhanced campaign.  Note, you would obviously need to first upgrade your campaign to “enhanced” before being able to implement the steps listed below.  Google has provided an upgrade guide in case you want to upgrade one of your campaigns prior to following the steps below.

How to Use Radius Targeting in Enhanced Campaigns (with Bid Adjustments):

1. Access one of your campaigns that has been upgraded to an “enhanced campaign” and click the “Settings” tab.

Campaign Settings in Enhanced Campaigns

2. Click the “Locations” tab within your campaign settings.  Then click “Edit locations”.

Location Tab in Campaign Settings (in Enhanced Campaigns)

3. This is where you can add locations to target.  In the field labeled, “In what locations do you want your ads to appear”, enter the city or zip code you want to target.  For example, I will enter “Princeton, NJ”.   AdWords will autopopulate locations based on what you’re entering. When you see the city you want to target, click the “Nearby” link.

Adding a location to target in enhanced campaigns.

4. Click “Radius Targeting” in the top row of links and re-enter the city and state combination.  Then choose a distance away from your location for the first target location.  Let’s enter 15 miles and click “Search”.

Using Radius Targeting in Enhanced Campaigns

5. AdWords will show the custom location underneath the radius targeting field.  You can then click “Add” to add that location as a target for your campaign.  Click “Done” at the bottom of the location targeting window after you have added the first location.

Adding Radius Targeting in Enhanced Campaigns

6. But we’re not done yet.  We need to add a second targeting option for 30 miles away.  Then we’ll adjust bids based on location.    You should see your first location in the “Edit Locations” window (with the radius targeting you just selected).  You should only see one location at this point.  In order to add another location target, click the “Nearby” link next to the first location target.  This will bring up the location targeting window again.

Adding a second location target in enhanced campaigns.

7. Similar to what we did earlier, click the “Radius Targeting” link in the top list of links.  Then enter your city and state combination, or zip code, and enter a radius.  This time, we will enter 30 miles.  Remember, we want to still reach users up to 30 miles away, but we don’t want to pay as much per click.  Once you click “Search”, AdWords will show your location with the radius you selected.  Then simply click “Add” to add that as a second location target.  Click “Done” at the bottom of the location targeting window after you have added the second location.

Adding a second location target using radius targeting in enhanced campaigns.

8. Below the “Edit Locations” window, you should see your two location targets (one for 15 miles out and the other for 30 miles out).  You should see a column for “Bid adj.” which will enable you to include bid adjustments per target.  Our strategy is to bid normally for users located up to 15 miles out from our location, but we want to lower that bid by 30% for users located up to 30 miles out.

To do this, simply click the dashed line in the Bid adj. field for the 30 mile target and select “Decrease by” and then enter 30 in the percentage field.  Click “Save” to lock in the bid adjustment for the location target.

Adding a bid adjustment in an enhanced campaign.

Including bid adjustments for location targets in enhanced campaigns.


That’s it!  You just set up a smart scenario for bidding normally for users located closer to your location, while lowering your bids for users outside your typical service zone.  Dare I say we’ve just covered an enhanced way to bid.  :)

Summary – Use the (Local) Power of Enhanced Campaigns
What I explained above is just one way for local businesses to take advantage of enhanced campaigns.  You can also mix various targeting methods with bid adjustments to create advanced-level targeting scenarios.  For example, combining desktop, mobile, and location to ensure you are bidding appropriately for the right devices, and for the right locations.  But that’s for another day, and another tutorial.

For now, I recommend familiarizing yourself with enhanced campaigns, and testing various location targeting scenarios.  Use bid adjustments by location to ensure you are bidding correctly, based on the user’s location.  Then you can expand from there.  Good luck.

GG

Posted by Glenn Gabe @ 2:09 pm in adwords, google, sem
No Comments »
Friday, February 15th, 2013

How to Combine Custom Audiences in Facebook Ads to Enhance Your Targeting [Tutorial]

Custom Audiences in Facebook

Facebook recently released a powerful new option for advertisers called Custom Audiences.  Using custom audiences, advertisers can leverage their current in-house list of customers for targeting ads.  By uploading a list of emails, phone numbers, or UID’s, you can create a custom audience that can be used for targeting Facebook campaigns.

In my opinion, this was a brilliant move by Facebook.  It brings a unique targeting capability to the social network, and can be extremely useful on several levels.  For example, are you launching a new product?  Then use your custom audience to make sure your current customers know about the new product by reaching them on Facebook.  Know that a certain group of customers are interested in a given category of products?  Then use a custom audience to target just those customers with specific ads, copy, and calls to action.  The sky is the limit with regard to ideas for targeting your current set of customers, and I’ve been using custom audiences more and more recently.

Using Segmentation to Move Beyond Your One In-house Email List
A business can easily export its in-house email list and upload it to Facebook to create a custom audience.  It’s relatively straight-forward to do so, and you can accomplish this via Power Editor.  Once Facebook processes your list, it’s available to use when targeting an audience.  But, you shouldn’t stop there…  You can slice and dice your in-house email list and upload several files (if you have criteria for segmenting your list).

For example, do you know which customers are interested in which categories you sell?  Break those out.  Do you know which customers are tied to which purchases?  Sure you do, break those out too.  Once you do, you’ll have several targeted lists of emails that you can combine to hone your targeting.  And who doesn’t like that idea?

Combining Custom Audiences
When using Remarketing in AdWords, there is something called custom combinations.  When advertisers create a custom combination, they can create a remarketing audience that includes one audience, but excludes another.  That’s extremely powerful and provides a lot of flexibility for businesses trying to reach their customers via retargeting efforts.  Well, combining custom audiences in Facebook Ads enables you to do the same thing.

Here’s a simple hypothetical situation.  Let’s say you sold amazing new earphones that are invisible to the naked eye.  You already blasted an email out to your current customers and received some orders.  If your full email list was uploaded to Facebook as a custom audience (which should be done anyway), then you could create a second audience that includes customers that already purchased the new earphones.

Then, when you create a new campaign targeting your in-house email list (promoting your new earphones), you can exclude the list of customers that already purchased them.  This saves you from looking foolish, cuts down on wasted impressions, wasted clicks, and wasted budget.  Yes, that’s a simple example, but shows the power of creating custom combinations in Facebook.

How To Use Custom Combinations with Facebook Ads
Let’s quickly walk through how to set this up in Facebook.  Below, I’m going to explain how to first create a custom audience, and then how to upload and use a second audience (that can be used to hone your targeting).  Let’s create a custom combination using custom audiences in Facebook:

1. Export a straight list of customer emails as a .csv file.

Exporting a CSV of emails to create a custom audience.

 

2. Launch Power Editor and click the “Custom Audiences” Tab.
Note, if you’ve never used Power Editor, set that up now, download all of your campaigns, and then revisit this tutorial.

Custom Audience Tab in Facebook Ads

 

3. Click the “Create Audience” button and enter the name, description, and choose the type of list. 
For this list, click the “Emails” radio button.  You should also click “Choose File” button to locate the csv file we just created in the previous step.

The Custom Audience Dialog Box in Facebook Ads

 

4. Click “Create” and Facebook will upload your list and create your custom audience. 
Note, it could take a few hours for Facebook to process the file.  That depends on your list.  Remember, Facebook is going to scan the emails and try and match them up to current Facebook users.

 

5. Wait for Facebook to process your custom audience.
The status for the custom audience will say, “Waiting” while Facebook is processing the file.  That will change to “Ready” when the audience is ready to go.
You should also see the audience size (based on the users that Facebook could match up).

Custom Audience Status Message

 

6. Repeat the process in steps 1-5 to create a second custom audience (the hypothetical list of customers that already purchased our killer new earphones).
Make sure you give the new custom audience a descriptive name like “customers-invisible-earphones”.

 

7. Create a new campaign that will be used to target your current customers that have not purchased your new earphones yet.
Simply use the standard process for setting up a new Facebook campaign.

Creating a New Facebook Campaign

 

8. Select your custom audience.

When you create a new ad within your new campaign, you can hop down to the “Audience” tab.  You can click the button labeled “Use Existing Audience”.  Then select your full in-house email list.  That’s the first custom audience we created.

Use Existing Audience in Facebook Ads

 

9. Now select the custom audience to exclude.

Next, click the “Advanced Options” tab under “Audience”.  You will see an option for “Excluded Audiences”.   You can start typing the name of the custom audience containing customers that already purchased your earphones (the second custom audience we created).  The audience name should auto-populate when you start typing.  After selecting the audience, you should see the “Estimated Reach” number drop, based on excluding the new list.

Combining Custom Audiences to Enhance Targeting

 

10. That’s it, you have now used a custom combination to hone your targeting using Custom Audiences!
Your ads will now only be displayed to customers on your email list that have not purchased your new earphones yet.

Summary – Combine Audiences for Power
As I explained earlier, using custom audiences is a new and powerful way to reach a targeted audience on Facebook.   It combines the power of a current, in-house email list with the flexibility and intelligence of segmenting your audience.  Don’t look foolish, don’t waste clicks, and don’t waste budget.  Use custom combinations to slice and dice your current customer list.  Now go ahead.  Set up your campaign now.  :)

GG

 

Posted by Glenn Gabe @ 11:12 am in facebook, social-advertising, tools
2 Comments »
Tuesday, January 22nd, 2013

Facebook Graph Search – My Initial Thoughts on Speed, Privacy, Reputation Management, Not Provided, and more

Facebook Graph Search

Last week, Facebook unveiled Facebook Graph Search, which is an important step forward for the social network.  Many in digital marketing, including myself, wondered why Facebook had never added a serious search feature to its platform.  It made no sense…  Until now, Facebook’s search functionality has been horrible.  I often joked that you couldn’t find yourself via Facebook search… :)

By the way, I also wrote a post in October titled “BeastRank”, after Mark Zuckerberg hinted at a Facebook Search Engine.  You should check out that post to find out the various ways that Facebook could rank its social search results (based on the enormous amount of data it knows about all of us).

Now that Graph Search has arrived, Facebook finally has the opportunity to better monetize its massive user-base (eventually).   Note, Graph search is only in an early form and will absolutely evolve over time.  And no, it is not being monetized yet.  But you better believe that’s in the works.

What I’m Going to Cover
Facebook Graph Search is definitely an interesting take on social search.  I’ve been testing it heavily over the past few days and will cover several of my findings in the post below.  You can currently search for people, places, photos, and interests, and I dug into those areas in detail over the past few days.

Note, I’m not going to cover Graph Search basic functionality, or a tutorial on how to use the new search features.  There are plenty of posts covering the basics.  Instead, I’m going to cover certain aspects of Graph Search that relate to privacy, adoption, SEO, speed, etc.  Let’s jump in.

Autocomplete and the Facebook Graph Search Funnel
Once you start using Graph Search, you’ll notice that there are autocomplete suggestions as you start to type.  These are essential for Facebook, since it’s not 100% intuitive how you are supposed to structure queries using Graph Search.  All of us are used to Google, where you can basically enter anything and have the search engine return results in seconds.  I found myself wanting to enter the exact triggers that would surface the right Facebook data.  That could be a challenge for Facebook.

Graph Search Functionality

If you perform a search that would trigger Graph Search results, then Facebook will offer its guidance.  For example  “Restaurants in {location}” will auto-populate with locations based on your profile, and will include the Graph Search icon for restaurants next to the suggestion.  If you search outside of Graph Search, Bing’s autocomplete takes over (since Bing provides backup for Graph Search).

Facebook Graph Search Icons

After performing hundreds of searches over the past few days, I found that Facebook definitely helps you out and starts to rework your searches to fit the Graph Search lingo.  For example, if you simply enter a location and restaurants, Graph Search changes that to the “Restaurants in {location}” format.  I think the average person might be confused with what they can search for, and how to best surface the information.  I know I found it a little confusing, and I’m neck deep in search… That makes me think my mom will have no idea what she is doing.  :)

Speed is Important, and Delays Matter
One thing I noticed is that the speed that Facebook returned the results lagged sometimes.  And when you are used to a fraction of a second with Google, a few seconds (or longer) felt like a lifetime.  I think it’s just another example of how we take Google for granted.  If you spend some time in front of Graph Search, I think you’ll appreciate the speed at which Google moves (and for extremely advanced queries).

There were several times that Facebook just hung after I conducted search… I literally viewed a blank screen for a while until the results were returned.  Again, this will kill the user experience for Facebook Search, who will undoubtedly leave for Google’s near-instant results.

Graph Search Performance

Local Graph Search – Places, Restaurants, etc.
I found place search pretty cool, but often ran into situations where Facebook didn’t have enough data to provide thorough results.  Just a straight search for “Restaurants near Princeton” returned some good results (along with social annotations).  I saw a lot of my favorites on the list, but I didn’t need Facebook to know them…  There’s a big difference right now between Graph Search and just using Yelp’s app on my phone.  Yelp has a ton of data, reviews, etc.  Graph Search seems lacking certain areas, and especially outside of the core categories like Restaurants.

Local Graph Search

If I searched for “restaurants liked by my friends”, I started to see some results that were interesting.  I just might want to take a closer look at several of those restaurants.  And that is the power of Graph Search, if it can consistently provide recommendations based on my friends’ data.  But, there needs to be enough data, and that information needs to be helpful.  That’s different than simply returning restaurants that my friends “liked”.

Local Graph Search with Friends' Likes

From the results page, I can like the restaurant, view it on a map, view photos taken by people, and also view visitors of the restaurant.  This is where some privacy concerns popped up…  Good segue.

Places and Privacy (Photos and Visitors)
When you click “photos in” from the local search results page, you will see all photos taken at the restaurant (or place).  By the way, almost all of those photos are from people I don’t know, I’m not friends with, etc.  Sure, the photos were shared publicly, but I couldn’t help but question if those people even knew the photos were shared publicly.  And now the photos are available to anyone searching Facebook (via Graph Search).

Finding Photos Connected to Facebook Places

For some restaurants, there were hundreds of photos of people and I could easily see their names, click through to their profiles, like their photo (creepy), view comments and likes, etc.  Again, I’m not sure all of those people understand this is happening.  Oh yeah, there were many photos of kids too…  Way to go parents.  You just posted public photos of your kids for the world to see.  Ugh.

Photos of Places in Graph Search

Visitors and Privacy
If I click “Visitors” when viewing a list of restaurants, Facebook will show me people that visited the restaurant (via check-ins).  On that page, I can add them as a friend, message them (creepy), or view their photos, friends, and interests.  There is also a faceted navigation that lets me drill into visitors by employer, city, school, hometown, etc.  Yes, even more concern over privacy here.

People Who Visited Places in Graph Search

Faceted Navigation, Graph Search-Style
When using Graph Search, you can often drill deeper into the search results via a faceted navigation.  For example, when you search for a place, the navigation is present by default and enables you to fine-tune your search.  When I searched for, “Restaurants near Princeton my friends like”, the faceted navigation enables me to choose other types of places like bars, hotels, gyms, parks, etc.

I can also choose which friends I want to focus on (if I know certain friends like similar things).  I can also change the location, select “visited by” and then choose a friend.

Faceted Navigation in Facebook Graph Search

Cool, But How Many Will Use The Faceted Navigation?
If you like shopping online and enjoy using a faceted navigation to refine your searches, then you’ll like this functionality.  I definitely found it useful and quickly began digging into more Facebook data.   That said, I’m not sure how many people will use it like I did… When you think of the average user (who is spoiled by Google doing everything for them), they might not want to refine their searches.  They might expect Graph Search to be smart enough to refine it for them.  Time will tell how many people use the faceted navigation in Graph Search.

Online Reputation Management (ORM) and Graph Search
I’ve worked on a lot of online reputation management projects over the years, so I started to dig into that aspect of Graph Search.  If you type someone’s name, autocomplete first tries to display your friends’ names.  But as you select a friends’ name or complete their name in the search bar, the autocomplete suggestions focus entirely on that person.  You’ll see “friends of”, “photos of”, etc.

Online Reputation Management and Graph Search

“See more”, and I mean more…
So, Graph Search provides autocomplete suggestions for specific people, but what happens when you click the “see more” link at the bottom of autocomplete?  Well, when you click the “see more” link, autocomplete suggestions prompt you to search for all sorts of interests from the person in question.  That includes music, sports, pages, books, groups, apps, etc.

That’s cool, but what’s that further down the list?  Oh, religious views and political views!  “Danger, Will Robinson!”  At some point in your life, you learn that politics and religion are extremely polarizing topics.  Well, how about if the world could easily drill into that data about you?  Using Graph Search, they potentially could…

Viewing more user data via Graph Search

So, if you don’t have your privacy settings set up correctly, the world can now view your political and religious affiliations.  If you’re cool with that, fine.  If you just fell out of your seat in horror, then go make changes now.  :)  And to make drilling into the data even easier, there’s a nice dropdown on the page that enables me to find out the religious and political affiliations of any of my friends.

Graph Search and “Not Provided” – Hiding the Referrer
As I mentioned earlier, Graph Search falls back to Bing’s search results when it can’t meet a certain query.  But how does that data get passed to the destination site?  Will we begin to see searches from Graph Search in our analytics packages, and can we view the keywords people are searching for?  That would be logical, right?  I mean what kind of search engine would hide keywords from webmasters?? :)

Well, you aren’t going to be happy with the answer.  I checked the referrer leading from Facebook Graph Search to websites, and it unfortunately looks like any other visit from Facebook.  So I guess we have our own version of “not provided” from Graph Search.  :)  Come on Facebook!  Pass the query along in the referrer… SEO’s will love you for it.  Until they change that, you will see a standard referral from Facebook even when that person used Graph Search.

Click the image below to see a larger version of the referrer:

The Hidden Referrer Problem and Graph Search

It’s worth noting that Facebook Graph Search almost always keeps you within Facebook.  It’s only when the results fall back to Bing that you get standard search results that take you off the site.  That’s when I think Facebook should send the referrer.

Summary – Pay Attention to the Details
Graph Search is big news, even if this beginning state isn’t perfect.  Sponsored results (advertising) can turn into a lot of revenue for Facebook, as long as the search experience is strong and the results are compelling.  But since it’s social in nature, and based on our data, Facebook has to be really careful with privacy.  Based on my research so far, I wouldn’t give Graph Search outstanding scores on that front.  In addition, the user experience needs to be faster, and improve, in order to meet the Google standard.  Sure, it’s a new twist on search, but Facebook can’t take a step back with performance.

Needless to say, I’m eager to see how Graph Search evolves.  And I’ll be closely watching adoption, user experience, privacy, the quality of the search results, etc.  I’ll be writing more posts on the subject in the coming months, so definitely check back often.  Now excuse me while I find a Chinese restaurant near Princeton that my friends like.  :)

GG

 

 

Posted by Glenn Gabe @ 5:22 pm in facebook, google, local, social-media
8 Comments »
Friday, January 4th, 2013

Faulty Redirects, Duplicate Content, and SEO – How a Redirect Glitch Created Hundreds of Thousands of Duplicate Pages

Redirect Glitch Causing SEO Problems

With the release of index status in Google Webmaster Tools, many webmasters are now questioning why their “not selected” numbers are high.  They wonder if those numbers are good, bad, normal, etc?  Unfortunately, there’s not an easy answer to that question, since it depends on the site at hand.  But, you can definitely look at the ratio of “not selected” to pages indexed to start to understand if there is a technical problem causing a spike in pages being categorized as “not selected”.

For example, if you have 200 pages indexed on your site, and you see 350 categorized as “not selected”, that might be ok.  But if you see 25K pages as “not selected” or more, then that could raise a red flag that something may not be right with the site…  For example, is there a site structure issue that’s causing thousands of variations of pages with extremely similar content (duplicate content)?

A Recent Example of a Poor “Not Selected” Ratio
During SEO audits, there are times I come across significant problems like the one I mentioned above.  And those problems could be inhibiting a company’s search efforts (to say the least).   During a recent SEO audit, I came across a very interesting situation.  Index status revealed an extremely high number of “not selected” pages (as compared to the number of pages indexed) and I found myself digging into the site to find out why.

High Percentage of "Not Selected" Pages in Index Status

I found several issues causing the problem, so there wasn’t just one issue pumping up the number.  That said, the problem I’m going to cover today was causing thousands of duplicate pages to be created, and without the site owner knowing.  The more pages I checked, the more duplicates I found.  And this is a problem that can easily slip through the cracks for many webmasters.  And that’s especially the case if a small or medium sized business is handling all website development on its own.   Below, I’m going to cover what I found, and more importantly, how you can avoid the problem in the first place.

The Danger of an Extra Character
As I was analyzing the site manually, and via a number of test crawls, I came across some URL’s that contained an extra character.  Specifically, the extra character was being appended to each canonical URL.  All of those URL’s were from one specific section of the site (which contained thousands of URL’s).  After digging into that section of the site, I found out that this problem was happening to almost every URL being linked to from a certain element within each page.  So, I honed in on that element within each page to find out how the duplicate pages were being created.  And by the way, it just so happens that the section of the site contains nearly 200K pages.  Yes, this was a huge problem that was uncovered.

How One Extra Character Created Thousands of Duplicate Pages

The Result – Duplicate Content Ad Infinitum
The core problem is that the extra character created a new URL, but that new URL was an exact duplicate of the canonical URL (the URL that should be resolving).  And as you can guess, both pages can be accessed on the site.  One part of the site links to the canonical version of the pages, while this problematic section linked to the duplicate versions of the pages.

So, right off the bat, we are dealing with at least 200K duplicate pages.  In addition, as more content is added to this section, more duplicate pages will be created over time (based on the extra character being added to each URL).  Also, the canonical URL tag was not being used on the duplicate pages, so that wasn’t helping this specific case.  And on that note, I wouldn’t advocate using the canonical URL tag to fix this problem…  Technical problems like this should be addressed at the code or structure level.

1 to 1 Ratio of Duplicate URL's to Canonical URL's

So, if this was left in place, this problem could generate an unlimited number of duplicate pages.  If 500K pages ended up there, then there would be 500K pages of duplicate content.  Not good, so I dug deeper to find out exactly what was causing the problem.

The Root Problem – Faulty Redirects
Let’s face it, we all need to implement redirects at some point.  And that introduces the possibility of a poor implementation, which can be catastrophic SEO-wise.  It’s one of the reasons that website redesigns and CMS migrations are so risky.  On that note, to learn how to avoid SEO disaster during a redesign or migration, you should check out my Search Engine Journal column on the subject.

For example, using 302’s versus 301’s, using meta refresh redirects, redirecting to the wrong pages, or having the redirect code bomb the URL’s.  For this situation, we ran into the “bombing of URL’s” problem.  The redirects were faulty, and were redirecting to URL’s with an extra character.

The Solution – Fix the Redirect Code!
So, hundreds of thousands of duplicate pages were being generated, and it was due to one piece of redirect code on the server.  The 301 redirects being generated simply added an extra character to the destination URL.  That’s it.  The fix will be implemented soon, and once the new redirect code is rolled out, the correct URL’s will resolve.

This situation underscores the fact that even one small piece of code could have serious implications SEO-wise.  If this situation was left unchanged, it could have ended up generating an unlimited number of duplicate pages.  Knowing the content on this site, my guess is the problem would have generated 500K-750K pages of duplicate content over the next 2-3 years.

How To Avoid This Situation
After reading this post, you might be scared that this could happen to you, or worse, that it’s happening right now.  I’m going to provide a short list of things you can do to make sure this doesn’t happen.  Of course, if you feel you are having problems already, you should have an SEO audit performed.

  • First, whenever you create redirects, make sure you have a system for testing those redirects before they launch.  You can do this a number of ways, including on a local server or test server prior to releasing the final code to production.  If you thoroughly test the redirects, you could nip serious problems in the bud.
  • Second, make sure your xml sitemaps contain the canonical url’s for the pages at hand.  Making sure you are feeding Google and Bing the correct URL’s can help them understand which ones should be considered the canonical url’s.
  • Third, you should develop a strategy for using the canonical URL tag on the site.  If the tag is present, then you can ensure that any duplicate pages pass their search equity to the canonical URL’s.  Note, I’m not saying that you should leave a technical problem in place!  Instead, I’m saying that having the canonical URL tag in place will make sure the engines pass any search equity to the correct pages on your site while you figure out solutions to your technical problems.
  • This final bullet assumes you are already experiencing problems with duplicate content from a technical problem.  If you are, and you cannot determine what’s going on, then invest in having a technical SEO audit completed.  To me, the provide the most bang for your SEO buck.  It’s a great way to find out what’s truly going on with your site (beyond just the problem I listed here).

Summary – Know Your Site
This case emphasizes something I’ve said a thousand times over the past few years.  It’s incredibly important to have a sound site structure in order to perform at your highest level SEO-wise.  Coding problems, site structure issues, flawed redirects, etc. can kill your SEO efforts.  It’s one of the reasons that I believe SEO audits are critically important.  They can catch all types of SEO issues and provide ways to remedy those problems.  You know, like generating hundreds of thousands of duplicate pages.  :)

GG

 

Posted by Glenn Gabe @ 10:38 am in google, seo
2 Comments »
Thursday, December 27th, 2012

Introducing SEO Bootcamp Princeton, A Hands-On SEO Training Course in Princeton NJ

SEO Training Topics

I absolutely love getting in front of a group of people to speak about SEO (and always have).  Over the past several years, I’ve led SEO training classes for clients covering a wide range of topics, from technical SEO to keyword research to content optimization to linkbuilding strategy.  Although I’ve really enjoyed leading classes like this, I’ve always wanted to launch a training program that anyone could sign up for, and not just clients.  Well, I finally put the program together, and it’s called SEO Bootcamp Princeton.

SEO Bootcamp Princeton is a three hour, in-person training course being held at the Johnson Education Center (at D&R Greenway) on January 17th, from 9AM to 12PM.  You can register online via EventBrite, and there’s a 20% off, early registration discount running through 12/31/12.  If you register by then, tickets are $145 versus the standard price of $179.

The Target Audience for SEO Bootcamp Princeton
So, what will you learn at SEO Bootcamp Princeton?  Put simply, you’ll learn a lot.  My goal is to make sure attendees can leave the training ready to make changes to their websites.  I’ve crafted the training so it can be valuable for any person marketing a business, from small business owners to corporate marketers.  SMB’s will learn the tactical knowledge necessary to build a solid SEO foundation, while corporate marketers can learn SEO best practices and techniques.

In addition, the training will be extremely valuable for creative professionals, including designers, programmers, copywriters, etc.  I used to work for a large agency in New York City, and I led a similar type of training there.  I can tell you that every creative professional left the training with a stronger understanding of Search Engine Optimization (SEO).  Actually, I know the training changed how some people performed their jobs on a regular basis…

For example, designers and programmers learned about search engine friendly ways to design and code sites, while copywriters learned how to perform keyword research and properly optimize content.  Professionals involved with information architecture (IA) learned how to best structure a navigation, while also learning the best ways to build an internal linking structure.  And everyone in the training learned about the risks of redesigning a website without taking SEO into account.

Those are just a few of the SEO topics you’ll learn more about at SEO Bootcamp Princeton.  Again, my goal is that you leave with a much deeper knowledge of SEO, that you can make changes immediately, and that you take SEO into account whenever working on a website, campaign, or redesign.  You can learn more about the topics I’m going to cover on the SEO Bootcamp Princeton webpage.

SEO Bootcamp Princeton is Job-Agnostic – All Levels and Positions Will Benefit

Technical and Creative Job Titles

Tools and Plugins
SEO is definitely a mix of art and science.  And in order to assist SEO professionals with several core tasks, there are many tools and plugins one can use.  During the training, I will highlight several of the tools and plugins that can make your job easier SEO-wise.  I’ve always said that when you combine the right tools with the right SEO knowledge, great things can happen.  And I’ll make sure to explain some of my favorites along the way.  From Firefox plugins to Chrome extensions to standalone software applications, you’ll leave the training with a list of tools that can help you on a regular basis.

SEO Tools Training

Major Algorithm Updates
I can’t leave this post without touching on a very important topic in SEO that’s affecting many business owners.  Google has launched several important algorithm updates since early 2011, including both Panda and Penguin.   As you can imagine, I receive calls every month from business owners that have gotten hammered by these updates.  During SEO Bootcamp Princeton, I will introduce each major algorithm update, and cover important insights based on helping a range of businesses deal with the aftermath of getting hit.  And more importantly, I can explain the best ways to avoid getting hit in the first place.  You can read several of my case studies about Panda recovery and Penguin recovery if you are interested in learning more.

Panda and Penguin Algorithm Updates

Next Steps, Register Today
In closing, I’m ultra-excited about SEO Bootcamp Princeton.  If you are interested in registering, you can sign up via the EventBrite page.  Again, there’s a 20% off, early registration discount running through 12/31.  After 12/31, the standard pricing will be $179 per seat.  If you have any questions about the training, don’t hesitate to contact me.  It would be great to see you there!

GG

 

Posted by Glenn Gabe @ 10:18 am in google, seo, tools
No Comments »
« Older Entries